Resilient File Transfer: Retry Logic & Signature Verification #

The transport edge is where correctness is quietly won or lost. Before a NACHA batch or an ISO 20022 camt.053 statement ever reaches a decoder, it has to be pulled off an SFTP endpoint or accepted over an AS2 channel, proven intact, proven attributable, and proven not-already-seen — and every one of those steps can fail in a way that looks like success. This guide sits inside the broader automated file ingestion and parsing pipelines framework and covers the reliability contract that stage owes the rest of the system: retrieve exactly the bytes the originator sent, retry transient failures without manufacturing duplicates, and verify integrity and signature before a single field is parsed. It builds directly on the transport primitives in secure file transfer protocols for banks and feeds the pydantic schema validation gate that runs immediately downstream.

The reason this deserves its own reference is that the transport edge operates under an at-least-once delivery model it does not control. Correspondent gateways retransmit, FedACH and CHIPS replay, operators re-run a stalled retrieval job, and an AS2 sender that never received a Message Disposition Notification (MDN) will resend a file it already delivered. Layer on top of that the ordinary hazards of a network transfer — a socket that drops mid-download, a truncated file that still opens cleanly, a byte flipped in flight — and the transport stage becomes the single place where a duplicate debit, a phantom exception, or a silently corrupted amount is either caught or waved through. The engineering answer is a small number of hard invariants applied in a fixed order: idempotent retrieval, bounded retry with backoff, verify-then-accept, and quarantine-on-failure.

The four invariants of a resilient transport edge #

Each stage enforces exactly one property, and the ordering is not negotiable — verification must precede parsing, and deduplication must precede any side effect that posts to a ledger.

• Idempotent retrieval. Pulling the same file twice must be indistinguishable, to the rest of the pipeline, from pulling it once. This is enforced with a content fingerprint (a streaming SHA-256) combined with a stable delivery identity, so a re-download or an AS2 resend collapses to a no-op rather than a second posting.
• At-least-once delivery with duplicate suppression. Because the transport layer will deliver some files more than once, the pipeline is designed to expect redelivery and suppress it, rather than to assume exactly-once and break when the assumption fails. Suppression happens at a dedup gate keyed on the fingerprint, before decoding.
• Integrity and signature verification before parsing. A checksum proves the bytes are intact; a detached signature proves who sent them and that they were not altered. Both are checked while the payload is still opaque. A file that fails either check is never handed to a parser — a corrupted 94-byte NACHA record can slice cleanly and post a wrong amount, so the gate has to close before the parser opens.
• Quarantine on failure, never silent drop. A transport failure, a checksum mismatch, or a signature rejection routes the file to a quarantine location with a structured diagnostic, where it is retried under policy or escalated to an operator. Nothing is discarded, because a discarded file is an audit gap.

The sections below implement each invariant in isolation, then compose them into one retrieval pass. Throughout, the fingerprint is computed once and reused as the retry idempotency key, the dedup key, and the audit anchor, so a single value ties the whole lifecycle of a file together.

Phase 1 — Idempotent retrieval as the foundation #

Idempotency is what makes every other invariant safe to retry. If retrieval is idempotent, a retry after an ambiguous failure — the download that may or may not have completed, the AS2 POST whose MDN was lost — can be issued freely, because a second successful pull produces no additional effect. The mechanism is a content fingerprint. A streaming SHA-256 over the downloaded bytes yields a stable identity that is independent of filename, timestamp, or transport metadata, all of which vary between a delivery and its redelivery.

The fingerprint alone is not quite enough, because two distinct originators can, in principle, send byte-identical control files. The durable identity is therefore a composite: the sending party plus the content hash. That composite is checked against a processed-file ledger — a small table or key-value store the pipeline owns — before any decoding or ledger side effect occurs. The lookup is $O(1)$; the hash pass is $O(n)$ in file length with $O(1)$ resident memory when streamed in fixed-size chunks, so idempotency costs one linear read and nothing more.

import hashlib
from pathlib import Path


def fingerprint(path: Path, chunk_size: int = 8192) -> str:
    """Streaming SHA-256 of a file — constant memory, O(n) in length."""
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        while chunk := handle.read(chunk_size):
            digest.update(chunk)
    return digest.hexdigest()


def dedup_key(originator_id: str, content_hash: str) -> str:
    """Stable delivery identity: who sent it + what it contains.

    Independent of filename and timestamp, which differ between a
    delivery and its redelivery, so a resend collapses to one key.
    """
    return f"{originator_id}:{content_hash}"

Phase 2 — A bounded retry policy with exponential backoff and jitter #

Transient transport failures — a reset connection, a gateway 502, a slow SSH banner — should be retried; permanent ones should not. The policy that separates them has three levers: a hard attempt cap so a corridor outage cannot loop forever, an exponential base so successive delays back off geometrically, and jitter so a fleet of workers retrying the same failed corridor does not synchronize into a thundering herd. The nominal delay for attempt $n$ is $d=\min (d_{max},b\cdot 2^{n-1})$; full jitter then draws the actual sleep uniformly from $[0,d]$, which decorrelates retries across workers while preserving the geometric ceiling.

The policy is a pure, frozen value object so it can be unit-tested deterministically (by seeding the RNG) and shared across corridors with per-counterparty overrides. Only a whitelisted set of exceptions is treated as retryable — a signature rejection or a checksum mismatch is a permanent failure and must never be retried, because retrying a genuinely corrupt or forged file just wastes the attempt budget.

import random
from dataclasses import dataclass


@dataclass(frozen=True)
class RetryPolicy:
    """Full-jitter exponential backoff with a bounded attempt cap."""

    max_attempts: int = 5
    base_seconds: float = 1.0
    cap_seconds: float = 60.0

    def delay_for(self, attempt: int) -> float:
        """Sleep before *attempt* (1-indexed). Full-jitter backoff.

        Nominal ceiling grows as base * 2**(attempt-1), clamped to
        cap_seconds; the returned delay is uniform over [0, ceiling].
        """
        if attempt < 1:
            raise ValueError("attempt is 1-indexed")
        ceiling = min(self.cap_seconds, self.base_seconds * 2 ** (attempt - 1))
        return random.uniform(0.0, ceiling)

    def should_retry(self, attempt: int, exc: BaseException) -> bool:
        """Retry only transient transport faults, never a verify failure."""
        transient = (ConnectionError, TimeoutError, OSError)
        return attempt < self.max_attempts and isinstance(exc, transient)

Phase 3 — The verify-then-accept gate #

Verification runs on the opaque payload, before parsing, and both checks must pass. The checksum is compared against an out-of-band manifest value the originator publishes alongside the file (or the AS2 Content-MIC returned in the signed MDN); the detached signature is verified against the originator's pre-registered public key. A file that fails either check is rejected as a permanent failure and quarantined — never retried, never parsed. Structuring the gate to return a typed decision rather than raising keeps the accept, quarantine, and suppress paths explicit at the call site.

from dataclasses import dataclass
from enum import Enum
from pathlib import Path
from typing import Callable, Protocol


class Decision(str, Enum):
    ACCEPT = "accept"
    QUARANTINE = "quarantine"
    SUPPRESS = "suppress"  # duplicate — already processed


class SignatureVerifier(Protocol):
    def __call__(self, payload: bytes, signature: bytes) -> bool: ...


@dataclass(frozen=True)
class GateResult:
    decision: Decision
    content_hash: str
    reason: str


def verify_then_accept(
    path: Path,
    signature: bytes,
    expected_checksum: str,
    verify_signature: SignatureVerifier,
    seen: Callable[[str], bool],
    originator_id: str,
) -> GateResult:
    """Integrity + signature gate, then dedup. Runs before any parse."""
    content_hash = fingerprint(path)

    # 1. Integrity: checksum must match the originator's manifest value.
    if content_hash != expected_checksum:
        return GateResult(Decision.QUARANTINE, content_hash, "checksum_mismatch")

    # 2. Attribution: detached signature must verify against a trusted key.
    if not verify_signature(path.read_bytes(), signature):
        return GateResult(Decision.QUARANTINE, content_hash, "signature_invalid")

    # 3. Idempotency: suppress a file we have already accepted.
    if seen(dedup_key(originator_id, content_hash)):
        return GateResult(Decision.SUPPRESS, content_hash, "duplicate_delivery")

    return GateResult(Decision.ACCEPT, content_hash, "verified")

Phase 4 — Composing the pass as a generator #

The full retrieval pass is a generator that walks the staged files, applies the gate to each, records the outcome, and yields only accepted paths downstream. A generator keeps the transport edge memory-bounded regardless of how many files a burst delivers, and it makes the accept/quarantine/suppress split a single, auditable branch. Quarantined and suppressed files are recorded and skipped; accepted files are marked as processed before they are yielded, closing the window in which a crash mid-batch could let the same file be re-accepted on restart.

from pathlib import Path
from typing import Callable, Iterable, Iterator


def resilient_ingest(
    staged: Iterable[Path],
    gate: Callable[[Path], GateResult],
    mark_processed: Callable[[str], None],
    quarantine: Callable[[Path, str], None],
) -> Iterator[Path]:
    """Yield only verified, de-duplicated files; route the rest safely."""
    for path in staged:
        result = gate(path)
        if result.decision is Decision.ACCEPT:
            # Mark processed BEFORE yielding so a crash cannot re-accept it.
            mark_processed(f"{result.content_hash}")
            yield path
        elif result.decision is Decision.QUARANTINE:
            quarantine(path, result.reason)
        # SUPPRESS: duplicate — already accounted for; drop silently but logged.

Failure modes at the transport edge #

Compliance & Auditability #

The transport edge is a control point examiners look at directly, because it is where transmission security and transaction completeness are established. Integrity and confidentiality controls over payment files in transit fall under the FFIEC guidance on authentication and transmission security, which expects both an authenticated channel and an attribution mechanism for high-value corridors — precisely the checksum-plus-signature pairing the gate enforces. The NACHA Operating Rules require that ACH data be transmitted using commercially reasonable encryption and security procedures over open networks, and that the receiving party be able to demonstrate the integrity of what it received; a recorded content hash and a verified signature fingerprint are the artifacts that demonstration rests on.

Auditability here means every branch of the flow is reconstructable. The append-only trail records, for each delivery, the attempt count and backoff timing, the computed checksum, the outcome of signature verification with the key fingerprint that satisfied it, and the dedup decision. A quarantined file retains its raw bytes and its diagnostic reason so an examiner or an operator can see exactly why it was held. Because the timestamp of verified receipt — not an arbitrary polling tick — is what anchors downstream regulatory clocks such as the Reg E consumer-notification window, it must be captured at the moment the gate accepts, from a single UTC source, and never inferred later. The structured payment fields that make cross-institution traceability possible on the modern messaging side are defined by the ISO 20022 standard.

Testing & Verification #

The two behaviors most worth pinning with tests are the backoff schedule and duplicate suppression, because both are easy to regress silently and both have direct financial consequences. Seeding the RNG makes full-jitter backoff deterministic, so the ceiling growth and the attempt cap can be asserted exactly. Dedup is tested by feeding the same fingerprint twice and asserting the second delivery is suppressed rather than accepted.

import random

from retrieval import Decision, RetryPolicy, dedup_key


def test_backoff_respects_cap_and_attempt_limit() -> None:
    policy = RetryPolicy(max_attempts=4, base_seconds=1.0, cap_seconds=8.0)
    random.seed(1)
    # Every drawn delay stays within [0, ceiling] and ceiling is clamped.
    for attempt in range(1, policy.max_attempts + 1):
        ceiling = min(policy.cap_seconds, policy.base_seconds * 2 ** (attempt - 1))
        assert 0.0 <= policy.delay_for(attempt) <= ceiling
    # The cap is exhausted at max_attempts — no fifth retry.
    assert policy.should_retry(4, ConnectionError()) is False
    assert policy.should_retry(1, ConnectionError()) is True
    # A signature failure is never retried, even on attempt 1.
    assert policy.should_retry(1, ValueError("signature_invalid")) is False


def test_duplicate_delivery_is_suppressed() -> None:
    ledger: set[str] = set()
    key = dedup_key("ORIG42", "a" * 64)

    def seen(k: str) -> bool:
        return k in ledger

    assert not seen(key)  # first delivery accepted
    ledger.add(key)
    assert seen(key)  # redelivery of identical content is suppressed
    # Suppression is a no-op, distinct from a quarantine.
    assert Decision.SUPPRESS is not Decision.QUARANTINE

Frequently Asked Questions #

Related guides in this collection #

• PGP Signature Verification for Inbound ACH Files — the detached-signature verify step of the gate, with a full reject-on-fail function and key-trust handling.
• AS2 and SFTP Retry Logic for Payment File Delivery — the bounded backoff-with-jitter retry wrapper, MDN receipts, and a dedup key that survives redelivery.
• Secure File Transfer Protocols for Banks — the SFTP, FTPS, and AS2 transport contracts and PGP payload encryption this stage builds on.
• Pydantic Schema Validation for Payments — the field-level validation gate that consumes the verified, de-duplicated files this stage produces.
• Async Batch Processing Architectures — how retrieval and retry are orchestrated concurrently with the CPU-bound decoding that follows.
• Automated File Ingestion & Parsing Pipelines — the parent reference that frames where the transport edge sits in the wider ingestion domain.