AS2 and SFTP Retry Logic for Payment File Delivery

Retry logic is where reliability and idempotency collide. A retry exists to recover from a transient failure, but a payment corridor delivers at-least-once, so an over-eager or careless retry is exactly how the same ACH file gets pulled — and posted — twice. This page isolates one task: wrapping an AS2 or SFTP delivery in a bounded, jittered retry that recovers from transport faults without ever reprocessing a file that was in fact already delivered. It is the retry half of the resilient file transfer gate, within the broader automated file ingestion and parsing pipelines reference, and it pairs with the PGP signature verification step that runs on every accepted file.

The hard part is not the backoff formula; it is the ambiguity. When an AS2 POST completes but the Message Disposition Notification never arrives, the sender does not know whether the receiver got the file. When an SFTP get times out mid-transfer, the client does not know whether a later attempt will collide with a partial artifact. In both cases the safe behavior is to retry — and the correct behavior is to make that retry a no-op if the file was already handled. That is what a dedup key buys, and why retry logic and idempotency have to be designed together rather than bolted on.

Concept spec: idempotency, backoff, and receipts

Three ideas define the wrapper.

Idempotency. Each delivery carries a stable key derived from its content and originator, so a redelivered file resolves to a key the pipeline has already seen and is suppressed before any side effect. The key is the content fingerprint plus the sender identity, independent of filename and timestamp — both of which change between a delivery and its redelivery.

Exponential backoff with full jitter. Successive retries wait progressively longer so a struggling corridor is not hammered, and jitter decorrelates a fleet of workers so they do not retry in lockstep. The delay before attempt is

where is the base delay and the cap. With full jitter the actual sleep is drawn uniformly from , which both smooths the herd and keeps the geometric ceiling. The attempt count is bounded by a hard cap , so a total corridor outage fails over to escalation after attempts rather than looping indefinitely.

MDN / receipt for AS2. AS2 (RFC 4130) confirms delivery with a Message Disposition Notification — optionally signed, and optionally carrying a Message Integrity Check (MIC) the receiver computes over the payload. A matching signed MDN is proof of receipt and integrity; a missing MDN is not proof of non-delivery. The wrapper therefore retries on a missing MDN but relies on the dedup key to absorb the case where the file actually did arrive.

Full annotated retry wrapper

The wrapper below is transport-agnostic: it takes a delivery callable (an SFTP get or an AS2 POST-and-await-MDN) and drives it under a bounded, jittered backoff, consulting a dedup ledger so a redelivered file is recognized and skipped. Backoff sleeps are computed from a monotonic clock so node clock skew cannot distort the schedule. Only transport faults are retried; a signature or checksum failure raised by the delivery callable is permanent and propagates immediately.

python
from __future__ import annotations

import logging
import random
import time
from dataclasses import dataclass
from typing import Callable, Protocol

logger = logging.getLogger("payment_delivery_retry")


class PermanentDeliveryError(Exception):
    """Non-retryable: bad signature, checksum mismatch, auth rejection."""


class TransientDeliveryError(Exception):
    """Retryable: timeout, reset connection, missing MDN, gateway 5xx."""


@dataclass(frozen=True)
class DeliveryResult:
    content_hash: str          # SHA-256 of the delivered payload
    dedup_key: str             # originator + content_hash
    mic_verified: bool         # AS2 MDN MIC matched (True for SFTP+checksum)


class DedupLedger(Protocol):
    def seen(self, key: str) -> bool: ...
    def record(self, key: str) -> None: ...


@dataclass(frozen=True)
class RetryConfig:
    max_attempts: int = 5
    base_seconds: float = 1.0
    cap_seconds: float = 60.0

    def backoff(self, attempt: int) -> float:
        """Full-jitter delay before *attempt* (1-indexed), from [0, ceiling]."""
        ceiling = min(self.cap_seconds, self.base_seconds * 2 ** (attempt - 1))
        return random.uniform(0.0, ceiling)


def deliver_with_retry(
    deliver: Callable[[], DeliveryResult],
    ledger: DedupLedger,
    config: RetryConfig = RetryConfig(),
    sleep: Callable[[float], None] = time.sleep,
) -> DeliveryResult | None:
    """Drive a payment-file delivery under bounded, jittered retry.

    Returns the DeliveryResult on first successful, non-duplicate delivery,
    or None if the payload was already processed (suppressed). Raises
    PermanentDeliveryError immediately on a non-retryable failure, and after
    the attempt cap is exhausted on persistent transient failures.

    The delivery callable is responsible for the transport (SFTP get or AS2
    POST + MDN await) and for classifying its own failures as transient or
    permanent. This wrapper owns only the backoff schedule and the dedup gate.
    """
    last_exc: TransientDeliveryError | None = None

    for attempt in range(1, config.max_attempts + 1):
        try:
            result = deliver()
        except PermanentDeliveryError:
            # Forged / corrupt / rejected — never retry, fail loud.
            logger.error("Permanent delivery failure; not retrying")
            raise
        except TransientDeliveryError as exc:
            last_exc = exc
            if attempt < config.max_attempts:
                delay = config.backoff(attempt)
                logger.warning(
                    "Transient failure on attempt %d/%d: %s — retrying in %.2fs",
                    attempt, config.max_attempts, exc, delay,
                )
                sleep(delay)
                continue
            break  # cap exhausted; fall through to raise below

        # Delivery returned — now enforce idempotency BEFORE any side effect.
        if ledger.seen(result.dedup_key):
            logger.info("Duplicate delivery %s suppressed", result.dedup_key)
            return None
        # Record before handing off so a crash cannot let it re-process.
        ledger.record(result.dedup_key)
        logger.info(
            "Accepted delivery %s (mic_verified=%s)",
            result.dedup_key, result.mic_verified,
        )
        return result

    raise TransientDeliveryError(
        f"Delivery failed after {config.max_attempts} attempts"
    ) from last_exc

The two invariants that make this safe: retries are bounded and jittered so a corridor outage degrades to escalation rather than a storm, and the dedup ledger is consulted on every returned delivery — including a retry that succeeded after an earlier attempt secretly also succeeded — so a redelivered file is suppressed before it can be reprocessed. Recording the key before returning closes the crash window in which a restart could re-accept the same payload.

Calibration & configuration

  • Max attempts. Five is a sane default for interactive corridors; scheduled batch corridors that run well ahead of the settlement cutoff can afford more. The cap exists to bound worst-case latency and to convert a persistent outage into a paged escalation, so tune it against how much time the downstream Reg E and settlement clocks can spare.
  • Base and cap. A base of one second and a cap in the range of 30–60 seconds suits most SFTP and AS2 endpoints. Raise the cap for a slow correspondent gateway so a legitimately sluggish response is not abandoned; lower it for a fast domestic endpoint where a long wait just adds latency.
  • Per-protocol tuning. SFTP faults are usually connection-level (reset, timeout, banner hang) and clear quickly, so a shorter base works. AS2 faults often involve a missing or delayed MDN, where the receiver may still be processing; a slightly longer base avoids resending a file whose MDN is merely late. In both cases jitter is mandatory at fleet scale — without it, workers synchronize into a thundering herd against a recovering corridor.

Before / after: the missing-MDN scenario

Consider an AS2 delivery of ACH_20260716.edi, content hash 9f2c…a107, dedup key ORIG42:9f2c…a107. The receiver accepts and processes the file, but the return path drops the signed MDN before the sender sees it.

Before (naive retry, no dedup gate). The sender's retry treats the missing MDN as a failure and resends. The receiver accepts the resent file, and because nothing recognizes it as a duplicate, it is parsed and posted a second time — a duplicate ACH debit for the full batch, surfacing days later as an unexplained exception the operations team has to unwind manually.

After (this wrapper). The resend arrives, the delivery callable computes the same content hash 9f2c…a107 and the same dedup key ORIG42:9f2c…a107, and ledger.seen(...) returns true. deliver_with_retry returns None, the file is suppressed with a logged no-op, and nothing is posted twice. The first, real delivery stands; the redelivery is absorbed exactly as the at-least-once corridor requires. Backoff, meanwhile, spaced the resend attempts as random.uniform(0, 1), random.uniform(0, 2), random.uniform(0, 4) seconds rather than three immediate hammer-blows.

Failure modes & guardrails

  1. Retry storm. Fixed-interval or pure-exponential backoff without jitter makes a fleet of workers retry a failed corridor in lockstep, amplifying the outage. Guard with full jitter drawn from [0, ceiling] and a hard attempt cap; drive the sleep from a monotonic clock so clock skew cannot collapse the spacing.
  2. Non-idempotent side effects. If a retry re-runs a step that already posted before the failure, the retry double-posts. Guard by making the content fingerprint the idempotency key and consulting the dedup ledger before any side effect — and by recording the key before handing the file downstream, not after.
  3. MDN lost but file delivered. The ambiguous case: the payload arrived, the receipt did not. Retrying is correct (you cannot prove delivery), but only because the dedup key makes the redelivery a no-op. Without the dedup gate, retrying a lost-MDN delivery is precisely what manufactures a duplicate — so never ship the retry logic without the suppression that backs it.

Frequently Asked Questions

Why full jitter instead of a fixed or purely exponential delay?

A fixed delay makes every worker retry at the same instant; pure exponential backoff without jitter does the same, just with growing gaps. Both synchronize a fleet into a thundering herd that hammers a recovering corridor in waves. Full jitter draws each delay uniformly from zero up to the exponential ceiling, which decorrelates the workers while preserving the geometric back-off. At any real fleet size, jitter is the difference between a corridor that recovers and one that is kept down by its own clients.

Should a missing AS2 MDN trigger a retry?

Yes — a missing MDN is not proof of non-delivery, so the safe action is to retry. What makes that safe rather than dangerous is the dedup key: if the file actually did arrive, the resend resolves to a key the pipeline has already seen and is suppressed before any reprocessing. Retry on the missing receipt, and let idempotency absorb the case where the receipt was merely lost in transit.

What belongs in the dedup key?

The content fingerprint (a SHA-256 over the payload bytes) plus the originator identity. Filenames and timestamps must be excluded because they differ between a delivery and its redelivery, which would defeat suppression. Content-plus-sender is stable across resends and distinguishes two originators who might coincidentally send byte-identical control files. The same key anchors the retry idempotency check and the audit trail.

Which failures should never be retried?

Anything permanent: an invalid signature, a checksum mismatch, an authentication rejection, or a malformed payload. Retrying these wastes the attempt budget and can mask a real corruption or forgery by eventually succeeding on a different delivery. Classify them as permanent at the delivery callable so they propagate immediately to quarantine, and reserve retries for transport faults — timeouts, reset connections, gateway 5xx, and missing MDNs.