Detecting Duplicate ACH Debits with Idempotency Hashing #

At 05:12 an RDFI operator re-transmits a NACHA file that the FTP session had dropped mid-transfer the night before. Most of the file is new, but 4,120 debit entries are byte-for-byte repeats of yesterday's successful settlement. If the reconciliation engine posts them, 4,120 consumers are debited twice and the Reg E clock starts on every one. Duplicate and replay debits are the highest-frequency, lowest-ambiguity fraud-adjacent anomaly in ACH, and they are the first signal in the fused scorer described under fraud-pattern detection signals, itself part of the broader exception routing and fraud-pattern detection practice. This guide builds the deterministic detector behind that signal: a composite idempotency key, a SHA-256 digest, and a bounded seen-window that catches the replay without flagging a legitimate recurring debit.

The whole problem is defining "same debit" precisely enough that a re-presentment is a duplicate but next month's identical subscription charge is not. A NACHA Entry Detail Record (record type 6) gives you the fields to do it: the receiving DFI routing number and account (positions 04–12 and 13–29), the amount in cents (positions 30–39), the individual identification number, and the 15-digit trace number (positions 80–94) whose first 8 digits are the ODFI routing prefix and last 7 are a monotonic sequence. The effective entry date lives in the batch header (Company/Batch Header record, positions 70–75, YYMMDD). Combining the right subset of these into one key is what separates a robust detector from one that either misses replays or drowns recurring billing in false positives.

Concept Spec: A Composite Idempotency Key #

A duplicate is an equality test, and the engineering choice is which fields participate in the equality. Include too few and distinct debits collide; include too many — a per-file sequence counter, say — and a genuine re-presentment looks new. The stable composite key for an ACH debit is:

where $\parallel$ is a delimited concatenation and $H$ is SHA-256. Hashing is not for security here — it is for fixed-width keys and cheap set membership. The trace number is the workhorse: NACHA rules require it to uniquely identify an entry within a batch, so two entries sharing a trace and the same originator, amount, and effective date are the same logical debit re-presented, not two coincidental payments. Membership testing a batch of $n$ debits against a seen-set is $O(n)$ time with $O(w)$ space for a window holding $w$ prior keys, since each SHA-256 digest and set lookup is constant work.

The subtlety is the window. A key that lives forever in the seen-set would reject a legitimate identical debit that recurs on schedule — the same $49.99 gym membership, same originator, same trace-sequence reuse — months later. So the seen-set is bounded to a re-presentment horizon (NACHA permits re-presentment of a returned debit up to two times, typically within 180 days, but same-file and same-day duplicates are the dominant operational case), and the effective date is inside the key precisely so that the next scheduled occurrence produces a different key and passes cleanly.

Full Annotated Implementation #

The detector below builds the key, hashes it, and tests membership against a bounded window implemented as an ordered mapping of digest to first-seen timestamp. Amounts are handled as integer cents and never as float; the key components are length-delimited so that concatenation is unambiguous (without a delimiter, originator 12 + amount 34 and originator 1 + amount 234 would collide).

from __future__ import annotations

import hashlib
from collections import OrderedDict
from dataclasses import dataclass
from datetime import date, timedelta


@dataclass(frozen=True, slots=True)
class AchDebit:
    """Fields lifted from a NACHA Entry Detail (6) record + batch header."""
    originator_id: str      # Company Identification, batch header pos 41-50
    amount_cents: int       # Entry Detail pos 30-39, integer cents never float
    trace_number: str       # Entry Detail pos 80-94, 15 digits
    effective_date: date    # Company/Batch Header pos 70-75, YYMMDD


def idempotency_key(debit: AchDebit) -> str:
    """Composite SHA-256 key identifying one logical debit.

    Components are length-delimited so no two distinct field tuples can
    concatenate to the same byte string. The effective date is included so
    that the *next* scheduled occurrence of a recurring debit hashes
    differently and is not mistaken for a re-presentment.
    """
    parts = (
        debit.originator_id,
        str(debit.amount_cents),
        debit.trace_number,
        debit.effective_date.isoformat(),
    )
    payload = "\x1f".join(parts)          # unit-separator delimiter
    return hashlib.sha256(payload.encode("utf-8")).hexdigest()


class DuplicateDebitDetector:
    """Bounded seen-window dedup for ACH debits.

    A key is a duplicate if its digest is already in the window. The window
    evicts keys older than `horizon_days`, so a legitimate recurring debit
    on a later effective date is never blocked (its key differs anyway).
    """

    def __init__(self, horizon_days: int = 5) -> None:
        self._horizon = timedelta(days=horizon_days)
        self._seen: OrderedDict[str, date] = OrderedDict()

    def _evict(self, as_of: date) -> None:
        cutoff = as_of - self._horizon
        while self._seen:
            _, first_seen = next(iter(self._seen.items()))
            if first_seen < cutoff:
                self._seen.popitem(last=False)
            else:
                break

    def is_duplicate(self, debit: AchDebit) -> bool:
        """Return True and do NOT re-insert when the debit was already seen."""
        self._evict(as_of=debit.effective_date)
        key = idempotency_key(debit)
        if key in self._seen:
            return True
        self._seen[key] = debit.effective_date
        return False

Calibration of the Dedup Window #

The horizon is the one tuning knob that decides whether you catch replays without punishing recurring billing. Size it to the operational reality of the three cases it must separate:

• Same-file and same-day duplicates (horizon 1–2 days). The dominant case — a re-transmitted file, a batch submitted twice, an ODFI that reuses a trace sequence within a processing day. A short horizon catches all of these and evicts fast, keeping the window small. This is the default for high-volume debit origination.
• Multi-day re-presentment (horizon up to the return window). A debit returned R01 (insufficient funds) may be re-presented, and NACHA allows up to two re-presentments. If your pipeline must distinguish an authorized re-presentment from an accidental duplicate, extend the horizon but key re-presentments explicitly (the Standard Entry Class and the reinitiated-entry indicator in the addenda) so an intended re-presentment is not scored as a replay.
• Legitimate recurring debits (never the same key). A monthly subscription is the same originator and amount but a different effective date every cycle, so its key differs and it passes regardless of horizon. The failure only appears if you drop the effective date from the key — then two months of the same charge collide and the second is wrongly flagged. Keep the effective date in the key.

The rule of thumb: set the horizon to the shortest interval that still spans your realistic re-transmission gap, and rely on the effective-date component — not a long horizon — to protect recurring billing.

Before and After #

Take four debits arriving across two processing days, all from originator 1204885grp for $49.99 (4999 cents):

Debit B is the dropped-session re-transmit and D is a deliberate same-day replay; both hash to the identical digest as A and are held. Debit C is next month's legitimate recurring charge — a different effective date (and a different trace sequence) yields a different key, so it posts straight through even though the originator and amount match A exactly. A naive detector that keyed only on originator + amount_cents would have flagged C as a duplicate and generated a false positive on a perfectly good subscription payment.

Failure Modes & Guardrails #

Three edge cases turn a clean detector into either a fraud gap or a false-positive machine:

1. A legitimate identical recurring payment gets flagged. This happens only when the effective date is dropped from the key or the horizon is set absurdly wide. The fix is structural: keep effective_date in the composite key so successive cycles hash differently, and keep the horizon to the re-transmission gap, not the billing period. If a genuine recurring debit somehow shares an effective date (an off-cycle catch-up charge), route it to review rather than auto-holding, and let an analyst confirm against the mandate.
2. A truncated or malformed reference. Some ODFIs left-pad or reformat the trace field, and a legacy export may truncate it. If the trace is unreliable, the key silently weakens — two distinct debits with a blanked trace collide. Validate the trace is a full 15 digits before it enters the key; on a malformed reference, do not guess, route the entry to review and log the raw bytes so the parsing defect is fixed upstream rather than masked.
3. Hash collision handling. SHA-256 collisions are not a practical concern, but a key collision — two genuinely different debits that share every keyed field — is. That is not a hash failure; it means your key is under-specified for your traffic. When a "duplicate" is flagged, the safe disposition is a hold-and-verify, not a silent drop: re-compare the full source records behind the two matching keys, and only suppress the second when the underlying Entry Detail records are truly identical. Never discard a debit on a key match alone.

Frequently Asked Questions #

Related #

• Fraud-Pattern Detection Signals in Payment Reconciliation — the parent guide where this duplicate signal fuses with velocity, structuring, and Benford checks.
• Velocity Checks for Wire-Transfer Anomaly Detection — the sliding-window sibling signal that catches bursts rather than exact repeats.
• Applying Benford's Law to Payment-Amount Anomaly Detection — a batch-level amount signal that complements per-entry deduplication.
• NACHA Record Layouts Explained — the byte offsets for the trace, amount, and effective-date fields this key is built from.