Streaming Parser Memory Optimization for Large NACHA Files #

A 1.8 GB FedACH return file lands at 04:12 and the ingestion worker is killed by the OOM reaper at 04:14, mid-settlement. The parser code is correct — it decodes every field at the right byte offset — but it called read_fwf without chunksize, so pandas materialized the whole file, promoted half the columns to object dtype, and the resident set climbed past the 4 GB cgroup limit before the first record was ever validated. This is the failure a streaming parser exists to prevent: not wrong output, but a memory profile that scales with file size instead of staying flat. Within the broader automated file ingestion and parsing pipelines practice, and specifically inside high-volume pandas parsing strategies, this page drills into one dimension only — memory — and shows how to hold a multi-gigabyte NACHA file at a resident set that does not grow with the row count.

The distinction that matters is between throughput complexity and resident-memory complexity. Both a streaming parser and a full-file load process $n$ records in $O(n)$ time — you cannot read a file in less than linear time. The difference is space: a generator that yields one typed record at a time holds $O(1)$ resident memory, while a full-DataFrame materialization holds $O(n)$. The chunked reader documented in the parent guide bounds that to $O(k)$ per chunk; this page pushes further, to a per-record generator that never assembles a chunk-sized frame at all, for the cases where even a 500k-row chunk of wide records is too much under peak concurrency. That path also feeds the same downstream contract the async batch processing architecture consumes.

Concept Spec: Time Is Linear Either Way, Memory Is Not #

Let a NACHA file carry $n$ entry-detail records. Reading it is unavoidably $O(n)$ in time. The lever is the resident memory function $M$. A full-file load builds an in-memory structure proportional to the input, so $M_{\text{load}}=O(n)$. A generator that reads one line, decodes it into a fixed-size typed record, yields it, and lets the caller drop it holds only the current record plus a constant working set, so:

The constant hidden inside $O(1)$ is what this page tunes. Every per-record object Python allocates carries overhead — a plain dataclass instance keeps a __dict__, so a record with eight fields costs a dict plus eight boxed references. Declaring the record frozen=True, slots=True removes the per-instance __dict__, replacing it with a fixed slot array and cutting the per-record footprint by roughly half. Multiply that saving across the transient churn of millions of records and it is the difference between a flat resident line and a saw-tooth that the garbage collector cannot keep ahead of. Streaming does not make the work sublinear; it makes the coefficient on resident memory a small constant instead of the whole file.

Full Annotated Python Implementation #

The parser below reads a fixed-width NACHA file line by line through a generator, decodes each Record Type 6 into a slotted frozen record, and yields it with bounded RAM. It reuses a single mutable buffer object across iterations (an arena), drops references explicitly at the chunk boundary, and freezes the interned constant set out of the garbage collector's reach so the hot loop never pays a collection pause for objects that will never be collected.

from __future__ import annotations

import gc
from dataclasses import dataclass
from decimal import Decimal
from typing import Iterator, TextIO

# 0-indexed, end-exclusive byte slices for a NACHA Entry Detail (Type 6) record.
_R6 = {
    "transaction_code": (1, 3),
    "routing_number": (3, 12),
    "account_number": (12, 29),
    "amount": (29, 39),
    "trace_number": (79, 94),
}
_CENTS = Decimal("100")


@dataclass(frozen=True, slots=True)
class EntryDetail:
    """One immutable NACHA entry-detail record.

    frozen=True + slots=True drops the per-instance __dict__: the object
    holds a fixed slot array of five references, roughly halving the
    per-record footprint versus a plain dataclass. Amount is a Decimal in
    dollars — never float — parsed from the implied-2-decimal integer field.
    """

    transaction_code: str
    routing_number: str
    account_number: str
    amount: Decimal
    trace_number: str


def _decode(line: str) -> EntryDetail:
    """Slice one 94-char line into a typed record. No intermediate dict, no
    DataFrame row — the only allocation that outlives the call is the record."""
    amt_raw = line[_R6["amount"][0]:_R6["amount"][1]]
    return EntryDetail(
        transaction_code=line[1:3],
        routing_number=line[3:12],
        account_number=line[12:29].rstrip(),
        amount=Decimal(amt_raw) / _CENTS,     # implied 2 decimals, exact
        trace_number=line[79:94],
    )


def stream_entry_details(
    handle: TextIO,
    gc_generation_freeze: bool = True,
) -> Iterator[EntryDetail]:
    """Yield typed NACHA entry-detail records at O(1) resident memory.

    The file is walked one line at a time — iterating a file object is itself
    a generator, so no full read() ever happens. Only Record Type 6 lines are
    decoded; control records (5/8/9) are skipped without allocation.

    GC tuning: gc.freeze() moves everything already alive (module constants,
    the interned slice map) into a permanent generation the collector never
    rescans, so the cyclic GC does not walk long-lived objects on every
    threshold trip during the hot loop. gc.disable() then removes the
    generational pauses entirely; the dataclass records hold no reference
    cycles, so refcounting alone reclaims them the instant the caller lets go.
    """
    if gc_generation_freeze:
        gc.collect()          # settle current garbage first
        gc.freeze()           # exempt long-lived objects from future scans
        gc.disable()          # no generational pauses inside the loop
    try:
        for line in handle:               # lazy line iterator, never .read()
            if line[0] != "6":            # skip non-entry-detail, zero alloc
                continue
            yield _decode(line)           # one record out; caller drops it
    finally:
        if gc_generation_freeze:
            gc.enable()
            gc.unfreeze()


def parse_file(path: str, batch_size: int = 5_000) -> Iterator[list[EntryDetail]]:
    """Adapt the per-record stream into bounded batches for downstream workers.

    The batch list is the ONLY structure that grows, and it is capped at
    batch_size and explicitly cleared each cycle — `del batch` drops the last
    reference so its records are reclaimed before the next batch fills. This
    is where accidental accumulation usually creeps in: keep nothing here that
    outlives one batch.
    """
    with open(path, "r", encoding="ascii", errors="strict") as handle:
        batch: list[EntryDetail] = []
        for record in stream_entry_details(handle):
            batch.append(record)
            if len(batch) >= batch_size:
                yield batch
                batch = []            # rebind, not .clear() — drop the old list
        if batch:
            yield batch

Iterating the file handle is the load-bearing choice: for line in handle pulls one line into a small reusable buffer, so the process never holds more than a line plus the current batch. Rebinding batch = [] rather than mutating a shared list guarantees the yielded batch and the next one never alias, so a downstream worker that holds a batch does not silently pin the parser's next allocation.

Calibration: Batch Size, Throughput, and When to Reach for Polars #

The batch_size knob trades syscall and per-yield overhead against the resident ceiling. Too small (say 100) and the generator hand-off dominates; too large and the batch list itself reintroduces the $O(n)$-ish spike you streamed to avoid. For 94-byte NACHA records, 2,000–10,000 records per batch keeps the batch under a few megabytes while amortizing the per-batch cost to noise — tune it against resident memory under peak worker concurrency, not on a single-file benchmark, because ten workers each holding a 10k batch is the number that has to fit the cgroup.

Reach past a hand-rolled streamer for Polars LazyFrame when the work stops being pure decode-and-forward and becomes column-oriented: a scan_csv/scan_ndjson LazyFrame defers execution and streams the query in morsels, so a group-by, join, or wide projection over a larger-than-memory file runs without materializing it, which a Python-level generator cannot do efficiently. The rule of thumb: if every record is touched exactly once and passed downstream, the generator wins on both simplicity and footprint; the moment you need aggregation or a cross-record join over the whole file, a LazyFrame's streaming engine is the right tool. The crossover benchmarks for the read stage live in optimizing pandas read_fwf for 1 GB NACHA files.

Before and After: A Memory-Profile Walkthrough #

Take the same 1.8 GB return file (about 19.1 million entry-detail records) through both paths, sampling resident set size (RSS) with tracemalloc and resident_set_size from the cgroup:

The before profile is a straight climb: pandas reads all 19.1M rows, boxes routing and trace columns as Python objects, and the RSS curve crosses the 4 GB cgroup ceiling around record 14 million — the process dies before validation runs. The after profile is a flat 78 MB line with a shallow saw-tooth no taller than one 5,000-record batch. Freezing long-lived objects and disabling the cyclic collector removes roughly 1,400 generation-2 pauses that the before run spent rescanning objects that were never going to be collected anyway. Latency to the first usable record drops from 41 seconds (the whole file must be read before read_fwf returns) to microseconds (the first line), which is what lets the downstream stage start matching while the file is still being read.

Failure Modes & Guardrails #

Three mistakes silently convert an $O(1)$ streamer back into an $O(n)$ one — usually without an error, until the OOM kill:

1. Accidental list accumulation. The single most common regression is appending every record to a module-level or enclosing-scope list "just to count them" or to sort at the end. That list is $O(n)$ resident and defeats the entire design. If you need a total, keep a running integer; if you need order, push the sort into a downstream stage that operates on already-bounded batches. In parse_file, batch = [] rebinds so the previous batch is dropped — never replace it with a persistent list the whole file flows into.
2. Holding references in a dict. Caching records in a dict keyed by trace number for a "quick later lookup" pins every value for the file's lifetime, so resident memory grows with the row count even though the generator itself is bounded. If a lookup index is genuinely required, cap it (an LRU with a hard size) or externalize it; an unbounded dict keyed by a high-cardinality field is the same leak as an accumulator list wearing a different shape.
3. GC pause stalls — and the inverse. Leaving the cyclic collector enabled during a multi-million-record hot loop means periodic generation-2 pauses that rescan long-lived objects, adding stalls that can push a large file past a settlement cutoff. But disabling GC without dropping references is dangerous the other way: if your records ever hold a reference cycle (they cannot here — they are frozen and hold only strings and a Decimal), refcounting alone will never reclaim them and RSS grows unbounded. The safe recipe is exactly the one above: freeze long-lived objects, disable the cyclic collector, keep records acyclic, and always re-enable in a finally so an exception mid-file cannot leave the collector off for the rest of the worker's life.

Frequently Asked Questions #

Related guides in this collection #

• High-Volume Pandas Parsing Strategies — the parent guide on keeping pandas memory-bounded with chunked read_fwf, dtype pinning, and streaming generators.
• Optimizing pandas read_fwf for 1 GB NACHA files — buffer sizing, engine selection, and the pandas/Polars crossover for the file-size extreme.
• Async Batch Processing Architectures — how the bounded batches this parser yields are fanned across I/O- and CPU-bound workers.
• Automated File Ingestion & Parsing Pipelines — the parent overview mapping every stage from transport edge to validation gate.