Implementing SFTP with PGP for ACH Files #

At 05:30 your originating bank drops a PGP-encrypted .ach.pgp file onto its SFTP endpoint, and your ingestion service has a 90-minute window to retrieve it, decrypt it, and hand a clean 94-byte NACHA stream to the parser before the settlement cutoff. Two failures dominate this window and both are silent: an unpinned host key lets a spoofed endpoint feed you a forged batch, and a decrypt step that "helpfully" normalizes line endings shifts every fixed-width field one byte and floods the exception queue with breaks that do not exist. This page isolates one surgical task — automating the SFTP retrieval and OpenPGP decryption of NACHA ACH files with byte-exact fidelity and an audit trail — as a concrete implementation of the secure file transfer protocols for banks contract, within the broader Core Architecture & Payment File Standards framework that governs how this institution ingests, validates, and audits payment files at scale.

The reason this deserves its own guide is that PGP-over-SFTP sits at the exact boundary where a cryptographic mistake becomes a reconciliation mistake. The transport (SSH) proves the bytes arrived intact; the payload envelope (OpenPGP) proves who sent them and that nobody read them in transit — but neither guarantees the plaintext handed downstream is byte-identical to what the originator signed. Get the decrypt encoding wrong and the corrupted record still passes transfer integrity, still parses far enough to post, and surfaces days later as an unexplained break in the transaction matching and reconciliation engine or, worse, as a control-total mismatch an examiner traces during a Reg E dispute.

Concept Spec: What the Pipeline Must Guarantee #

The retrieval runs three ordered stages, each enforcing exactly one invariant:

1. Authenticated transport. SFTP runs a single encrypted channel over SSH (RFC 4253) on TCP port 22. The invariant is endpoint identity: the client must reject any server whose host key fingerprint does not match a pre-pinned value, or a man-in-the-middle can serve a counterfeit file. Fingerprint comparison reads a fixed-length key, so it is constant-time in file size, $O(1)$.
2. Confidential, attributable payload. The file is an OpenPGP message (RFC 4880): the NACHA plaintext is compressed, symmetrically encrypted with a one-time session key, and that session key is itself encrypted to the recipient's public key. Decryption is a single pass over the ciphertext, $O(n)$ in file length, and the invariant is attribution — every successful decrypt records the exact key fingerprint that unlocked it.
3. Byte-exact plaintext. NACHA files are fixed-width: every record is exactly 94 bytes and lines are terminated by \n or \r\n with nothing in between. The invariant is width — the decrypted bytes must be written to disk with zero newline translation, encoding coercion, or BOM insertion, or the downstream fixed-width slice reflows and every field after the mutation point is wrong.

The subtle failure mode is that pgpy returns a decrypted message as either bytes or a Python str, and the naive str(msg) path invites an implicit UTF-8 round-trip that mangles high bytes in name and addenda fields. The correct handling maps every byte one-to-one, which is why the implementation below reaches for latin-1 (a total 0–255 mapping) only as the string-to-bytes bridge, never as a semantic decode.

Full Annotated Python Implementation #

The pipeline uses paramiko for SFTP transport and pgpy for pure-Python OpenPGP decryption. Private keys never live in the process environment — they load from an encrypted keyring or an HSM-backed file, and the passphrase is injected at runtime from a secret manager, never baked into a container image or CI/CD variable. Context managers guarantee the SSH connection and SFTP channel tear down even when decryption raises, and every path — success or failure — writes a structured audit record.

import hashlib
import logging
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Dict

import paramiko
import pgpy

logger = logging.getLogger("ach_ingestion_pipeline")


class ACHSecureIngestor:
    """Retrieve and decrypt PGP-encrypted NACHA ACH files over SFTP.

    Pins the SFTP host key by fingerprint, decrypts with a runtime-supplied
    passphrase, and preserves the plaintext byte-for-byte so downstream
    fixed-width parsing and control-total checks stay valid.
    """

    def __init__(
        self,
        sftp_host: str,
        sftp_port: int,
        sftp_user: str,
        sftp_key_path: Path,
        pgp_key_path: Path,
        pgp_passphrase: str,
        host_key_fingerprint: str,  # hex, pre-pinned out of band from the originator
    ) -> None:
        self.sftp_config: Dict[str, Any] = {
            "hostname": sftp_host,
            "port": sftp_port,
            "username": sftp_user,
            "key_filename": str(sftp_key_path),
        }
        self.pgp_key_path = pgp_key_path
        self.pgp_passphrase = pgp_passphrase
        self.expected_host_key = host_key_fingerprint.lower()
        self.pgp_key, _ = pgpy.PGPKey.from_file(str(self.pgp_key_path))

    @staticmethod
    def _sha256(data: bytes) -> str:
        return hashlib.sha256(data).hexdigest()

    def _establish_sftp(self) -> paramiko.SFTPClient:
        """Open an SFTP channel only after pinning the server host key."""
        client = paramiko.SSHClient()
        # RejectPolicy: never auto-trust an unknown key. We verify explicitly below.
        client.set_missing_host_key_policy(paramiko.RejectPolicy())
        client.load_system_host_keys()
        try:
            client.connect(**self.sftp_config, timeout=30, banner_timeout=30)
            transport = client.get_transport()
            if transport is None:
                raise ConnectionError("SSH transport failed to initialize.")

            # Strict host-key fingerprint check — the anti-MITM guarantee.
            actual_fp = transport.get_remote_server_key().get_fingerprint().hex()
            if actual_fp != self.expected_host_key:
                raise ConnectionError(
                    f"Host key mismatch: expected {self.expected_host_key}, "
                    f"got {actual_fp}"
                )
            return client.open_sftp()
        except Exception:
            client.close()
            raise

    def _decrypt(self, ciphertext: bytes) -> bytes:
        """Decrypt an OpenPGP blob to raw NACHA bytes, preserving width.

        pgpy may hand back the message body as str OR bytes. latin-1 is the
        one encoding that maps chr(0)..chr(255) to bytes 0..255 one-to-one,
        so it bridges str -> bytes with zero data loss and never raises.
        """
        message = pgpy.PGPMessage.from_blob(ciphertext)
        with self.pgp_key.unlock(self.pgp_passphrase):
            decrypted = self.pgp_key.decrypt(message)
        body = decrypted.message
        return body.encode("latin-1") if isinstance(body, str) else bytes(body)

    def retrieve_and_decrypt(self, remote_path: str, local_dir: Path) -> Dict[str, Any]:
        """Download, verify size, decrypt, and persist one ACH file.

        Returns an audit record whether the pipeline succeeds or fails. The
        encrypted intermediate is always shredded from local disk in finally.
        """
        local_dir.mkdir(parents=True, exist_ok=True)
        encrypted_path = local_dir / Path(remote_path).name
        decrypted_path = local_dir / f"{encrypted_path.stem}.nacha"

        audit: Dict[str, Any] = {
            "received_at": datetime.now(timezone.utc).isoformat(),
            "remote_path": remote_path,
            "status": "pending",
        }

        sftp = self._establish_sftp()
        try:
            # 1. Verify the advertised size before download to catch truncation.
            expected_size = sftp.stat(remote_path).st_size
            sftp.get(remote_path, str(encrypted_path))
            ciphertext = encrypted_path.read_bytes()
            if len(ciphertext) != expected_size:
                raise IOError(
                    f"Short read: expected {expected_size} bytes, "
                    f"got {len(ciphertext)}"
                )
            audit["encrypted_sha256"] = self._sha256(ciphertext)

            # 2. Decrypt with byte-exact fidelity.
            plaintext = self._decrypt(ciphertext)

            # 3. Persist raw bytes — write_bytes() does NO newline translation.
            decrypted_path.write_bytes(plaintext)

            audit.update(
                status="success",
                key_fingerprint=str(self.pgp_key.fingerprint),
                decrypted_sha256=self._sha256(plaintext),
                record_count=len(plaintext) // 94,  # 94-byte NACHA records
                decrypted_path=str(decrypted_path),
            )
            logger.info("Decrypted %s (%d records)", remote_path, audit["record_count"])
            return audit

        except pgpy.errors.PGPError as exc:
            audit.update(status="decryption_failed", error=str(exc))
            logger.error("PGP decryption failed for %s: %s", remote_path, exc)
            raise
        except Exception as exc:
            audit.update(status="pipeline_failed", error=str(exc))
            logger.error("Ingestion pipeline error for %s: %s", remote_path, exc)
            raise
        finally:
            sftp.close()
            if encrypted_path.exists():
                encrypted_path.unlink()  # never leave ciphertext on local disk

Three details carry disproportionate weight. RejectPolicy() plus the explicit fingerprint comparison is the only thing standing between your parser and a spoofed batch — AutoAddPolicy is the single most common production footgun here and silently disables the guarantee. The sftp.stat().st_size check before decryption catches a truncated transfer while the payload is still opaque ciphertext, so a partial download never reaches the OpenPGP layer as a confusing "bad packet" error. And write_bytes() on a bytes object is binary mode by definition — it is the affirmative choice that keeps the 94-byte grid intact, exactly the positional contract the NACHA record layouts parser depends on.

Calibration & Configuration #

The pipeline's behavior shifts by corridor, and three settings are worth tuning per counterparty:

• Timeouts and transfer windows. For domestic ACH origination files (typically a few MB), the default 30-second timeout and banner_timeout are ample. For large consolidated files or slow correspondent gateways, raise banner_timeout first — a hung SSH banner is the usual cause of a stall, not the data channel. Coordinate the retrieval schedule with the originator's drop window rather than polling aggressively; a fixed poll after the contracted drop time keeps the audit timeline deterministic.
• Host-key pinning source. Pin host_key_fingerprint from a value exchanged out of band (a signed onboarding document), not from the first connection. Rotate it deliberately during the originator's documented key-rotation window, and keep the previous fingerprint valid for an overlap period so a mid-rotation drop does not hard-fail.
• Passphrase and key custody. For high-value wire corridors, back the private key with an HSM and inject the passphrase from a secret manager at call time. For lower-value bulk ACH, an encrypted keyring is acceptable, but the one non-negotiable is that pgp_passphrase never appears in an image layer, environment variable dump, or log line. Note the audit record deliberately logs the key fingerprint, never the passphrase.

Do not "fix" a decrypt problem by loosening the encoding — if latin-1 output looks wrong downstream, the file is genuinely EBCDIC or otherwise mis-encoded, and that is a content problem handled by encoding-drift detection after decryption, not by mutating bytes here.

Validation Example: Before and After #

Consider one PGP-encrypted ACH file whose plaintext Entry Detail record ends with a Windows \r\n terminator and carries a hyphenated payee name that includes a CP1252 en-dash byte (0x96):

6220210000210001234567        0000125000SMITH-JONES PAYROLL     0091000010000001\r\n

Before (decrypted.message coerced through str(msg) / UTF-8): the en-dash byte fails a strict UTF-8 decode or is replaced with U+FFFD, and the implicit round-trip re-emits the \r\n as a normalized \n on some platforms. The record is now 93 or 95 bytes instead of 94: the trace-number slice [79:94] reads shifted, the amount field misaligns, the batch control total fails, and the entry lands in the exception queue as a phantom break.

After (_decrypt with the latin-1 bridge and write_bytes()): every byte, including 0x96 and both \r and \n, survives one-to-one. The record stays exactly 94 payload bytes, record_count computes as len(plaintext) // 94 with no remainder, amount slices as 0000125000 → integer cents 125000 (widened to Decimal("1250.00") only at the aggregation edge, never a float), and the audit record captures encrypted_sha256, decrypted_sha256, and the exact key_fingerprint that unlocked the payload. Zero false breaks, and a hash chain an examiner can replay.

Failure Modes & Guardrails #

Three edge cases silently corrupt an ACH ingestion pipeline if left unguarded:

1. Auto-trusting the host key. Swapping RejectPolicy() for AutoAddPolicy() — a common "make it connect" shortcut — removes the anti-spoofing guarantee entirely: the client now accepts whatever key answers on port 22, and a hijacked DNS entry or rogue endpoint can feed forged batches that decrypt cleanly if the attacker also holds your public key. Keep RejectPolicy and pin the fingerprint; treat a mismatch as a hard security incident, not a config nag.
2. Newline normalization on write. Any path that opens the output in text mode (open(path, "w")) or pipes the plaintext through a middleware that "cleans" line endings will convert \r\n to \n, shortening records by one byte each and reflowing every subsequent slice — with no exception raised. Guard it by asserting len(plaintext) % 94 == 0 before handing off, so a width violation fails loud at the ingestion edge instead of silently downstream.
3. Leaving decrypted plaintext or ciphertext on disk. The finally block shreds the encrypted intermediate, but the decrypted .nacha file contains cleartext account and routing numbers subject to FFIEC and Regulation E retention rules. Write it to an access-controlled, encrypted volume and delete it on a defined schedule once ingested — never a world-readable temp directory. The decryption timestamp recorded in UTC ISO 8601 is what anchors the 10-day consumer notification window in a Reg E dispute, so it must reflect verified receipt, not an arbitrary polling tick.

Frequently Asked Questions #

Related #

• Secure File Transfer Protocols for Banks — the parent guide comparing SFTP, FTPS, and AS2 transport contracts and where PGP payload encryption fits.
• How to Validate NACHA Batch Headers Programmatically — the byte-level validation gate the decrypted 94-byte stream feeds into next.
• Handling Encoding Drift in Legacy Bank Exports — what to do when the decrypted plaintext turns out to be EBCDIC or a mixed codepage rather than clean ASCII.
• Validating NACHA Addenda Records with Pydantic — the strict schema stage that consumes the byte-exact records this pipeline produces.