Day 68 — Atomic IO: Ensuring File Integrity During High Volume

The “Just Open and Write” Trap

Every junior engineer eventually writes this line inside a fill-handler:

with open("data/positions.json", "w") as f:
    json.dump(positions, f)

It works in the backtest. It works in the demo. It works for three weeks in paper trading. Then, during a CPI print at 8:30 AM when your WebSocket is pushing 40 fills a second, the process gets OOM-killed by the OS scheduler mid-json.dump, or a SIGTERM lands between the open() and the final flush(). positions.json is now half a JSON object — a truncated file with a dangling brace. Your risk manager restarts, tries to load state, hits a JSONDecodeError, and either crashes on boot (best case) or — if someone wrapped that load in a bare except: positions = {} (worst case) — silently believes you are flat when you are short 400 shares of a stock that just gapped down. That is not a bug. That is account ruin, and it is self-inflicted by a write pattern, not by the market.

The root cause is that open(path, "w") truncates the file in place. The instant the file descriptor is opened for writing, the old, valid contents are gone. From that moment until the final byte is flushed and the descriptor is closed, the file on disk is in an undefined, partially-written state. Any crash, kill signal, power event, or even a concurrent reader opening the same path with the wrong timing observes garbage. In a backtest this never happens because nothing else touches the file and nothing kills the process mid-loop. In production, at scale, with a process supervisor that can SIGKILL you on a health-check timeout, it happens constantly — and it happens exactly when you can least afford it, because crashes cluster around volatility spikes and resource contention, which is precisely when your state file matters most.

The Failure Mode, Precisely

There are three concrete ways in-place writes corrupt state under load:

1. Truncate-then-crash: open(path, "w") truncates immediately. If the writer dies before write() completes, you’re left with a zero-byte or partial file. Recovery code that doesn’t distinguish “file doesn’t exist” from “file exists but is corrupt” will happily load {} and proceed with no memory of your open positions.

2. Torn reads: even without a crash, if another thread or process opens the same path for reading while the writer is mid-write, it can read a file that is neither the old version nor the new version — just whatever bytes have hit the page cache so far. Pandas’ read_json will throw on this most of the time, but roughly 4-5% of torn reads land on syntactically valid-but-wrong JSON (a truncation that happens to close all braces early), which is far more dangerous than a clean crash.

3. Non-atomic multi-file consistency: if your trade log and your position snapshot are two separate files written independently, a crash between the two writes leaves them inconsistent with each other, even if each file individually is intact. Your reconciliation job (Day 70) will disagree with itself.

None of this shows up in single-threaded, no-interrupt backtests. It shows up under concurrent access and unreliable process lifetimes — which is the actual production environment.

The AutoQuant-Alpha Architecture: Write-Rename, Not Write-Truncate