Day 69 — Rebuild Logic: Restoring Account State from Disk

Your strategy process is going to die. Not “might” — will. OOM killer, a bad deploy, a Kubernetes node eviction, a SIGKILL from a runaway supervisor script at 3 a.m. during a CPI print. The question Day 69 answers is not “how do I prevent crashes” — you can’t, not fully — it’s “how do I guarantee that when the process comes back up, it knows exactly what it owns, what it owes, and what’s still in flight, with zero ambiguity.” Get this wrong and you don’t get a stack trace. You get a phantom position, a doubled order, or a strategy that sizes a new trade against cash it doesn’t actually have.

The “Dump State as JSON” Trap

Here’s how it usually gets built the first time. A background thread wakes up every 30 seconds, serializes self.positions, self.cash, and self.open_orders into a dict, and writes it to state.json. On boot, you json.load() that file and you’re “restored.” It passes code review. It passes the demo. It runs fine in paper trading for two weeks.

Then it doesn’t.

The Failure Mode

Two independent failure modes converge here, and either one alone is enough to blow up the account:

1. Torn writes. json.dump() is not atomic. It’s a sequence of write() syscalls against a file you opened in truncate mode. If the process is killed between the first and last write() — which is exactly when a supervisor sends SIGKILL after a missed heartbeat during a volatility spike — state.json is now a truncated, unparseable blob. You haven’t lost the last 30 seconds of state. You’ve lost all of it, because there is no previous version to fall back to; you overwrote it in place.

2. Silent drift from the broker. Even when the write succeeds cleanly, a periodic full-state snapshot only captures what your process believed was true at that instant. A fill confirmation that arrived over the WebSocket in the two seconds between your last snapshot and the crash is gone. You reboot, load a state that’s stale by one fill, and your strategy now thinks it’s flat on a symbol it’s actually long 200 shares of. The next signal fires a fresh buy. You are not flat anymore — you’re double long, and nothing in your logs will tell you that until the risk check that should have caught it also reads from the same corrupted local truth.

Both failure modes trace back to one root cause: treating “my last known state” and “the truth” as the same object. They are not. The broker is the truth. Your local state is a cache, and every cache needs an invalidation strategy.

The AutoQuant-Alpha Architecture