Day 85 — Total Equity: Syncing Cash + Unrealized Updates
The “Two Ledgers” Trap
By Day 84 your book has everything it needs to compute total equity: a cash balance that moves on every fill, and a mark-to-market monitor (Day 82) that revalues open positions on every tick. The number a risk engine actually needs — total_equity = cash + sum(position_market_values) — looks like a one-liner. So that’s what a junior engineer ships: a CashLedger with get_cash_cents(), an MtmMonitor with get_positions_value_cents(), and a call site that adds them together wherever equity is needed — the leverage check, the drawdown tracker from Day 76, the dashboard.
It passes every unit test. Unit tests call things synchronously, in order, with no scheduler in between. The bug this pattern hides only exists once you’re running an async pipeline — which, as of Day 84, you are.
The Failure Mode
Your fill handler does two things when an execution report arrives: update cash, and update the position’s value at the new price. Somewhere in a production system, one of those steps is followed by an await — writing the fill to a WAL, publishing to a message bus, whatever. That’s a legitimate yield point. It is also a window.
If a concurrently-running coroutine — say, the leverage check that fires on every order request — reads cash_ledger.get_cash_cents() and then mtm_monitor.get_positions_value_cents() as two separate statements, and the fill handler’s yield point lands between your two ledger’s writes, the leverage check computes an equity figure that never corresponded to any real state the book was ever actually in. New cash, paired with stale position value. Or the reverse. It’s not “slightly wrong” — it’s a number with no referent, used to decide whether you’re allowed to take more risk.
This is a torn read, and it’s the concurrent-systems cousin of the floating-point drift you’ve spent thirteen weeks avoiding with integer cents. The bug doesn’t show up in a backtest. It shows up during a volatility spike, when fills and marks are both arriving fast enough that the odds of an unlucky interleaving actually hitting stop being negligible — which is precisely the moment you need your risk numbers to be correct.
The AutoQuant-Alpha Architecture
The fix isn’t a lock. Locks around read paths in an async system are how you turn a data race into a deadlock. The fix is making “mutate state, publish the new total” a single operation that never yields in the middle — so there is no window for a reader to observe.
TotalEquityEngine owns both _cash_cents and every symbol’s _position_value_cents internally — one object, not two. Every method that touches either one — apply_fill, apply_mark — runs synchronously from the first mutation to the moment it constructs and assigns a new, frozen EquitySnapshot. There’s no await inside that span. A reader either gets the snapshot from before the fill, complete and self-consistent, or the snapshot from after — never a hybrid. Consumers only ever call engine.snapshot(), an O(1) read of a cached, immutable object. They can never construct a torn value themselves, because they never touch the two components separately.
That’s the whole architectural idea: collapse the atomicity boundary to match the invariant boundary. total_equity_cents == cash_cents + positions_value_cents needs to hold at every observable instant, so the code that can break that invariant needs to be the same code, in the same non-yielding stretch, that publishes the observable state.
Implementation Deep Dive
Three mechanics carry the lesson:
Atomic mutate-then-publish. apply_fill updates _cash_cents and the fill’s symbol entry in _position_value_cents in the same method body, then calls a private _publish() that bumps a monotonic version counter and constructs the new EquitySnapshot dataclass — frozen, so once built it can’t be partially mutated by anything downstream either. The snapshot assignment (self._snapshot = self._publish(...)) is the last line; nothing after it can observe an in-between state because there is no “after.”
Idempotent replay. Every fill and mark carries a monotonic seq. If a duplicate delivery arrives — a broker retry, a WAL replay after a crash — apply_fill checks seq <= self._last_fill_seq and returns the existing snapshot unchanged instead of double-applying. Combined with the append-only fill WAL (atomic os.replace() for the snapshot cache, fsync‘d append-only JSON-lines for the fill log), a crashed process can restart, replay its WAL through a fresh engine, and land on the exact same state — not an approximation of it.
Oracle reconciliation, kept off the hot path. apply_fill and apply_mark are O(1): they touch two dict entries and a handful of ints. But O(1) incremental state has a well-known failure mode of its own — a bug in the delta math compounds silently instead of failing loudly, because nothing ever recomputes from scratch to catch it. reconcile_oracle() is the brute-force check: replay the full fill log, recompute cash and position value from zero, and diff against the incremental snapshot. It’s O(n), so it runs on a timer or every N fills from a background task — never inline with order flow. A drift over 1 cent flips sync_state to DRIFTED; a drift over $5.00 flips it to HALTED and the engine refuses further fills until someone reconciles manually. The state machine (SYNCED → STALE → DRIFTED → HALTED) is the operator-visible contract for how much you should trust the number on screen.
Production Readiness
Three things to watch once this is live:
sync_statetransitions. Any time spent outsideSYNCEDis time your risk checks are running on possibly-stale or possibly-wrong data. Alert onDRIFTED/HALTEDimmediately; logSTALEdurations.Oracle drift magnitude, in cents, over time. A drift that’s consistently zero is your proof the incremental path is correct. A drift that creeps upward under load is a bug in the delta math, not noise — integer cents don’t accumulate rounding error, so any nonzero drift has a specific, findable cause.
apply_fill/apply_marklatency at the tail. The stress test in this lesson’s workspace shows per-call cost staying flat (~0.7 microseconds) from 1,000 to 80,000 fills, while a naive full-history recompute grows linearly and is already 5,000x slower at 80k fills. If your production latency isn’t flat, something reintroduced an O(n) path into the hot loop.
Step-by-Step Guide
https://github.com/sysdr/quantpython-p/tree/main/day85/day85_workspace
Prerequisites: Python 3.11+, pip install rich pytest.
Execution:
./scripts/start.sh # install deps, sanity-check environment
./scripts/demo.sh # torn-read repro, then the live Rich dashboard
./scripts/verify.sh # full test suite + stress test + success criterion
Verification: verify.sh must print ALL CHECKS PASSED — that means 20/20 tests green, the O(1) scaling claim holds numerically, and a fresh 5,000-event oracle reconciliation reports zero drift with sync_state == SYNCED.
Success Criterion
You may advance to Day 86 only when all of the following are true, verified by running ./scripts/verify.sh from the workspace root:
Full test suite passes: 20/20.
pytest tests/test_engine.py tests/test_persistence.py -qreports20 passed, with zero failures and zero skips. This includes the forced, deterministic torn-read reproduction (test_naive_split_ledger_reproduces_torn_read) and the drift/halt state machine tests.O(1) scaling holds empirically.
tests/stress_test.pymust showapply_fill‘s per-call cost staying within the same order of magnitude across 1,000 → 80,000 fills (no monotonic growth trend), while the naive full-history recompute shows clear linear growth and is at least 1000x slower than the incremental path at 80,000 fills.Zero-drift oracle reconciliation over 5,000 events. A fresh
TotalEquityEnginerun through 5,000 deterministic fill/mark events (seed=99) and then reconciled against the brute-force oracle must reportdrift_cents == 0andsync_state == "SYNCED". This is the concrete, measurable version of “cash and unrealized P&L are in sync”: the incremental accumulator and the from-scratch replay agree to the cent.The invariant holds on every snapshot you inspect manually. Pick any
EquitySnapshotprinted by the dashboard (./scripts/demo.sh) and confirm by hand:total_equity_cents == cash_cents + positions_value_cents. There is no snapshot, at any version, where this does not hold — because it’s enforced by construction in_publish(), not checked after the fact.
./scripts/verify.sh exits 0 and prints [verify] ALL CHECKS PASSED. That single line, with a zero exit code, is the pass/fail gate for Day 85.
Homework
Extend TotalEquityEngine to handle a cash dividend event (DividendEvent(symbol, cents_per_share, ts)) that credits cash based on the current held quantity at the ex-date — without ever letting a reader observe cash credited for shares that were sold before the dividend posted. Write the oracle check for it before you write the incremental path: define what “correct” means in the brute-force replay first, then make the O(1) version agree with it, exactly as this lesson did for fills and marks.