Plan Document

Durable Options Tape History

Make the options tape a signal-first live instrument with scroll-gated historical depth: keep the hot cache at 100 option prints, load older rows from ClickHouse only at the scroll gate, preserve execution context, and render ClickHouse-backed rows exactly like any other valid flow row.

Created 2026-05-16 17:11 Mode: Plan Surface: Options Tape

Plan Summary

Treat stale strictly as feed health, not as historical-row quality. The user should be able to analyze current live prints and earlier flow in one continuous tape, with no visual distinction between hot-cache rows and ClickHouse-backed rows.

Goals

Keep the options tape scrolling infinitely from the user's perspective.
Hold only the 100 newest option prints in the hot live cache.
Use ClickHouse as the durable source for older rows once the scroll gate requests history.
Store all option-print data, including synthetic prints and execution context such as NBBO, spot, and IV.
Surface historical flow as real analyzable flow, not as stale, old, or degraded data.
Keep the default tape view signal-first while exposing all/raw prints from the existing Filter menu.

Proposed Changes

Keep islandflow-0sa's useful pieces: scroll-hold behavior, LIVE_OPTIONS_HEAD_LIMIT = 100, lazy /history/options loading, cache-first scoped snapshots, and preserved execution-context columns.
Stop tests and UI copy from asserting that valid rows older than 24 hours are stale when shown as history.
Keep freshness gating only for live fanout/cache admission and channel health, not for historical validity.
Remove dead LiveHistoryBuffer and auto-hydration scaffolding if it remains unused after the flow is explicit.
Keep the default options tape view as signal, and add a filter-menu view control with Signal and All prints.
Ensure hot-cache rows and ClickHouse history rows use the same row component, same styling, same sorting, and same interactions.
Keep cursor/key-based deduping so scroll-gated history does not duplicate the 100-row hot head.

Relevant Context

Prior work in islandflow-0sa already introduced scroll hold, a 100-row options head, lazy history, and cache-first scoped snapshots.
The current storage/types path already includes execution context fields such as execution_nbbo_*, execution_underlying_*, and execution_iv*.
Synthetic options prints already emit some execution context; the durable fix should verify this data survives ClickHouse writes and reads.
The UI should prefer preserved execution context in row rendering before falling back to current NBBO lookup.
Beads has related work in islandflow-biq for raw live options delivery and filter/backpressure observability.

Implementation Steps

Audit the existing options tape flow from ingest, ClickHouse write/read, live snapshot, history endpoint, and web composition.
Adjust API/live semantics so valid ClickHouse history can be older than freshness thresholds without being treated as degraded.
Add the Filter-menu view toggle for Signal and All prints, with short copy explaining the difference.
Ensure buildOptionTapeQueryParams, live subscriptions, and /history/options all receive the selected view consistently.
Confirm option row rendering uses preserved execution_nbbo_side, execution_underlying_spot, and execution_iv when present.
Remove deprecated or unused history/autohydration code paths that no longer help the intended scroll-gated flow.
Add a deliberate reset path for local and VPS ClickHouse, documented as destructive and operator-confirmed.

Risks, Limitations, and Mitigations

Risk: Resetting VPS data is destructive. Mitigation: make it a runbook or explicit command with backup/confirmation, never automatic app startup behavior.
Risk: The signal/raw toggle could affect both options and flow filters unexpectedly. Mitigation: test option subscriptions, history query params, and flow packet filtering separately.
Risk: Older history fetch latency could be visible at the scroll gate. Mitigation: keep lazy loading, expose loading/error state if needed, and avoid background auto-hydration.
Risk: Prior fixes may have left overlapping history logic. Mitigation: remove unused scaffolding only after tests cover the intended hot-cache plus ClickHouse path.

Validation

Storage tests: fetchRecentOptionPrints and fetchOptionPrintsBefore return execution NBBO, spot, IV, signal metadata, and raw/signal filtering correctly.
API/live tests: generic options snapshots cap at 100, scoped snapshots prefer hot cache, history preserves next_before, and rows older than 24 hours return as valid history.
Web tests: Filter menu toggles Signal/All prints, scroll gate calls loadOlder("options"), ClickHouse rows compose with no duplicate seam and no distinct styling, and preserved execution context drives Spot, Side, Details, and IV display.
Validation commands: bun test packages/storage/tests/option-prints.test.ts services/api/tests/live.test.ts apps/web/app/terminal.test.ts and bun --cwd=apps/web run build.

Open Questions

Exact VPS reset command sequence should be confirmed against the live deployment state before execution.
Decide during implementation whether to track the reset/runbook in a new Beads issue or fold it into islandflow-biq.

Fixed assumptions: historical ClickHouse rows should be visually indistinguishable from hot-cache rows, and local plus VPS wipe should be an operator-confirmed reset path rather than a background migration.