Plan Document
+Durable Options Tape History
++ Make the options tape a signal-first live instrument with scroll-gated historical depth: keep the hot cache at + 100 option prints, load older rows from ClickHouse only at the scroll gate, preserve execution context, and + render ClickHouse-backed rows exactly like any other valid flow row. +
+ +Plan Summary
++ Treat stale strictly as feed health, not as historical-row quality. The user should be able to + analyze current live prints and earlier flow in one continuous tape, with no visual distinction between hot-cache + rows and ClickHouse-backed rows. +
+Goals
+-
+
- Keep the options tape scrolling infinitely from the user's perspective. +
- Hold only the 100 newest option prints in the hot live cache. +
- Use ClickHouse as the durable source for older rows once the scroll gate requests history. +
- Store all option-print data, including synthetic prints and execution context such as NBBO, spot, and IV. +
- Surface historical flow as real analyzable flow, not as stale, old, or degraded data. +
- Keep the default tape view signal-first while exposing all/raw prints from the existing Filter menu. +
Proposed Changes
+-
+
-
+ Keep
islandflow-0sa's useful pieces: scroll-hold behavior,LIVE_OPTIONS_HEAD_LIMIT = 100, + lazy/history/optionsloading, cache-first scoped snapshots, and preserved execution-context columns. +
+ - + Stop tests and UI copy from asserting that valid rows older than 24 hours are stale when shown as + history. + +
- + Keep freshness gating only for live fanout/cache admission and channel health, not for historical validity. + +
-
+ Remove dead
LiveHistoryBufferand auto-hydration scaffolding if it remains unused after the flow is + explicit. +
+ -
+ Keep the default options tape view as
signal, and add a filter-menu view control with + Signal and All prints. +
+ - + Ensure hot-cache rows and ClickHouse history rows use the same row component, same styling, same sorting, and + same interactions. + +
- + Keep cursor/key-based deduping so scroll-gated history does not duplicate the 100-row hot head. + +
Relevant Context
+-
+
-
+ Prior work in
islandflow-0saalready introduced scroll hold, a 100-row options head, lazy history, + and cache-first scoped snapshots. +
+ -
+ The current storage/types path already includes execution context fields such as
execution_nbbo_*, +execution_underlying_*, andexecution_iv*. +
+ - + Synthetic options prints already emit some execution context; the durable fix should verify this data survives + ClickHouse writes and reads. + +
- + The UI should prefer preserved execution context in row rendering before falling back to current NBBO lookup. + +
-
+ Beads has related work in
islandflow-biqfor raw live options delivery and filter/backpressure + observability. +
+
Implementation Steps
+-
+
- + Audit the existing options tape flow from ingest, ClickHouse write/read, live snapshot, history endpoint, and web + composition. + +
- + Adjust API/live semantics so valid ClickHouse history can be older than freshness thresholds without being treated + as degraded. + +
-
+ Add the Filter-menu view toggle for
SignalandAll prints, with short copy explaining + the difference. +
+ -
+ Ensure
buildOptionTapeQueryParams, live subscriptions, and/history/optionsall receive + the selected view consistently. +
+ -
+ Confirm option row rendering uses preserved
execution_nbbo_side,execution_underlying_spot, + andexecution_ivwhen present. +
+ - + Remove deprecated or unused history/autohydration code paths that no longer help the intended scroll-gated flow. + +
- + Add a deliberate reset path for local and VPS ClickHouse, documented as destructive and operator-confirmed. + +
Risks, Limitations, and Mitigations
+-
+
- + Risk: Resetting VPS data is destructive. Mitigation: make it a runbook or explicit + command with backup/confirmation, never automatic app startup behavior. + +
- + Risk: The signal/raw toggle could affect both options and flow filters unexpectedly. + Mitigation: test option subscriptions, history query params, and flow packet filtering separately. + +
- + Risk: Older history fetch latency could be visible at the scroll gate. Mitigation: + keep lazy loading, expose loading/error state if needed, and avoid background auto-hydration. + +
- + Risk: Prior fixes may have left overlapping history logic. Mitigation: remove unused + scaffolding only after tests cover the intended hot-cache plus ClickHouse path. + +
Validation
+-
+
-
+ Storage tests:
fetchRecentOptionPrintsandfetchOptionPrintsBeforereturn execution NBBO, + spot, IV, signal metadata, and raw/signal filtering correctly. +
+ -
+ API/live tests: generic options snapshots cap at 100, scoped snapshots prefer hot cache, history preserves
+
next_before, and rows older than 24 hours return as valid history. +
+ -
+ Web tests: Filter menu toggles
Signal/All prints, scroll gate calls +loadOlder("options"), ClickHouse rows compose with no duplicate seam and no distinct styling, and + preserved execution context drives Spot, Side, Details, and IV display. +
+ -
+ Validation commands:
bun test packages/storage/tests/option-prints.test.ts services/api/tests/live.test.ts apps/web/app/terminal.test.ts+ andbun --cwd=apps/web run build. +
+
Open Questions
+-
+
- + Exact VPS reset command sequence should be confirmed against the live deployment state before execution. + +
-
+ Decide during implementation whether to track the reset/runbook in a new Beads issue or fold it into
+
islandflow-biq. +
+
+
+ + Fixed assumptions: historical ClickHouse rows should be visually indistinguishable from hot-cache rows, and local + plus VPS wipe should be an operator-confirmed reset path rather than a background migration. +
+