From 2abdd24e2c3b8849916e1cd70428eaec7d98295a Mon Sep 17 00:00:00 2001 From: dirtydishes Date: Sat, 16 May 2026 17:44:51 -0400 Subject: [PATCH] implement durable options tape history --- .codex/hooks.json | 26 ++ ...-16-1711-durable-options-tape-history.html | 363 ++++++++++++++++++ 2 files changed, 389 insertions(+) create mode 100644 .codex/hooks.json create mode 100644 docs/plans/2026-05-16-1711-durable-options-tape-history.html diff --git a/.codex/hooks.json b/.codex/hooks.json new file mode 100644 index 0000000..94fbf97 --- /dev/null +++ b/.codex/hooks.json @@ -0,0 +1,26 @@ +{ + "hooks": { + "PreCompact": [ + { + "matcher": "", + "hooks": [ + { + "type": "command", + "command": "bd prime" + } + ] + } + ], + "SessionStart": [ + { + "matcher": "", + "hooks": [ + { + "type": "command", + "command": "bd prime" + } + ] + } + ] + } +} diff --git a/docs/plans/2026-05-16-1711-durable-options-tape-history.html b/docs/plans/2026-05-16-1711-durable-options-tape-history.html new file mode 100644 index 0000000..997af42 --- /dev/null +++ b/docs/plans/2026-05-16-1711-durable-options-tape-history.html @@ -0,0 +1,363 @@ + + + + + + Plan: Durable Options Tape History + + + +
+
+

Plan Document

+

Durable Options Tape History

+

+ Make the options tape a signal-first live instrument with scroll-gated historical depth: keep the hot cache at + 100 option prints, load older rows from ClickHouse only at the scroll gate, preserve execution context, and + render ClickHouse-backed rows exactly like any other valid flow row. +

+
+ Created 2026-05-16 17:11 + Mode: Plan + Surface: Options Tape +
+
+ +
+

Plan Summary

+

+ Treat stale strictly as feed health, not as historical-row quality. The user should be able to + analyze current live prints and earlier flow in one continuous tape, with no visual distinction between hot-cache + rows and ClickHouse-backed rows. +

+
+ +
+

Goals

+
    +
  • Keep the options tape scrolling infinitely from the user's perspective.
    • +
  • Hold only the 100 newest option prints in the hot live cache.
    • +
  • Use ClickHouse as the durable source for older rows once the scroll gate requests history.
    • +
  • Store all option-print data, including synthetic prints and execution context such as NBBO, spot, and IV.
    • +
  • Surface historical flow as real analyzable flow, not as stale, old, or degraded data.
    • +
  • Keep the default tape view signal-first while exposing all/raw prints from the existing Filter menu.
    • +
+
+ +
+

Proposed Changes

+
    +
  • + Keep islandflow-0sa's useful pieces: scroll-hold behavior, LIVE_OPTIONS_HEAD_LIMIT = 100, + lazy /history/options loading, cache-first scoped snapshots, and preserved execution-context columns. +
    • +
  • + Stop tests and UI copy from asserting that valid rows older than 24 hours are stale when shown as + history. +
    • +
  • + Keep freshness gating only for live fanout/cache admission and channel health, not for historical validity. +
    • +
  • + Remove dead LiveHistoryBuffer and auto-hydration scaffolding if it remains unused after the flow is + explicit. +
    • +
  • + Keep the default options tape view as signal, and add a filter-menu view control with + Signal and All prints. +
    • +
  • + Ensure hot-cache rows and ClickHouse history rows use the same row component, same styling, same sorting, and + same interactions. +
    • +
  • + Keep cursor/key-based deduping so scroll-gated history does not duplicate the 100-row hot head. +
    • +
+
+ +
+

Relevant Context

+
    +
  • + Prior work in islandflow-0sa already introduced scroll hold, a 100-row options head, lazy history, + and cache-first scoped snapshots. +
    • +
  • + The current storage/types path already includes execution context fields such as execution_nbbo_*, + execution_underlying_*, and execution_iv*. +
    • +
  • + Synthetic options prints already emit some execution context; the durable fix should verify this data survives + ClickHouse writes and reads. +
    • +
  • + The UI should prefer preserved execution context in row rendering before falling back to current NBBO lookup. +
    • +
  • + Beads has related work in islandflow-biq for raw live options delivery and filter/backpressure + observability. +
    • +
+
+ +
+

Implementation Steps

+
    +
  • + Audit the existing options tape flow from ingest, ClickHouse write/read, live snapshot, history endpoint, and web + composition. +
    • +
  • + Adjust API/live semantics so valid ClickHouse history can be older than freshness thresholds without being treated + as degraded. +
    • +
  • + Add the Filter-menu view toggle for Signal and All prints, with short copy explaining + the difference. +
    • +
  • + Ensure buildOptionTapeQueryParams, live subscriptions, and /history/options all receive + the selected view consistently. +
    • +
  • + Confirm option row rendering uses preserved execution_nbbo_side, execution_underlying_spot, + and execution_iv when present. +
    • +
  • + Remove deprecated or unused history/autohydration code paths that no longer help the intended scroll-gated flow. +
    • +
  • + Add a deliberate reset path for local and VPS ClickHouse, documented as destructive and operator-confirmed. +
    • +
+
+ +
+

Risks, Limitations, and Mitigations

+
    +
  • + Risk: Resetting VPS data is destructive. Mitigation: make it a runbook or explicit + command with backup/confirmation, never automatic app startup behavior. +
    • +
  • + Risk: The signal/raw toggle could affect both options and flow filters unexpectedly. + Mitigation: test option subscriptions, history query params, and flow packet filtering separately. +
    • +
  • + Risk: Older history fetch latency could be visible at the scroll gate. Mitigation: + keep lazy loading, expose loading/error state if needed, and avoid background auto-hydration. +
    • +
  • + Risk: Prior fixes may have left overlapping history logic. Mitigation: remove unused + scaffolding only after tests cover the intended hot-cache plus ClickHouse path. +
    • +
+
+ +
+

Validation

+
    +
  • + Storage tests: fetchRecentOptionPrints and fetchOptionPrintsBefore return execution NBBO, + spot, IV, signal metadata, and raw/signal filtering correctly. +
    • +
  • + API/live tests: generic options snapshots cap at 100, scoped snapshots prefer hot cache, history preserves + next_before, and rows older than 24 hours return as valid history. +
    • +
  • + Web tests: Filter menu toggles Signal/All prints, scroll gate calls + loadOlder("options"), ClickHouse rows compose with no duplicate seam and no distinct styling, and + preserved execution context drives Spot, Side, Details, and IV display. +
    • +
  • + Validation commands: bun test packages/storage/tests/option-prints.test.ts services/api/tests/live.test.ts apps/web/app/terminal.test.ts + and bun --cwd=apps/web run build. +
    • +
+
+ +
+

Open Questions

+
    +
  • + Exact VPS reset command sequence should be confirmed against the live deployment state before execution. +
    • +
  • + Decide during implementation whether to track the reset/runbook in a new Beads issue or fold it into + islandflow-biq. +
    • +
+
+

+ Fixed assumptions: historical ClickHouse rows should be visually indistinguishable from hot-cache rows, and local + plus VPS wipe should be an operator-confirmed reset path rather than a background migration. +

+
+
+
+ +