Plan Document

# Durable Options Tape History

Make the options tape a signal-first live instrument with scroll-gated historical depth: keep the hot cache at 100 option prints, load older rows from ClickHouse only at the scroll gate, preserve execution context, and render ClickHouse-backed rows exactly like any other valid flow row.

Created 2026-05-16 17:11 Mode: Plan Surface: Options Tape

## Plan Summary

Treat stale strictly as feed health, not as historical-row quality. The user should be able to analyze current live prints and earlier flow in one continuous tape, with no visual distinction between hot-cache rows and ClickHouse-backed rows.

## Goals

- Keep the options tape scrolling infinitely from the user's perspective.
- Hold only the 100 newest option prints in the hot live cache.
- Use ClickHouse as the durable source for older rows once the scroll gate requests history.
- Store all option-print data, including synthetic prints and execution context such as NBBO, spot, and IV.
- Surface historical flow as real analyzable flow, not as stale, old, or degraded data.
- Keep the default tape view signal-first while exposing all/raw prints from the existing Filter menu.

## Proposed Changes

- Keep `islandflow-0sa`'s useful pieces: scroll-hold behavior, `LIVE_OPTIONS_HEAD_LIMIT = 100`, lazy `/history/options` loading, cache-first scoped snapshots, and preserved execution-context columns.
- Stop tests and UI copy from asserting that valid rows older than 24 hours are stale when shown as history.
- Keep freshness gating only for live fanout/cache admission and channel health, not for historical validity.
- Remove dead `LiveHistoryBuffer` and auto-hydration scaffolding if it remains unused after the flow is explicit.
- Keep the default options tape view as `signal`, and add a filter-menu view control with Signal and All prints.
- Ensure hot-cache rows and ClickHouse history rows use the same row component, same styling, same sorting, and same interactions.
- Keep cursor/key-based deduping so scroll-gated history does not duplicate the 100-row hot head.

## Relevant Context

- Prior work in `islandflow-0sa` already introduced scroll hold, a 100-row options head, lazy history, and cache-first scoped snapshots.
- The current storage/types path already includes execution context fields such as `execution_nbbo_*`, `execution_underlying_*`, and `execution_iv*`.
- Synthetic options prints already emit some execution context; the durable fix should verify this data survives ClickHouse writes and reads.
- The UI should prefer preserved execution context in row rendering before falling back to current NBBO lookup.
- Beads has related work in `islandflow-biq` for raw live options delivery and filter/backpressure observability.

## Implementation Steps

- Audit the existing options tape flow from ingest, ClickHouse write/read, live snapshot, history endpoint, and web composition.
- Adjust API/live semantics so valid ClickHouse history can be older than freshness thresholds without being treated as degraded.
- Add the Filter-menu view toggle for `Signal` and `All prints`, with short copy explaining the difference.
- Ensure `buildOptionTapeQueryParams`, live subscriptions, and `/history/options` all receive the selected view consistently.
- Confirm option row rendering uses preserved `execution_nbbo_side`, `execution_underlying_spot`, and `execution_iv` when present.
- Remove deprecated or unused history/autohydration code paths that no longer help the intended scroll-gated flow.
- Add a deliberate reset path for local and VPS ClickHouse, documented as destructive and operator-confirmed.

## Risks, Limitations, and Mitigations

- Risk: Resetting VPS data is destructive. Mitigation: make it a runbook or explicit command with backup/confirmation, never automatic app startup behavior.
- Risk: The signal/raw toggle could affect both options and flow filters unexpectedly. Mitigation: test option subscriptions, history query params, and flow packet filtering separately.
- Risk: Older history fetch latency could be visible at the scroll gate. Mitigation: keep lazy loading, expose loading/error state if needed, and avoid background auto-hydration.
- Risk: Prior fixes may have left overlapping history logic. Mitigation: remove unused scaffolding only after tests cover the intended hot-cache plus ClickHouse path.

## Validation

- Storage tests: `fetchRecentOptionPrints` and `fetchOptionPrintsBefore` return execution NBBO, spot, IV, signal metadata, and raw/signal filtering correctly.
- API/live tests: generic options snapshots cap at 100, scoped snapshots prefer hot cache, history preserves `next_before`, and rows older than 24 hours return as valid history.
- Web tests: Filter menu toggles `Signal`/`All prints`, scroll gate calls `loadOlder("options")`, ClickHouse rows compose with no duplicate seam and no distinct styling, and preserved execution context drives Spot, Side, Details, and IV display.
- Validation commands: `bun test packages/storage/tests/option-prints.test.ts services/api/tests/live.test.ts apps/web/app/terminal.test.ts` and `bun --cwd=apps/web run build`.

## Open Questions

- Exact VPS reset command sequence should be confirmed against the live deployment state before execution.
- Decide during implementation whether to track the reset/runbook in a new Beads issue or fold it into `islandflow-biq`.

Fixed assumptions: historical ClickHouse rows should be visually indistinguishable from hot-cache rows, and local plus VPS wipe should be an operator-confirmed reset path rather than a background migration.