From eaddf4b7a0e995ca9758579b88e10aa3d33afc4f Mon Sep 17 00:00:00 2001 From: dirtydishes Date: Sat, 16 May 2026 14:14:56 -0400 Subject: [PATCH 1/5] Update AGENTS.md --- .beads/issues.jsonl | 1 + AGENTS.md | 1 + 2 files changed, 2 insertions(+) diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index bbea524..d2acc2b 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -10,6 +10,7 @@ {"_type":"issue","id":"islandflow-ayo","title":"Drop stale backlog events from live fanout","description":"Follow-up to live freshness rollout: /ws/live was still fanning out stale backlog events for freshness-gated channels, which kept tape panes in Live feed behind despite active synthetic ingest. Gate fanout and cache ingest by freshness for options/nbbo/equities/flow.","status":"closed","priority":1,"issue_type":"bug","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T21:26:39Z","created_by":"dirtydishes","updated_at":"2026-04-28T21:26:44Z","started_at":"2026-04-28T21:26:44Z","closed_at":"2026-04-28T21:26:44Z","close_reason":"Completed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-0v6","title":"Fix tape freshness, NBBO coverage, pause controls, and filter popup","description":"Implement the tape fixes requested for synthetic options notional sizing, strict live freshness, live-mode pause/resume behavior, stronger NBBO snapshot coverage, and moving flow filters behind a popup. Includes server-side live cache changes, web terminal state/UI changes, and tests for synthetic pricing, live snapshot freshness/NBBO retention, and live pause/filter interactions.","status":"closed","priority":1,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T21:02:52Z","created_by":"dirtydishes","updated_at":"2026-04-28T21:13:38Z","started_at":"2026-04-28T21:02:57Z","closed_at":"2026-04-28T21:13:38Z","close_reason":"Completed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-e4r","title":"Implement smart-money flow filtering and synthetic firehose modes","description":"Implement the approved multi-surface plan for named synthetic market profiles, options raw-vs-signal filtering, live/API filter contracts, Tape page client-side flow filters, firehose-readiness improvements, tests, and README updates.","status":"closed","priority":1,"issue_type":"feature","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T20:10:49Z","created_by":"dirtydishes","updated_at":"2026-04-28T20:29:29Z","started_at":"2026-04-28T20:10:53Z","closed_at":"2026-04-28T20:29:29Z","close_reason":"Implemented synthetic market profiles, options signal-path filtering, signal-aware API/replay contracts, Tape page filters, tests, and README updates. Follow-up tracked in islandflow-biq.","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"islandflow-0sa","title":"Fix live tape auto-hold, history seam, and remove manual pause control","description":"The live tape should automatically hold when the user scrolls away from the top, resume when they return to the top or use Jump to top, and keep older prints available seamlessly beyond the hot window. Manual Pause/Resume control is now redundant and should be removed from live tape panes. This work should also fix the current regression where paused/held tapes still mutate, and align the options tape with a strict 100-row hot head backed by ClickHouse history.","status":"in_progress","priority":2,"issue_type":"bug","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-16T18:12:51Z","created_by":"dirtydishes","updated_at":"2026-05-16T18:12:54Z","started_at":"2026-05-16T18:12:54Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-2db","title":"Manually remove stale islandflow local-infra containers from VPS","description":"The live VPS still has an older compose project named islandflow created from the repo-root docker-compose.yml. Inspection shows it is separate from the supported islandflow-vps deployment stack and exposes NATS, ClickHouse, and Redis on host ports. Container removal commands currently hang when run as the delta user through Docker, so cleanup likely needs a focused maintenance window and possibly host-level intervention or a Docker daemon restart.","notes":"The duplicate islandflow compose project on the VPS was confirmed live during inspection. Nginx Proxy Manager routes public traffic only to islandflow-vps web/api by Docker name, so the stale islandflow project appears to be stray local-infra state rather than part of the supported production path. Attempts to remove the stale containers with docker compose down and docker rm -f as the delta user hung and timed out, so manual cleanup likely needs a maintenance window and possibly Docker daemon intervention.","status":"open","priority":2,"issue_type":"task","owner":"dishes@dpdrm.com","created_at":"2026-05-16T01:27:27Z","created_by":"dirtydishes","updated_at":"2026-05-16T01:28:59Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-c87","title":"Clean up duplicate Islandflow Docker infra on VPS","description":"The live VPS is currently running both the production-style islandflow-vps Docker stack and an older root-level islandflow infra stack that publishes NATS, ClickHouse, and Redis on host ports. Investigate whether the older stack is unused, remove it safely if so, and update docs/deploy guidance so the server topology is clearer.","notes":"Inspected the live VPS and confirmed the duplicate compose project: islandflow-vps is the supported deployment stack, while a separate islandflow project from the repo-root docker-compose.yml still runs exposed NATS/ClickHouse/Redis containers. Verified Nginx Proxy Manager routes only to islandflow-vps web/api by Docker name. Attempted cleanup via docker compose down and docker rm -f on the stale islandflow containers, but those commands hung for the delta user and timed out. Added repo guardrails and docs so deploy warns when the duplicate project exists, and opened islandflow-2db for manual host-level cleanup during a maintenance window.","status":"closed","priority":2,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-16T01:16:05Z","created_by":"dirtydishes","updated_at":"2026-05-16T01:28:07Z","started_at":"2026-05-16T01:16:09Z","closed_at":"2026-05-16T01:28:07Z","close_reason":"Completed the repo-side investigation and guardrails. Actual server-side container removal is blocked by hanging Docker operations and is tracked separately in islandflow-2db for a maintenance window.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-4gj","title":"Clarify Docker-first deploy workflow and mark native runtime experimental","description":"After inspecting the live VPS, native deployment is not ready for routine use: Nginx Proxy Manager routes to Docker container names, Bun is not installed on the host, sudo systemctl is not passwordless, and no Islandflow units exist. Update deploy messaging and docs so Docker remains the clearly recommended deployment path and native runtime is labeled experimental/future-facing with server prerequisites called out.","notes":"Updated deploy messaging and docs after live VPS inspection. scripts/deploy.ts now marks Docker as the default and recommended runtime, labels native as experimental, switches native systemctl default to sudo -n systemctl, and prints explicit native precheck failures for missing Bun/systemctl access/units. Updated README.md, deployment/docker/README.md, and deployment/native/README.md to reflect the current Docker + Nginx Proxy Manager topology. Validation: ./deploy --help, ./deploy main --runtime native --no-build (fails fast with Bun-missing message), bun run check:docker-workspace.","status":"closed","priority":2,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-16T01:10:11Z","created_by":"dirtydishes","updated_at":"2026-05-16T01:12:39Z","started_at":"2026-05-16T01:10:14Z","closed_at":"2026-05-16T01:12:39Z","close_reason":"Closed","dependency_count":0,"dependent_count":0,"comment_count":0} diff --git a/AGENTS.md b/AGENTS.md index 351b68c..b5c7b69 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -117,6 +117,7 @@ Each turn document must include these sections: 2. **Changes Made** 3. **Context** 4. **Important Implementation Details** +5. **Impact for End-Users** 5. **Validation** 6. **Issues, Limitations, and Mitigations** 7. **Follow-up Work** From 39fb5ce9f104ba59e249aefedc24b9e711168e03 Mon Sep 17 00:00:00 2001 From: dirtydishes Date: Sat, 16 May 2026 14:23:51 -0400 Subject: [PATCH 2/5] Fix live tape scroll hold and lazy history --- .beads/issues.jsonl | 2 +- apps/web/app/terminal.test.ts | 27 +-- apps/web/app/terminal.tsx | 87 ++++------ ...6-05-16-live-tape-scroll-hold-history.html | 158 ++++++++++++++++++ services/api/src/live.ts | 82 +++++++-- services/api/tests/live.test.ts | 51 ++++++ 6 files changed, 332 insertions(+), 75 deletions(-) create mode 100644 docs/turns/2026-05-16-live-tape-scroll-hold-history.html diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index d2acc2b..065a612 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -10,7 +10,7 @@ {"_type":"issue","id":"islandflow-ayo","title":"Drop stale backlog events from live fanout","description":"Follow-up to live freshness rollout: /ws/live was still fanning out stale backlog events for freshness-gated channels, which kept tape panes in Live feed behind despite active synthetic ingest. Gate fanout and cache ingest by freshness for options/nbbo/equities/flow.","status":"closed","priority":1,"issue_type":"bug","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T21:26:39Z","created_by":"dirtydishes","updated_at":"2026-04-28T21:26:44Z","started_at":"2026-04-28T21:26:44Z","closed_at":"2026-04-28T21:26:44Z","close_reason":"Completed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-0v6","title":"Fix tape freshness, NBBO coverage, pause controls, and filter popup","description":"Implement the tape fixes requested for synthetic options notional sizing, strict live freshness, live-mode pause/resume behavior, stronger NBBO snapshot coverage, and moving flow filters behind a popup. Includes server-side live cache changes, web terminal state/UI changes, and tests for synthetic pricing, live snapshot freshness/NBBO retention, and live pause/filter interactions.","status":"closed","priority":1,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T21:02:52Z","created_by":"dirtydishes","updated_at":"2026-04-28T21:13:38Z","started_at":"2026-04-28T21:02:57Z","closed_at":"2026-04-28T21:13:38Z","close_reason":"Completed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-e4r","title":"Implement smart-money flow filtering and synthetic firehose modes","description":"Implement the approved multi-surface plan for named synthetic market profiles, options raw-vs-signal filtering, live/API filter contracts, Tape page client-side flow filters, firehose-readiness improvements, tests, and README updates.","status":"closed","priority":1,"issue_type":"feature","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T20:10:49Z","created_by":"dirtydishes","updated_at":"2026-04-28T20:29:29Z","started_at":"2026-04-28T20:10:53Z","closed_at":"2026-04-28T20:29:29Z","close_reason":"Implemented synthetic market profiles, options signal-path filtering, signal-aware API/replay contracts, Tape page filters, tests, and README updates. Follow-up tracked in islandflow-biq.","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"islandflow-0sa","title":"Fix live tape auto-hold, history seam, and remove manual pause control","description":"The live tape should automatically hold when the user scrolls away from the top, resume when they return to the top or use Jump to top, and keep older prints available seamlessly beyond the hot window. Manual Pause/Resume control is now redundant and should be removed from live tape panes. This work should also fix the current regression where paused/held tapes still mutate, and align the options tape with a strict 100-row hot head backed by ClickHouse history.","status":"in_progress","priority":2,"issue_type":"bug","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-16T18:12:51Z","created_by":"dirtydishes","updated_at":"2026-05-16T18:12:54Z","started_at":"2026-05-16T18:12:54Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"islandflow-0sa","title":"Fix live tape auto-hold, history seam, and remove manual pause control","description":"The live tape should automatically hold when the user scrolls away from the top, resume when they return to the top or use Jump to top, and keep older prints available seamlessly beyond the hot window. Manual Pause/Resume control is now redundant and should be removed from live tape panes. This work should also fix the current regression where paused/held tapes still mutate, and align the options tape with a strict 100-row hot head backed by ClickHouse history.","notes":"Implemented live scroll-hold with no live pause button, demand-loaded ClickHouse history, a 100-row options hot head, and cache-first scoped snapshots. Validated with bun test apps/web/app/terminal.test.ts services/api/tests/live.test.ts and bun --cwd=apps/web run build.","status":"closed","priority":2,"issue_type":"bug","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-16T18:12:51Z","created_by":"dirtydishes","updated_at":"2026-05-16T18:23:43Z","started_at":"2026-05-16T18:12:54Z","closed_at":"2026-05-16T18:23:43Z","close_reason":"Closed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-2db","title":"Manually remove stale islandflow local-infra containers from VPS","description":"The live VPS still has an older compose project named islandflow created from the repo-root docker-compose.yml. Inspection shows it is separate from the supported islandflow-vps deployment stack and exposes NATS, ClickHouse, and Redis on host ports. Container removal commands currently hang when run as the delta user through Docker, so cleanup likely needs a focused maintenance window and possibly host-level intervention or a Docker daemon restart.","notes":"The duplicate islandflow compose project on the VPS was confirmed live during inspection. Nginx Proxy Manager routes public traffic only to islandflow-vps web/api by Docker name, so the stale islandflow project appears to be stray local-infra state rather than part of the supported production path. Attempts to remove the stale containers with docker compose down and docker rm -f as the delta user hung and timed out, so manual cleanup likely needs a maintenance window and possibly Docker daemon intervention.","status":"open","priority":2,"issue_type":"task","owner":"dishes@dpdrm.com","created_at":"2026-05-16T01:27:27Z","created_by":"dirtydishes","updated_at":"2026-05-16T01:28:59Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-c87","title":"Clean up duplicate Islandflow Docker infra on VPS","description":"The live VPS is currently running both the production-style islandflow-vps Docker stack and an older root-level islandflow infra stack that publishes NATS, ClickHouse, and Redis on host ports. Investigate whether the older stack is unused, remove it safely if so, and update docs/deploy guidance so the server topology is clearer.","notes":"Inspected the live VPS and confirmed the duplicate compose project: islandflow-vps is the supported deployment stack, while a separate islandflow project from the repo-root docker-compose.yml still runs exposed NATS/ClickHouse/Redis containers. Verified Nginx Proxy Manager routes only to islandflow-vps web/api by Docker name. Attempted cleanup via docker compose down and docker rm -f on the stale islandflow containers, but those commands hung for the delta user and timed out. Added repo guardrails and docs so deploy warns when the duplicate project exists, and opened islandflow-2db for manual host-level cleanup during a maintenance window.","status":"closed","priority":2,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-16T01:16:05Z","created_by":"dirtydishes","updated_at":"2026-05-16T01:28:07Z","started_at":"2026-05-16T01:16:09Z","closed_at":"2026-05-16T01:28:07Z","close_reason":"Completed the repo-side investigation and guardrails. Actual server-side container removal is blocked by hanging Docker operations and is tracked separately in islandflow-2db for a maintenance window.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-4gj","title":"Clarify Docker-first deploy workflow and mark native runtime experimental","description":"After inspecting the live VPS, native deployment is not ready for routine use: Nginx Proxy Manager routes to Docker container names, Bun is not installed on the host, sudo systemctl is not passwordless, and no Islandflow units exist. Update deploy messaging and docs so Docker remains the clearly recommended deployment path and native runtime is labeled experimental/future-facing with server prerequisites called out.","notes":"Updated deploy messaging and docs after live VPS inspection. scripts/deploy.ts now marks Docker as the default and recommended runtime, labels native as experimental, switches native systemctl default to sudo -n systemctl, and prints explicit native precheck failures for missing Bun/systemctl access/units. Updated README.md, deployment/docker/README.md, and deployment/native/README.md to reflect the current Docker + Nginx Proxy Manager topology. Validation: ./deploy --help, ./deploy main --runtime native --no-build (fails fast with Bun-missing message), bun run check:docker-workspace.","status":"closed","priority":2,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-16T01:10:11Z","created_by":"dirtydishes","updated_at":"2026-05-16T01:12:39Z","started_at":"2026-05-16T01:10:14Z","closed_at":"2026-05-16T01:12:39Z","close_reason":"Closed","dependency_count":0,"dependent_count":0,"comment_count":0} diff --git a/apps/web/app/terminal.test.ts b/apps/web/app/terminal.test.ts index 8878fd9..0362723 100644 --- a/apps/web/app/terminal.test.ts +++ b/apps/web/app/terminal.test.ts @@ -164,6 +164,7 @@ describe("live manifest", () => { expect(optionsSubscription?.underlying_ids).toEqual(["AAPL"]); expect(optionsSubscription?.option_contract_id).toBe("AAPL-2025-01-17-200-C"); + expect(optionsSubscription?.snapshot_limit).toBe(100); expect(equitiesSubscription?.underlying_ids).toEqual(["AAPL"]); }); @@ -635,23 +636,23 @@ describe("live tape history helpers", () => { expect(next.map((item) => item.trace_id)).toEqual(["existing", "older-1"]); }); - it("keeps scoped option and equity history on the normal retention cap", () => { + it("keeps option and equity history effectively unbounded while scrolling", () => { expect( getLiveHistoryRetentionCap({ channel: "options", underlying_ids: ["AAPL"], option_contract_id: "AAPL-2025-01-17-200-C" } as any) - ).toBeGreaterThan(0); + ).toBe(0); expect( getLiveHistoryRetentionCap({ channel: "equities", underlying_ids: ["AAPL"] } as any) - ).toBeGreaterThan(0); + ).toBe(0); }); - it("keeps auto-hydrating scoped live history while next_before exists", () => { + it("does not auto-hydrate scoped live history before the scroll gate is reached", () => { const manifest = getLiveManifest( "/tape", "AAPL", @@ -669,18 +670,12 @@ describe("live tape history helpers", () => { expect( getScopedLiveAutoHydrationChannels(true, "/tape", manifest, historyCursors, {}) - ).toEqual(["options", "equities"]); + ).toEqual([]); expect( getScopedLiveAutoHydrationChannels(true, "/tape", manifest, historyCursors, { [getLiveSubscriptionKey(manifest.find((subscription) => subscription.channel === "options")!)]: true }) - ).toEqual(["equities"]); - expect( - getScopedLiveAutoHydrationChannels(true, "/tape", manifest, { - ...historyCursors, - [getLiveSubscriptionKey(manifest.find((subscription) => subscription.channel === "equities")!)]: null - }, {}) - ).toEqual(["options"]); + ).toEqual([]); }); it("restores the same anchor key after live insertions at the top", () => { @@ -864,9 +859,15 @@ describe("signals helpers", () => { expect(getAlertWindowAnchorTs([], 42)).toBe(42); }); - it("returns connected/stale live status labels without live wording", () => { + it("returns connected/held/stale live status labels without live wording", () => { expect(statusLabel("connected", false, "live")).toBe("Connected"); + expect(statusLabel("connected", true, "live")).toBe("Held"); expect(statusLabel("stale", false, "live")).toBe("Feed behind"); + expect(statusLabel("stale", true, "live")).toBe("Feed behind"); + }); + + it("keeps replay pause wording on replay tapes", () => { + expect(statusLabel("connected", true, "replay")).toBe("Paused"); }); it("treats healthy scoped channels as connected even when no matching rows are visible", () => { diff --git a/apps/web/app/terminal.tsx b/apps/web/app/terminal.tsx index 20070fe..33eec33 100644 --- a/apps/web/app/terminal.tsx +++ b/apps/web/app/terminal.tsx @@ -77,6 +77,7 @@ const LIVE_HOT_WINDOW_OPTIONS = parseBoundedInt( 1, 100000 ); +const LIVE_OPTIONS_HEAD_LIMIT = 100; const LIVE_HISTORY_SOFT_CAP = parseBoundedInt( process.env.NEXT_PUBLIC_LIVE_HISTORY_SOFT_CAP, 5000, @@ -846,7 +847,7 @@ export const getLiveHistoryRetentionCap = (subscription: LiveSubscription): numb switch (subscription.channel) { case "options": case "equities": - return LIVE_HISTORY_SOFT_CAP; + return 0; default: return LIVE_HISTORY_SOFT_CAP; } @@ -859,27 +860,12 @@ export const getScopedLiveAutoHydrationChannels = ( historyCursors: Partial>, historyLoading: Partial> ): Array> => { - if (!enabled || pathname !== "/tape") { - return []; - } - - const channels: Array> = []; - for (const subscription of manifest) { - const scoped = - (subscription.channel === "options" && - (subscription.underlying_ids?.length || subscription.option_contract_id)) || - (subscription.channel === "equities" && subscription.underlying_ids?.length); - if (!scoped) { - continue; - } - - const key = getLiveSubscriptionKey(subscription); - if (historyCursors[key] && !historyLoading[key]) { - channels.push(subscription.channel); - } - } - - return channels; + void enabled; + void pathname; + void manifest; + void historyCursors; + void historyLoading; + return []; }; export const getLiveFeedStatus = ( @@ -2027,7 +2013,10 @@ export const prunePinnedEntries = ( export const statusLabel = (status: WsStatus, paused: boolean, mode: TapeMode): string => { if (paused) { - return "Paused"; + if (mode === "replay") { + return "Paused"; + } + return status === "connected" ? "Held" : statusLabel(status, false, mode); } if (mode === "replay") { @@ -2512,22 +2501,20 @@ type PausableTapeViewConfig = { const usePausableTapeView = ( config: PausableTapeViewConfig ): TapeState => { - const [paused, setPaused] = useState(false); const [data, setData] = useState>(EMPTY_PAUSABLE_TAPE); + const holdForScroll = config.enabled ? (config.shouldHold ? config.shouldHold() : false) : false; useEffect(() => { if (!config.enabled) { - setPaused(false); setData(EMPTY_PAUSABLE_TAPE); return; } - const holdForScroll = config.shouldHold ? config.shouldHold() : false; setData((current) => { const next = reducePausableTapeData( current, config.sourceItems, - paused || holdForScroll, + holdForScroll, config.retentionLimit ?? LIVE_HOT_WINDOW ); if (next === current) { @@ -2535,7 +2522,7 @@ const usePausableTapeView = ( } const unseenCount = next.seenKeys.size - current.seenKeys.size; - if (!paused && unseenCount > 0) { + if (unseenCount > 0) { config.onNewItems?.(unseenCount); config.captureScroll?.(); } @@ -2548,17 +2535,11 @@ const usePausableTapeView = ( config.onNewItems, config.captureScroll, config.retentionLimit, - config.shouldHold, - paused + holdForScroll ]); useEffect(() => { - if (!config.enabled || paused) { - return; - } - - const holdForScroll = config.shouldHold ? config.shouldHold() : false; - if (holdForScroll) { + if (!config.enabled || holdForScroll) { return; } @@ -2581,14 +2562,9 @@ const usePausableTapeView = ( config.onNewItems, config.retentionLimit, config.resumeSignal, - config.shouldHold, - paused + holdForScroll ]); - const togglePause = useCallback(() => { - setPaused((current) => !current); - }, []); - const status = config.enabled ? config.sourceStatus : "disconnected"; const projected = projectPausableTapeState(data.visible, status, config.lastUpdate); const historyItems = config.historyTail ?? []; @@ -2602,9 +2578,9 @@ const usePausableTapeView = ( lastUpdate: projected.lastUpdate, replayTime: null, replayComplete: false, - paused, + paused: holdForScroll, dropped: data.dropped, - togglePause + togglePause: () => {} }; }; @@ -3052,7 +3028,7 @@ export const getLiveManifest = ( ? undefined : optionPrintFilters ?? flowFilters, ...optionScope, - snapshot_limit: LIVE_HOT_WINDOW_OPTIONS + snapshot_limit: LIVE_OPTIONS_HEAD_LIMIT }); } if (features.nbbo) { @@ -3337,7 +3313,7 @@ const useLiveSession = ( switch (subscription.channel) { case "options": - mergeItems(setOptions, optionsRef, items as OptionPrint[], LIVE_HOT_WINDOW_OPTIONS, { + mergeItems(setOptions, optionsRef, items as OptionPrint[], LIVE_OPTIONS_HEAD_LIMIT, { setter: setOptionsHistory, ref: optionsHistoryRef, cap: getLiveHistoryRetentionCap(subscription) @@ -3794,6 +3770,7 @@ const TapeStatus = ({ }; type TapeControlsProps = { + mode: TapeMode; paused: boolean; onTogglePause: () => void; isAtTop: boolean; @@ -3801,13 +3778,15 @@ type TapeControlsProps = { onJump: () => void; }; -const TapeControls = ({ paused, onTogglePause, isAtTop, missed, onJump }: TapeControlsProps) => { +const TapeControls = ({ mode, paused, onTogglePause, isAtTop, missed, onJump }: TapeControlsProps) => { const active = !isAtTop && missed > 0; return (
- + {mode === "replay" ? ( + + ) : null} @@ -5373,7 +5352,7 @@ const useTerminalState = () => { sourceItems: liveSession.options, historyTail: liveSession.optionsHistory, lastUpdate: liveSession.lastUpdate, - retentionLimit: LIVE_HOT_WINDOW_OPTIONS, + retentionLimit: LIVE_OPTIONS_HEAD_LIMIT, captureScroll: optionsAnchor.capture, onNewItems: optionsScroll.onNewItems, shouldHold: () => !optionsScroll.isAtTopRef.current, @@ -7141,6 +7120,7 @@ const OptionsPane = memo(({ state, limit }: OptionsPaneProps) => { } actions={ { } actions={ { } actions={ { } actions={ + + + + + Turn Summary: Live tape scroll hold and lazy history + + + +
+
+

Live tape now holds on scroll, resumes at top, and lazy-loads deep history

+

+ The live tape no longer depends on a manual pause button in live mode. Scrolling away from the top now holds the + tape automatically, Jump to top resumes it, the options hot head is capped at 100 rows, and older + history is fetched from ClickHouse only when the scroll gate requests it. +

+
+ +
Created 2026-05-16 during issue islandflow-0sa.
+ +
+

Summary

+
+

+ This change aligns the tape with the intended operator workflow: hold the live head while investigating older + rows, keep historical prints valid even when old, and avoid preloading a large ClickHouse backlog until the + user actually scrolls into it. +

+
+
+ +
+

Changes Made

+
+
    +
  • Removed the live-mode Pause/Resume control from tape pane actions while keeping replay pause behavior intact.
    • +
  • Changed live tape status copy from manual Paused semantics to scroll-held Held.
    • +
  • Capped the live options head at 100 rows.
    • +
  • Stopped scoped live history from auto-hydrating in the background.
    • +
  • Made scoped options and equities snapshots prefer hot cached rows first, then backfill from ClickHouse when needed.
    • +
  • Made options and equities history retention effectively unbounded on the client so deep scrolling does not get trimmed away prematurely.
    • +
+
+
+ +
+

Context

+
+

+ The tape previously mixed several behaviors: a manual pause button, automatic scroll holding, scoped background + auto-hydration, and a much deeper options hot head. That created two user-visible problems: the live control model + felt redundant, and older prints could disappear or feel inconsistent when switching views or waiting for newer + rows to arrive. +

+
+
+ +
+

Important Implementation Details

+
+
    +
  • apps/web/app/terminal.tsx: live usePausableTapeView now treats scroll position as the hold source of truth.
    • +
  • apps/web/app/terminal.tsx: options live snapshot and retention now use a strict LIVE_OPTIONS_HEAD_LIMIT = 100.
    • +
  • apps/web/app/terminal.tsx: scoped history auto-hydration helper now returns no channels, so ClickHouse history stays lazy.
    • +
  • services/api/src/live.ts: scoped option/equity snapshots now filter the hot cache first, then merge ClickHouse backfill without seam duplicates.
    • +
+ 
statusLabel("connected", true, "live") === "Held"
+statusLabel("connected", true, "replay") === "Paused"
+
+
+ +
+

Validation

+
+
    +
  • Passed: bun test apps/web/app/terminal.test.ts services/api/tests/live.test.ts
    • +
  • Passed: bun --cwd=apps/web run build
    • +
+
+
+ +
+

Issues, Limitations, and Mitigations

+
+
    +
  • Scoped snapshots can still backfill from ClickHouse when the hot cache does not have enough matching rows. This is intentional so focused views do not start empty.
    • +
  • Deep history is now lazy rather than eager, which reduces surprise and load, but the first deep-scroll request still depends on ClickHouse latency.
    • +
+
+
+ +
+

Follow-up Work

+
+

No additional follow-up issues were created in this turn.

+
+
+
+ + diff --git a/services/api/src/live.ts b/services/api/src/live.ts index ca228fc..ab4ceee 100644 --- a/services/api/src/live.ts +++ b/services/api/src/live.ts @@ -39,7 +39,8 @@ import { type Cursor, type EquityCandle, type EquityPrint, - type LiveChannel + type LiveChannel, + type OptionPrint } from "@islandflow/types"; import { createMetrics } from "@islandflow/observability"; import type { RedisClientType } from "redis"; @@ -456,6 +457,54 @@ export const buildOptionSnapshotFilters = ( }; }; +const matchesScopedOptionSnapshot = ( + item: OptionPrint, + subscription: Extract +): boolean => { + if (!matchesOptionPrintFilters(item, subscription.filters)) { + return false; + } + + if (subscription.option_contract_id && item.option_contract_id !== subscription.option_contract_id) { + return false; + } + + if (!subscription.underlying_ids?.length) { + return true; + } + + const allowed = new Set(subscription.underlying_ids.map((value) => value.toUpperCase())); + return allowed.has(item.underlying_id.toUpperCase()); +}; + +const matchesScopedEquitySnapshot = ( + item: EquityPrint, + subscription: Extract +): boolean => { + if (!subscription.underlying_ids?.length) { + return true; + } + + const allowed = new Set(subscription.underlying_ids.map((value) => value.toUpperCase())); + return allowed.has(item.underlying_id.toUpperCase()); +}; + +const mergeSnapshotBackfill = ( + cached: T[], + backfill: T[], + limit: number, + cursorOf: (item: T) => Cursor +): T[] => { + const deduped = new Map(); + + for (const item of [...cached, ...backfill]) { + const cursor = cursorOf(item); + deduped.set(`${cursor.ts}:${cursor.seq}`, item); + } + + return sortGenericItems(Array.from(deduped.values()), cursorOf).slice(0, limit); +}; + const candleRedisKey = (underlyingId: string, intervalMs: number): string => `live:equity-candles:${underlyingId}:${intervalMs}`; @@ -740,12 +789,20 @@ export class LiveStateManager { async getSnapshot(subscription: LiveSubscription): Promise> { switch (subscription.channel) { case "options": { + const config = this.generic.options; + const limit = snapshotLimitFor(subscription, config.limit); const scoped = Boolean(subscription.underlying_ids?.length) || Boolean(subscription.option_contract_id); if (subscription.filters?.view === "raw" || scoped) { - this.stats.scopedClickHouseSnapshots += 1; - const limit = snapshotLimitFor(subscription, this.generic.options.limit); - const storageFilters = buildOptionSnapshotFilters(subscription); - const items = await fetchRecentOptionPrints(this.clickhouse, limit, undefined, storageFilters); + const cached = (this.genericItems.get("options") ?? []) + .filter((entry) => matchesScopedOptionSnapshot(entry, subscription)) + .slice(0, limit); + let items = cached; + if (cached.length < limit) { + this.stats.scopedClickHouseSnapshots += 1; + const storageFilters = buildOptionSnapshotFilters(subscription); + const backfill = await fetchRecentOptionPrints(this.clickhouse, limit, undefined, storageFilters); + items = mergeSnapshotBackfill(cached, backfill, limit, (entry) => ({ ts: entry.ts, seq: entry.seq })); + } return { subscription, items, @@ -754,9 +811,7 @@ export class LiveStateManager { }; } - const config = this.generic.options; this.stats.genericCacheSnapshots += 1; - const limit = snapshotLimitFor(subscription, config.limit); const items = (this.genericItems.get("options") ?? []) .filter((entry) => matchesOptionPrintFilters(entry, subscription.filters)) .slice(0, limit); @@ -785,9 +840,16 @@ export class LiveStateManager { const config = this.generic.equities; const limit = snapshotLimitFor(subscription, config.limit); if (subscription.underlying_ids?.length) { - this.stats.scopedClickHouseSnapshots += 1; - const filters: EquityPrintQueryFilters = { underlyingIds: subscription.underlying_ids }; - const items = await fetchRecentEquityPrints(this.clickhouse, limit, filters); + const cached = (this.genericItems.get("equities") ?? []) + .filter((entry) => matchesScopedEquitySnapshot(entry, subscription)) + .slice(0, limit); + let items = cached; + if (cached.length < limit) { + this.stats.scopedClickHouseSnapshots += 1; + const filters: EquityPrintQueryFilters = { underlyingIds: subscription.underlying_ids }; + const backfill = await fetchRecentEquityPrints(this.clickhouse, limit, filters); + items = mergeSnapshotBackfill(cached, backfill, limit, config.cursor); + } return { subscription, items, diff --git a/services/api/tests/live.test.ts b/services/api/tests/live.test.ts index bd4d0c8..fff1d61 100644 --- a/services/api/tests/live.test.ts +++ b/services/api/tests/live.test.ts @@ -627,6 +627,57 @@ describe("LiveStateManager", () => { ]); }); + it("prefers cached scoped option rows before clickhouse backfill", async () => { + const now = Date.now(); + const manager = new LiveStateManager( + makeClickHouse((query) => + query.includes("FROM option_prints") + ? [ + { + source_ts: now - 1_000, + ingest_ts: now - 999, + seq: 1, + trace_id: "opt-backfill", + ts: now - 1_000, + option_contract_id: "AAPL-2025-01-17-200-C", + underlying_id: "AAPL", + price: 1, + size: 10, + exchange: "X", + signal_pass: false + } + ] + : [] + ), + null + ); + + await manager.ingest("options", { + source_ts: now, + ingest_ts: now + 1, + seq: 2, + trace_id: "opt-hot", + ts: now, + option_contract_id: "AAPL-2025-01-17-200-C", + underlying_id: "AAPL", + price: 2, + size: 10, + exchange: "X", + signal_pass: true + }); + + const snapshot = await manager.getSnapshot({ + channel: "options", + underlying_ids: ["AAPL"], + option_contract_id: "AAPL-2025-01-17-200-C" + }); + + expect((snapshot.items as Array<{ trace_id: string }>).map((item) => item.trace_id).slice(0, 2)).toEqual([ + "opt-hot", + "opt-backfill" + ]); + }); + it("seeds scoped equity snapshots from clickhouse rows older than 24h", async () => { const now = Date.now(); const staleTs = now - 25 * 60 * 60 * 1000; From e3940eb0a6a487441aa9e8a16f54d0f20823a36b Mon Sep 17 00:00:00 2001 From: dirtydishes Date: Sat, 16 May 2026 14:57:33 -0400 Subject: [PATCH 3/5] Update AGENTS.md --- AGENTS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index b5c7b69..3ab1cf0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -117,7 +117,7 @@ Each turn document must include these sections: 2. **Changes Made** 3. **Context** 4. **Important Implementation Details** -5. **Impact for End-Users** +5. **Expected Impact for End-Users** 5. **Validation** 6. **Issues, Limitations, and Mitigations** 7. **Follow-up Work** From bd60d0d5d5e4c83b67a6378eb88b0a8acc3bf049 Mon Sep 17 00:00:00 2001 From: dirtydishes Date: Sat, 16 May 2026 17:27:02 -0400 Subject: [PATCH 4/5] implement durable options tape history --- .beads/issues.jsonl | 1 + apps/web/app/globals.css | 11 + apps/web/app/terminal.test.ts | 46 ++-- apps/web/app/terminal.tsx | 63 +++-- docs/clickhouse-reset-runbook.md | 57 ++++ ...-16-1725-durable-options-tape-history.html | 245 ++++++++++++++++++ packages/storage/tests/option-prints.test.ts | 27 +- services/api/src/live.ts | 2 +- services/api/tests/live.test.ts | 27 ++ 9 files changed, 423 insertions(+), 56 deletions(-) create mode 100644 docs/clickhouse-reset-runbook.md create mode 100644 docs/turns/2026-05-16-1725-durable-options-tape-history.html diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index 065a612..605077e 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -1,3 +1,4 @@ +{"_type":"issue","id":"islandflow-200","title":"Implement durable options tape history","description":"Implement the plan from docs/plans/2026-05-16-1711-durable-options-tape-history.html: durable ClickHouse-backed options history, signal/all prints view selection, preserved execution context, stale semantics limited to live health, reset runbook, tests, and turn documentation.","status":"closed","priority":1,"issue_type":"feature","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-16T21:21:30Z","created_by":"dirtydishes","updated_at":"2026-05-16T21:26:51Z","started_at":"2026-05-16T21:21:33Z","closed_at":"2026-05-16T21:26:51Z","close_reason":"Implemented durable options tape history, signal/raw view selection, reset runbook, tests, and turn documentation.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-k4f","title":"Gate deploy script on docker workspace snapshot sync","description":"Prevent frozen-lockfile build failures during deploy by adding a local preflight in scripts/deploy.ts that runs bun run check:docker-workspace and aborts with a clear sync+commit remediation message when stale.","status":"closed","priority":1,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-15T23:01:44Z","created_by":"dirtydishes","updated_at":"2026-05-15T23:04:11Z","started_at":"2026-05-15T23:01:48Z","closed_at":"2026-05-15T23:04:11Z","close_reason":"Closed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-xll","title":"Fix bun.lock drift causing frozen-lockfile Docker build failures","description":"Docker image builds fail in multiple targets (candles, web, ingest services) because bun install --frozen-lockfile detects lockfile changes. Update workspace lockfile to match manifests and verify frozen install succeeds.","status":"closed","priority":1,"issue_type":"bug","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-15T22:52:38Z","created_by":"dirtydishes","updated_at":"2026-05-15T22:55:23Z","started_at":"2026-05-15T22:52:40Z","closed_at":"2026-05-15T22:55:23Z","close_reason":"Closed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-9nd","title":"Hosted synthetic tape redesign with internal control surface","description":"Implement hosted synthetic market redesign with shared deterministic regime engine, internal JetStream KV control plane, ingest coupling across options and equities, and an internal bottom-right synthetic-control drawer with Next proxy routes. Preserve the six public smart-money categories while adding hidden subtype families, soft coverage accounting, and backend-only admin endpoints.\n","status":"closed","priority":1,"issue_type":"feature","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-14T01:25:02Z","created_by":"dirtydishes","updated_at":"2026-05-14T02:10:03Z","started_at":"2026-05-14T01:25:09Z","closed_at":"2026-05-14T02:10:03Z","close_reason":"Completed","dependency_count":0,"dependent_count":0,"comment_count":0} diff --git a/apps/web/app/globals.css b/apps/web/app/globals.css index 23bdb2e..1b2205c 100644 --- a/apps/web/app/globals.css +++ b/apps/web/app/globals.css @@ -606,6 +606,13 @@ h3 { text-transform: uppercase; } +.flow-filter-section-copy { + margin: -2px 0 0; + color: var(--text-muted); + font-size: 0.78rem; + line-height: 1.35; +} + .flow-filter-checkbox-grid, .flow-filter-chip-grid { display: grid; @@ -617,6 +624,10 @@ h3 { grid-template-columns: repeat(3, minmax(0, 1fr)); } +.flow-filter-chip-grid-two { + grid-template-columns: repeat(2, minmax(0, 1fr)); +} + .flow-filter-check { display: inline-flex; align-items: center; diff --git a/apps/web/app/terminal.test.ts b/apps/web/app/terminal.test.ts index 0362723..03114c4 100644 --- a/apps/web/app/terminal.test.ts +++ b/apps/web/app/terminal.test.ts @@ -17,7 +17,6 @@ import { getEffectiveOptionPrintFilters, getAlertWindowAnchorTs, getHotChannelFeedStatus, - getScopedLiveAutoHydrationChannels, getLiveHistoryRetentionCap, getOptionTableSnapshot, getOptionScope, @@ -298,6 +297,24 @@ describe("contract-focused option helpers", () => { }); }); + it("includes the selected options view in tape query params", () => { + expect( + buildOptionTapeQueryParams( + { + ...buildDefaultFlowFilters(), + view: "raw", + securityTypes: undefined, + nbboSides: undefined, + optionTypes: undefined + }, + { underlying_ids: ["AAPL"] } + ) + ).toEqual({ + view: "raw", + underlying_ids: "AAPL" + }); + }); + it("keeps the focus seed until the matching scoped subscription has loaded it", () => { const seedItem = makeOptionPrint({ trace_id: "focused-seed", @@ -652,32 +669,6 @@ describe("live tape history helpers", () => { ).toBe(0); }); - it("does not auto-hydrate scoped live history before the scroll gate is reached", () => { - const manifest = getLiveManifest( - "/tape", - "AAPL", - 60000, - buildDefaultFlowFilters(), - { - underlying_ids: ["AAPL"], - option_contract_id: "AAPL-2025-01-17-200-C" - }, - { underlying_ids: ["AAPL"] } - ); - const historyCursors = Object.fromEntries( - manifest.map((subscription) => [getLiveSubscriptionKey(subscription), { ts: 1, seq: 1 }]) - ); - - expect( - getScopedLiveAutoHydrationChannels(true, "/tape", manifest, historyCursors, {}) - ).toEqual([]); - expect( - getScopedLiveAutoHydrationChannels(true, "/tape", manifest, historyCursors, { - [getLiveSubscriptionKey(manifest.find((subscription) => subscription.channel === "options")!)]: true - }) - ).toEqual([]); - }); - it("restores the same anchor key after live insertions at the top", () => { const nextKeys = ["new-1", "new-2", "anchor", "after-1", "after-2"]; expect(findAnchorRestoreIndex(nextKeys, "anchor", ["anchor", "after-1", "after-2"])).toBe(2); @@ -806,6 +797,7 @@ describe("flow filter popup helpers", () => { expect(countActiveFlowFilterGroups(defaults)).toBe(0); expect(countActiveFlowFilterGroups(next)).toBe(3); + expect(countActiveFlowFilterGroups({ ...defaults, view: "raw" })).toBe(1); expect(buildDefaultFlowFilters()).toEqual(defaults); }); }); diff --git a/apps/web/app/terminal.tsx b/apps/web/app/terminal.tsx index 33eec33..2135a75 100644 --- a/apps/web/app/terminal.tsx +++ b/apps/web/app/terminal.tsx @@ -34,6 +34,7 @@ import type { LiveHotChannelHealthMap, LiveSubscription, OptionFlowFilters, + OptionFlowView, OptionNbboSide, OptionSecurityType, OptionType, @@ -853,21 +854,6 @@ export const getLiveHistoryRetentionCap = (subscription: LiveSubscription): numb } }; -export const getScopedLiveAutoHydrationChannels = ( - enabled: boolean, - pathname: string, - manifest: LiveSubscription[], - historyCursors: Partial>, - historyLoading: Partial> -): Array> => { - void enabled; - void pathname; - void manifest; - void historyCursors; - void historyLoading; - return []; -}; - export const getLiveFeedStatus = ( sourceStatus: WsStatus, freshestTs: number | null, @@ -1436,6 +1422,9 @@ export const countActiveFlowFilterGroups = (filters: OptionFlowFilters): number if ((filters.minNotional ?? undefined) !== (defaults.minNotional ?? undefined)) { count += 1; } + if ((filters.view ?? defaults.view) !== defaults.view) { + count += 1; + } return count; }; @@ -3684,18 +3673,6 @@ const useLiveSession = ( [enabled, manifest, historyCursors, historyLoading] ); - useEffect(() => { - for (const channel of getScopedLiveAutoHydrationChannels( - enabled, - pathname, - manifest, - historyCursors, - historyLoading - )) { - void loadOlder(channel); - } - }, [enabled, pathname, manifest, historyCursors, historyLoading, loadOlder]); - return { status, connectedAt, @@ -6904,6 +6881,17 @@ export const FlowFilterPopover = ({ filters, onChange }: FlowFilterPopoverProps) })); }; + const applyView = (view: OptionFlowView) => { + onChange((prev) => ({ + ...prev, + view, + securityTypes: view === "raw" ? undefined : prev.securityTypes ?? DEFAULT_FLOW_SECURITY_TYPES, + nbboSides: view === "raw" ? undefined : prev.nbboSides, + optionTypes: view === "raw" ? undefined : prev.optionTypes, + minNotional: view === "raw" ? undefined : prev.minNotional + })); + }; + useEffect(() => { if (!open) { return; @@ -6968,6 +6956,27 @@ export const FlowFilterPopover = ({ filters, onChange }: FlowFilterPopoverProps)
+ +
+ {[ + { label: "Signal", value: "signal" as const }, + { label: "All prints", value: "raw" as const } + ].map((preset) => ( + + ))} +
+

+ Signal keeps classifier-ready prints. All prints includes raw option tape rows. +

+ +
{(["stock", "etf"] as OptionSecurityType[]).map((value) => ( diff --git a/docs/clickhouse-reset-runbook.md b/docs/clickhouse-reset-runbook.md new file mode 100644 index 0000000..dac1775 --- /dev/null +++ b/docs/clickhouse-reset-runbook.md @@ -0,0 +1,57 @@ +# ClickHouse Reset Runbook + +This runbook is for deliberately wiping durable market-data history from ClickHouse in local development or on the VPS. It is destructive. Do not run these commands from application startup, deployment hooks, or unattended scripts. + +## When To Use + +Use this only when an operator has decided that existing option, equity, flow, and derived-event history should be discarded and rebuilt from fresh ingest. + +Before running a reset: + +- Confirm the target environment: local Docker or VPS Docker. +- Confirm there is no active analysis depending on the existing history. +- Take a backup if the data may be needed later. +- Stop ingest and API services so new writes do not race the reset. + +## Local Docker Reset + +From the repository root: + +```bash +bun run dev:infra +docker compose exec clickhouse clickhouse-client --query "SHOW TABLES" +docker compose exec clickhouse clickhouse-client --query "TRUNCATE TABLE IF EXISTS option_prints" +docker compose exec clickhouse clickhouse-client --query "TRUNCATE TABLE IF EXISTS option_nbbo" +docker compose exec clickhouse clickhouse-client --query "TRUNCATE TABLE IF EXISTS equity_prints" +docker compose exec clickhouse clickhouse-client --query "TRUNCATE TABLE IF EXISTS equity_quotes" +docker compose exec clickhouse clickhouse-client --query "TRUNCATE TABLE IF EXISTS equity_print_joins" +docker compose exec clickhouse clickhouse-client --query "TRUNCATE TABLE IF EXISTS flow_packets" +docker compose exec clickhouse clickhouse-client --query "TRUNCATE TABLE IF EXISTS smart_money_events" +docker compose exec clickhouse clickhouse-client --query "TRUNCATE TABLE IF EXISTS classifier_hits" +docker compose exec clickhouse clickhouse-client --query "TRUNCATE TABLE IF EXISTS alerts" +docker compose exec clickhouse clickhouse-client --query "TRUNCATE TABLE IF EXISTS inferred_dark_events" +``` + +If the local compose project uses `deployment/docker/docker-compose.yml`, run the same commands with `docker compose -f deployment/docker/docker-compose.yml exec clickhouse ...`. + +## VPS Docker Reset + +On the VPS, first identify the active compose project and ClickHouse service: + +```bash +docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}" +docker compose -f deployment/docker/docker-compose.yml ps +``` + +Then stop writers and run the same `TRUNCATE TABLE IF EXISTS ...` commands against the active ClickHouse container. Prefer `docker compose exec clickhouse clickhouse-client --query ""` when the compose project is healthy; otherwise use `docker exec clickhouse-client --query ""`. + +## Verification + +After the reset: + +```bash +docker compose exec clickhouse clickhouse-client --query "SELECT count() FROM option_prints" +docker compose exec clickhouse clickhouse-client --query "SELECT count() FROM flow_packets" +``` + +Restart ingest/API services through the normal dev or deployment path. The options tape should repopulate its 100-row hot head from new signal prints, and older rows should appear only after the scroll gate asks `/history/options` for ClickHouse-backed history. diff --git a/docs/turns/2026-05-16-1725-durable-options-tape-history.html b/docs/turns/2026-05-16-1725-durable-options-tape-history.html new file mode 100644 index 0000000..a586496 --- /dev/null +++ b/docs/turns/2026-05-16-1725-durable-options-tape-history.html @@ -0,0 +1,245 @@ + + + + + + Durable Options Tape History + + + +
+
+

Turn Document

+

Durable Options Tape History

+

+ Implemented the durable options tape plan: the live hot head is capped at 100 rows, older rows are preserved behind + the scroll gate, ClickHouse history keeps execution context, and the Filter menu now exposes Signal versus All + prints semantics. +

+
+ 2026-05-16 17:25 + Beads: islandflow-200 + Surface: Options Tape +
+
+ +
+

Summary

+

+ The options tape now behaves as a continuous instrument: the live cache stays lean, historical rows arrive only + when scrolling asks for them, and old valid rows are not treated as degraded just because they came from durable + history. +

+
+ +
+

Changes Made

+
    +
  • Changed the API default options live cache limit to 100.
    • +
  • Removed the unused scoped live auto-hydration path so history is loaded by the scroll gate.
    • +
  • Fixed unbounded options/equities history retention so a cap of 0 means keep the loaded tail.
    • +
  • Added a Filter menu Options View toggle for Signal and All prints.
    • +
  • Ensured All prints clears signal-only side/type/min-notional/security constraints.
    • +
  • Added a destructive ClickHouse reset runbook for local and VPS operators.
    • +
+
+ +
+

Context

+

+ The prior plan called out useful partial work already in the repo: ClickHouse history endpoints, execution-context + columns, scroll-hold behavior, and a shared row renderer. This implementation keeps those pieces and removes the + ambiguous history/autohydration behavior around them. +

+
+ +
+

Important Implementation Details

+
    +
  • /history/options still uses the selected option filters and scope, including raw contract drilldowns.
    • +
  • Storage tests now verify execution NBBO side, underlying spot, IV, and signal reasons survive normalization.
    • +
  • The options row path already preferred execution_nbbo_side, execution_underlying_spot, and execution_iv; tests cover that behavior.
    • +
  • The reset runbook is documented in docs/clickhouse-reset-runbook.md and is explicitly operator-confirmed.
    • +
+
+ +
+

Expected Impact for End-Users

+

+ Traders can stay on a signal-first tape by default, switch to raw prints when investigating, and scroll into older + ClickHouse-backed flow without seeing a separate stale-history treatment. +

+
+ +
+

Validation

+
    +
  • Passed: bun test packages/storage/tests/option-prints.test.ts services/api/tests/live.test.ts apps/web/app/terminal.test.ts
    • +
  • Passed: bun --cwd=apps/web run build
    • +
+
+ +
+

Issues, Limitations, and Mitigations

+
    +
  • The ClickHouse reset remains destructive. Mitigation: documented as a manual runbook only, never automatic startup behavior.
    • +
  • No live browser smoke test was run in this turn. Mitigation: unit coverage and production build exercised the changed web paths.
    • +
+
+ +
+

Follow-up Work

+

No new follow-up issue was needed. The implementation task is tracked and completed in islandflow-200.

+
+
+ + diff --git a/packages/storage/tests/option-prints.test.ts b/packages/storage/tests/option-prints.test.ts index 17b3e29..139b66a 100644 --- a/packages/storage/tests/option-prints.test.ts +++ b/packages/storage/tests/option-prints.test.ts @@ -48,6 +48,25 @@ describe("option-prints storage helpers", () => { queries.push(query); return { async json() { + if (query.includes("trace-ctx")) { + return [ + { + ...basePrint, + trace_id: "trace-ctx", + conditions: [], + execution_nbbo_bid: "1.20", + execution_nbbo_ask: "1.30", + execution_nbbo_mid: "1.25", + execution_nbbo_side: "A", + execution_underlying_spot: "450.05", + execution_underlying_source: "equity_quote_mid", + execution_iv: "0.42", + execution_iv_source: "synthetic_pressure_model", + signal_reasons: ["large_notional"], + signal_pass: 1 + } + ] as T; + } return [] as T; } }; @@ -63,8 +82,9 @@ describe("option-prints storage helpers", () => { optionContractId: "AAPL-2025-01-17-200-C", sinceTs: 123 }); - await fetchOptionPrintsBefore(client, 100, 5, 20, "alpaca"); + await fetchOptionPrintsBefore(client, 100, 5, 20, "alpaca", { view: "raw" }); await fetchOptionPrintsByTraceIds(client, ["trace-1", "trace-2"]); + const rows = await fetchRecentOptionPrints(client, 1, "trace-ctx", { view: "signal" }); expect(queries[0]).toContain("signal_pass = 1"); expect(queries[0]).toContain("(is_etf = 0 OR is_etf IS NULL)"); @@ -76,7 +96,12 @@ describe("option-prints storage helpers", () => { expect(queries[0]).toContain("ts >= 123"); expect(queries[1]).toContain("(ts, seq) < (100, 5)"); expect(queries[1]).toContain("startsWith(trace_id, 'alpaca')"); + expect(queries[1]).not.toContain("signal_pass = 1"); expect(queries[1]).toContain("ORDER BY ts DESC, seq DESC LIMIT 20"); expect(queries[2]).toContain("trace_id IN ('trace-1', 'trace-2')"); + expect(rows[0].execution_nbbo_side).toBe("A"); + expect(rows[0].execution_underlying_spot).toBe(450.05); + expect(rows[0].execution_iv).toBe(0.42); + expect(rows[0].signal_reasons).toEqual(["large_notional"]); }); }); diff --git a/services/api/src/live.ts b/services/api/src/live.ts index ab4ceee..024935e 100644 --- a/services/api/src/live.ts +++ b/services/api/src/live.ts @@ -72,7 +72,7 @@ const CHART_LIMITS = { } as const; const DEFAULT_LIVE_LIMITS: GenericLiveLimits = { - options: 1000, + options: 100, nbbo: 1000, equities: 1000, "equity-quotes": 500, diff --git a/services/api/tests/live.test.ts b/services/api/tests/live.test.ts index fff1d61..78807ca 100644 --- a/services/api/tests/live.test.ts +++ b/services/api/tests/live.test.ts @@ -69,6 +69,7 @@ describe("LiveStateManager", () => { expect(limits.flow).toBe(500); expect(limits["equity-quotes"]).toBe(500); expect(limits.alerts).toBe(300); + expect(resolveGenericLiveLimits({} as NodeJS.ProcessEnv).options).toBe(100); }); it("hydrates snapshots from redis generic windows", async () => { @@ -520,6 +521,32 @@ describe("LiveStateManager", () => { ]); }); + it("caps generic options snapshots at the 100-row hot head by default", async () => { + const manager = new LiveStateManager(makeClickHouse(), null); + const now = Date.now(); + + for (let seq = 1; seq <= 150; seq += 1) { + await manager.ingest("options", { + source_ts: now + seq, + ingest_ts: now + seq, + seq, + trace_id: `opt-${seq}`, + ts: now + seq, + option_contract_id: "AAPL-2025-01-17-200-C", + price: 1, + size: 10, + exchange: "X", + signal_pass: true + }); + } + + const snapshot = await manager.getSnapshot({ channel: "options" }); + + expect(snapshot.items).toHaveLength(100); + expect((snapshot.items as Array<{ trace_id: string }>)[0].trace_id).toBe("opt-150"); + expect(snapshot.next_before).toEqual({ ts: now + 51, seq: 51 }); + }); + it("seeds scoped option snapshots from clickhouse rows older than 24h", async () => { const now = Date.now(); const staleTs = now - 25 * 60 * 60 * 1000; From 2abdd24e2c3b8849916e1cd70428eaec7d98295a Mon Sep 17 00:00:00 2001 From: dirtydishes Date: Sat, 16 May 2026 17:44:51 -0400 Subject: [PATCH 5/5] implement durable options tape history --- .codex/hooks.json | 26 ++ ...-16-1711-durable-options-tape-history.html | 363 ++++++++++++++++++ 2 files changed, 389 insertions(+) create mode 100644 .codex/hooks.json create mode 100644 docs/plans/2026-05-16-1711-durable-options-tape-history.html diff --git a/.codex/hooks.json b/.codex/hooks.json new file mode 100644 index 0000000..94fbf97 --- /dev/null +++ b/.codex/hooks.json @@ -0,0 +1,26 @@ +{ + "hooks": { + "PreCompact": [ + { + "matcher": "", + "hooks": [ + { + "type": "command", + "command": "bd prime" + } + ] + } + ], + "SessionStart": [ + { + "matcher": "", + "hooks": [ + { + "type": "command", + "command": "bd prime" + } + ] + } + ] + } +} diff --git a/docs/plans/2026-05-16-1711-durable-options-tape-history.html b/docs/plans/2026-05-16-1711-durable-options-tape-history.html new file mode 100644 index 0000000..997af42 --- /dev/null +++ b/docs/plans/2026-05-16-1711-durable-options-tape-history.html @@ -0,0 +1,363 @@ + + + + + + Plan: Durable Options Tape History + + + +
+
+

Plan Document

+

Durable Options Tape History

+

+ Make the options tape a signal-first live instrument with scroll-gated historical depth: keep the hot cache at + 100 option prints, load older rows from ClickHouse only at the scroll gate, preserve execution context, and + render ClickHouse-backed rows exactly like any other valid flow row. +

+
+ Created 2026-05-16 17:11 + Mode: Plan + Surface: Options Tape +
+
+ +
+

Plan Summary

+

+ Treat stale strictly as feed health, not as historical-row quality. The user should be able to + analyze current live prints and earlier flow in one continuous tape, with no visual distinction between hot-cache + rows and ClickHouse-backed rows. +

+
+ +
+

Goals

+
    +
  • Keep the options tape scrolling infinitely from the user's perspective.
    • +
  • Hold only the 100 newest option prints in the hot live cache.
    • +
  • Use ClickHouse as the durable source for older rows once the scroll gate requests history.
    • +
  • Store all option-print data, including synthetic prints and execution context such as NBBO, spot, and IV.
    • +
  • Surface historical flow as real analyzable flow, not as stale, old, or degraded data.
    • +
  • Keep the default tape view signal-first while exposing all/raw prints from the existing Filter menu.
    • +
+
+ +
+

Proposed Changes

+
    +
  • + Keep islandflow-0sa's useful pieces: scroll-hold behavior, LIVE_OPTIONS_HEAD_LIMIT = 100, + lazy /history/options loading, cache-first scoped snapshots, and preserved execution-context columns. +
    • +
  • + Stop tests and UI copy from asserting that valid rows older than 24 hours are stale when shown as + history. +
    • +
  • + Keep freshness gating only for live fanout/cache admission and channel health, not for historical validity. +
    • +
  • + Remove dead LiveHistoryBuffer and auto-hydration scaffolding if it remains unused after the flow is + explicit. +
    • +
  • + Keep the default options tape view as signal, and add a filter-menu view control with + Signal and All prints. +
    • +
  • + Ensure hot-cache rows and ClickHouse history rows use the same row component, same styling, same sorting, and + same interactions. +
    • +
  • + Keep cursor/key-based deduping so scroll-gated history does not duplicate the 100-row hot head. +
    • +
+
+ +
+

Relevant Context

+
    +
  • + Prior work in islandflow-0sa already introduced scroll hold, a 100-row options head, lazy history, + and cache-first scoped snapshots. +
    • +
  • + The current storage/types path already includes execution context fields such as execution_nbbo_*, + execution_underlying_*, and execution_iv*. +
    • +
  • + Synthetic options prints already emit some execution context; the durable fix should verify this data survives + ClickHouse writes and reads. +
    • +
  • + The UI should prefer preserved execution context in row rendering before falling back to current NBBO lookup. +
    • +
  • + Beads has related work in islandflow-biq for raw live options delivery and filter/backpressure + observability. +
    • +
+
+ +
+

Implementation Steps

+
    +
  • + Audit the existing options tape flow from ingest, ClickHouse write/read, live snapshot, history endpoint, and web + composition. +
    • +
  • + Adjust API/live semantics so valid ClickHouse history can be older than freshness thresholds without being treated + as degraded. +
    • +
  • + Add the Filter-menu view toggle for Signal and All prints, with short copy explaining + the difference. +
    • +
  • + Ensure buildOptionTapeQueryParams, live subscriptions, and /history/options all receive + the selected view consistently. +
    • +
  • + Confirm option row rendering uses preserved execution_nbbo_side, execution_underlying_spot, + and execution_iv when present. +
    • +
  • + Remove deprecated or unused history/autohydration code paths that no longer help the intended scroll-gated flow. +
    • +
  • + Add a deliberate reset path for local and VPS ClickHouse, documented as destructive and operator-confirmed. +
    • +
+
+ +
+

Risks, Limitations, and Mitigations

+
    +
  • + Risk: Resetting VPS data is destructive. Mitigation: make it a runbook or explicit + command with backup/confirmation, never automatic app startup behavior. +
    • +
  • + Risk: The signal/raw toggle could affect both options and flow filters unexpectedly. + Mitigation: test option subscriptions, history query params, and flow packet filtering separately. +
    • +
  • + Risk: Older history fetch latency could be visible at the scroll gate. Mitigation: + keep lazy loading, expose loading/error state if needed, and avoid background auto-hydration. +
    • +
  • + Risk: Prior fixes may have left overlapping history logic. Mitigation: remove unused + scaffolding only after tests cover the intended hot-cache plus ClickHouse path. +
    • +
+
+ +
+

Validation

+
    +
  • + Storage tests: fetchRecentOptionPrints and fetchOptionPrintsBefore return execution NBBO, + spot, IV, signal metadata, and raw/signal filtering correctly. +
    • +
  • + API/live tests: generic options snapshots cap at 100, scoped snapshots prefer hot cache, history preserves + next_before, and rows older than 24 hours return as valid history. +
    • +
  • + Web tests: Filter menu toggles Signal/All prints, scroll gate calls + loadOlder("options"), ClickHouse rows compose with no duplicate seam and no distinct styling, and + preserved execution context drives Spot, Side, Details, and IV display. +
    • +
  • + Validation commands: bun test packages/storage/tests/option-prints.test.ts services/api/tests/live.test.ts apps/web/app/terminal.test.ts + and bun --cwd=apps/web run build. +
    • +
+
+ +
+

Open Questions

+
    +
  • + Exact VPS reset command sequence should be confirmed against the live deployment state before execution. +
    • +
  • + Decide during implementation whether to track the reset/runbook in a new Beads issue or fold it into + islandflow-biq. +
    • +
+
+

+ Fixed assumptions: historical ClickHouse rows should be visually indistinguishable from hot-cache rows, and local + plus VPS wipe should be an operator-confirmed reset path rather than a background migration. +

+
+
+
+ +