Merge pull request #27 from dirtydishes/tape-overhaul
Options Tape Overhaul (Phase 1)
This commit is contained in:
dirtydishes
GitHub
commit
a8cc2e3875
35 changed files with 3236 additions and 527 deletions
|
|
@ -1,5 +1,8 @@
|
||||||
|
{"_type":"issue","id":"islandflow-sh1","title":"Fix live websocket stale lag and reconnect loop","description":"Investigate and fix API live consumer lag causing stale timestamps, feed-behind status, and reconnect loops. Optimize live cache persistence path, add lag telemetry/alerts, and validate in runtime.","status":"closed","priority":1,"issue_type":"bug","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-04T17:04:34Z","created_by":"dirtydishes","updated_at":"2026-05-04T17:09:44Z","started_at":"2026-05-04T17:04:38Z","closed_at":"2026-05-04T17:09:44Z","close_reason":"Completed: optimized live cache persistence path, added lag telemetry, deployed api via docker compose on di, verified ws freshness and low hotFeedLagMs","dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||||
|
{"_type":"issue","id":"islandflow-b3o","title":"Implement options tape table with execution spot","description":"Redesign OptionsPane into a dense classifier-colored table and preserve execution-time underlying spot on option prints from equity quote mid.","status":"closed","priority":1,"issue_type":"feature","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-04T04:41:59Z","created_by":"dirtydishes","updated_at":"2026-05-04T05:14:26Z","started_at":"2026-05-04T04:42:08Z","closed_at":"2026-05-04T05:14:26Z","close_reason":"Completed","dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||||
{"_type":"issue","id":"islandflow-ug1","title":"Fix false NBBO-missing badges in live Options tape","description":"Investigate and fix client-side cases where Options rows show NBBO missing/stale even when a fresh NBBO quote exists in the live nbbo map. Update rendering logic to prefer fresh quote-derived status and add regression tests.","status":"closed","priority":1,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-29T15:58:31Z","created_by":"dirtydishes","updated_at":"2026-04-29T16:01:28Z","started_at":"2026-04-29T15:58:35Z","closed_at":"2026-04-29T16:01:28Z","close_reason":"Completed","dependency_count":0,"dependent_count":0,"comment_count":0}
|
{"_type":"issue","id":"islandflow-ug1","title":"Fix false NBBO-missing badges in live Options tape","description":"Investigate and fix client-side cases where Options rows show NBBO missing/stale even when a fresh NBBO quote exists in the live nbbo map. Update rendering logic to prefer fresh quote-derived status and add regression tests.","status":"closed","priority":1,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-29T15:58:31Z","created_by":"dirtydishes","updated_at":"2026-04-29T16:01:28Z","started_at":"2026-04-29T15:58:35Z","closed_at":"2026-04-29T16:01:28Z","close_reason":"Completed","dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||||
{"_type":"issue","id":"islandflow-ayo","title":"Drop stale backlog events from live fanout","description":"Follow-up to live freshness rollout: /ws/live was still fanning out stale backlog events for freshness-gated channels, which kept tape panes in Live feed behind despite active synthetic ingest. Gate fanout and cache ingest by freshness for options/nbbo/equities/flow.","status":"closed","priority":1,"issue_type":"bug","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T21:26:39Z","created_by":"dirtydishes","updated_at":"2026-04-28T21:26:44Z","started_at":"2026-04-28T21:26:44Z","closed_at":"2026-04-28T21:26:44Z","close_reason":"Completed","dependency_count":0,"dependent_count":0,"comment_count":0}
|
{"_type":"issue","id":"islandflow-ayo","title":"Drop stale backlog events from live fanout","description":"Follow-up to live freshness rollout: /ws/live was still fanning out stale backlog events for freshness-gated channels, which kept tape panes in Live feed behind despite active synthetic ingest. Gate fanout and cache ingest by freshness for options/nbbo/equities/flow.","status":"closed","priority":1,"issue_type":"bug","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T21:26:39Z","created_by":"dirtydishes","updated_at":"2026-04-28T21:26:44Z","started_at":"2026-04-28T21:26:44Z","closed_at":"2026-04-28T21:26:44Z","close_reason":"Completed","dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||||
{"_type":"issue","id":"islandflow-0v6","title":"Fix tape freshness, NBBO coverage, pause controls, and filter popup","description":"Implement the tape fixes requested for synthetic options notional sizing, strict live freshness, live-mode pause/resume behavior, stronger NBBO snapshot coverage, and moving flow filters behind a popup. Includes server-side live cache changes, web terminal state/UI changes, and tests for synthetic pricing, live snapshot freshness/NBBO retention, and live pause/filter interactions.","status":"closed","priority":1,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T21:02:52Z","created_by":"dirtydishes","updated_at":"2026-04-28T21:13:38Z","started_at":"2026-04-28T21:02:57Z","closed_at":"2026-04-28T21:13:38Z","close_reason":"Completed","dependency_count":0,"dependent_count":0,"comment_count":0}
|
{"_type":"issue","id":"islandflow-0v6","title":"Fix tape freshness, NBBO coverage, pause controls, and filter popup","description":"Implement the tape fixes requested for synthetic options notional sizing, strict live freshness, live-mode pause/resume behavior, stronger NBBO snapshot coverage, and moving flow filters behind a popup. Includes server-side live cache changes, web terminal state/UI changes, and tests for synthetic pricing, live snapshot freshness/NBBO retention, and live pause/filter interactions.","status":"closed","priority":1,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T21:02:52Z","created_by":"dirtydishes","updated_at":"2026-04-28T21:13:38Z","started_at":"2026-04-28T21:02:57Z","closed_at":"2026-04-28T21:13:38Z","close_reason":"Completed","dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||||
{"_type":"issue","id":"islandflow-e4r","title":"Implement smart-money flow filtering and synthetic firehose modes","description":"Implement the approved multi-surface plan for named synthetic market profiles, options raw-vs-signal filtering, live/API filter contracts, Tape page client-side flow filters, firehose-readiness improvements, tests, and README updates.","status":"closed","priority":1,"issue_type":"feature","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T20:10:49Z","created_by":"dirtydishes","updated_at":"2026-04-28T20:29:29Z","started_at":"2026-04-28T20:10:53Z","closed_at":"2026-04-28T20:29:29Z","close_reason":"Implemented synthetic market profiles, options signal-path filtering, signal-aware API/replay contracts, Tape page filters, tests, and README updates. Follow-up tracked in islandflow-biq.","dependency_count":0,"dependent_count":0,"comment_count":0}
|
{"_type":"issue","id":"islandflow-e4r","title":"Implement smart-money flow filtering and synthetic firehose modes","description":"Implement the approved multi-surface plan for named synthetic market profiles, options raw-vs-signal filtering, live/API filter contracts, Tape page client-side flow filters, firehose-readiness improvements, tests, and README updates.","status":"closed","priority":1,"issue_type":"feature","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T20:10:49Z","created_by":"dirtydishes","updated_at":"2026-04-28T20:29:29Z","started_at":"2026-04-28T20:10:53Z","closed_at":"2026-04-28T20:29:29Z","close_reason":"Implemented synthetic market profiles, options signal-path filtering, signal-aware API/replay contracts, Tape page filters, tests, and README updates. Follow-up tracked in islandflow-biq.","dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||||
|
{"_type":"issue","id":"islandflow-igk","title":"Add plan mode","description":"Implement a user-facing plan mode in the application so users can switch into planning before taking action. Scope to be clarified from existing app patterns.","status":"closed","priority":2,"issue_type":"feature","owner":"dishes@dpdrm.com","created_at":"2026-05-04T04:22:37Z","created_by":"dirtydishes","updated_at":"2026-05-04T04:26:18Z","started_at":"2026-05-04T04:22:40Z","closed_at":"2026-05-04T04:26:18Z","close_reason":"Implemented as a global pi extension toggled with Shift+P","dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||||
{"_type":"issue","id":"islandflow-biq","title":"Finish raw live options delivery and filter/backpressure observability","description":"The smart-money signal path and Tape filters are in place, but the next firehose pass should finish server-side selective raw live delivery for options subscriptions and add explicit filtered-out/backpressure observability for API/web counters. This was discovered while landing islandflow-e4r.\n","status":"in_progress","priority":2,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T20:28:58Z","created_by":"dirtydishes","updated_at":"2026-04-29T03:54:12Z","started_at":"2026-04-29T03:54:12Z","dependencies":[{"issue_id":"islandflow-biq","depends_on_id":"islandflow-e4r","type":"discovered-from","created_at":"2026-04-28T16:28:58Z","created_by":"dirtydishes","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0}
|
{"_type":"issue","id":"islandflow-biq","title":"Finish raw live options delivery and filter/backpressure observability","description":"The smart-money signal path and Tape filters are in place, but the next firehose pass should finish server-side selective raw live delivery for options subscriptions and add explicit filtered-out/backpressure observability for API/web counters. This was discovered while landing islandflow-e4r.\n","status":"in_progress","priority":2,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T20:28:58Z","created_by":"dirtydishes","updated_at":"2026-04-29T03:54:12Z","started_at":"2026-04-29T03:54:12Z","dependencies":[{"issue_id":"islandflow-biq","depends_on_id":"islandflow-e4r","type":"discovered-from","created_at":"2026-04-28T16:28:58Z","created_by":"dirtydishes","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||||
|
|
|
||||||
|
|
@ -92,6 +92,7 @@ REPLAY_LOG_EVERY=1000
|
||||||
LIVE_LIMIT_OPTIONS=10000
|
LIVE_LIMIT_OPTIONS=10000
|
||||||
LIVE_LIMIT_NBBO=10000
|
LIVE_LIMIT_NBBO=10000
|
||||||
LIVE_LIMIT_EQUITIES=10000
|
LIVE_LIMIT_EQUITIES=10000
|
||||||
|
LIVE_LIMIT_EQUITY_QUOTES=10000
|
||||||
LIVE_LIMIT_EQUITY_JOINS=10000
|
LIVE_LIMIT_EQUITY_JOINS=10000
|
||||||
LIVE_LIMIT_FLOW=10000
|
LIVE_LIMIT_FLOW=10000
|
||||||
LIVE_LIMIT_CLASSIFIER_HITS=10000
|
LIVE_LIMIT_CLASSIFIER_HITS=10000
|
||||||
|
|
|
||||||
24
AGENTS.md
24
AGENTS.md
|
|
@ -44,3 +44,27 @@ bd close <id> # Complete work
|
||||||
- NEVER say "ready to push when you are" - YOU must push
|
- NEVER say "ready to push when you are" - YOU must push
|
||||||
- If push fails, resolve and retry until it succeeds
|
- If push fails, resolve and retry until it succeeds
|
||||||
<!-- END BEADS INTEGRATION -->
|
<!-- END BEADS INTEGRATION -->
|
||||||
|
|
||||||
|
## Minimal Repo Operating Instructions
|
||||||
|
|
||||||
|
This is a Bun + TypeScript monorepo for an event-sourced market-data pipeline:
|
||||||
|
- Flow: ingest services publish to NATS/JetStream, compute/candles derive events, API serves REST/WS, web consumes live/replay streams.
|
||||||
|
- Main folders: `services/*` (runtime services), `packages/*` (shared libs/types/storage), `apps/web` (Next.js UI).
|
||||||
|
- Infra dependency: local dev assumes Docker services (NATS, ClickHouse, Redis) are available.
|
||||||
|
|
||||||
|
Use these repo-specific commands:
|
||||||
|
- Install deps: `bun install`
|
||||||
|
- Start full stack: `bun run dev`
|
||||||
|
- Start infra only: `bun run dev:infra`
|
||||||
|
- Start backend services only: `bun run dev:services`
|
||||||
|
- Start web only: `bun run dev:web`
|
||||||
|
|
||||||
|
Testing and validation in this repo are Bun-first:
|
||||||
|
- Run tests: `bun test`
|
||||||
|
- Run scoped tests: `bun test services/compute/tests` (or another package/service path)
|
||||||
|
- Validate web production build when UI code changes: `bun --cwd=apps/web run build`
|
||||||
|
|
||||||
|
Working style that avoids common problems here:
|
||||||
|
- Prefer editing in the touched workspace (`services/<name>`, `packages/<name>`, `apps/web`) and keep shared contract changes in `packages/types`.
|
||||||
|
- Keep `.env` aligned with `.env.example`; adapters default to synthetic modes for local development.
|
||||||
|
- Dev runners persist child PID state in `.tmp/`; if a previous run crashed, restart via the standard `bun run dev*` commands so stale processes are cleaned up.
|
||||||
|
|
|
||||||
|
|
@ -261,6 +261,7 @@ Default `smart-money` policy rejects lower-information prints and keeps high-con
|
||||||
| `LIVE_LIMIT_OPTIONS` | `10000` | In-memory/Redis live cache depth for options channel (clamped `1..100000`). |
|
| `LIVE_LIMIT_OPTIONS` | `10000` | In-memory/Redis live cache depth for options channel (clamped `1..100000`). |
|
||||||
| `LIVE_LIMIT_NBBO` | `10000` | Live cache depth for options NBBO channel (clamped `1..100000`). |
|
| `LIVE_LIMIT_NBBO` | `10000` | Live cache depth for options NBBO channel (clamped `1..100000`). |
|
||||||
| `LIVE_LIMIT_EQUITIES` | `10000` | Live cache depth for equities channel (clamped `1..100000`). |
|
| `LIVE_LIMIT_EQUITIES` | `10000` | Live cache depth for equities channel (clamped `1..100000`). |
|
||||||
|
| `LIVE_LIMIT_EQUITY_QUOTES` | `10000` | Live cache depth for equity quotes channel (clamped `1..100000`). |
|
||||||
| `LIVE_LIMIT_EQUITY_JOINS` | `10000` | Live cache depth for equity join channel (clamped `1..100000`). |
|
| `LIVE_LIMIT_EQUITY_JOINS` | `10000` | Live cache depth for equity join channel (clamped `1..100000`). |
|
||||||
| `LIVE_LIMIT_FLOW` | `10000` | Live cache depth for flow packet channel (clamped `1..100000`). |
|
| `LIVE_LIMIT_FLOW` | `10000` | Live cache depth for flow packet channel (clamped `1..100000`). |
|
||||||
| `LIVE_LIMIT_CLASSIFIER_HITS` | `10000` | Live cache depth for classifier hits channel (clamped `1..100000`). |
|
| `LIVE_LIMIT_CLASSIFIER_HITS` | `10000` | Live cache depth for classifier hits channel (clamped `1..100000`). |
|
||||||
|
|
@ -303,7 +304,10 @@ Default `smart-money` policy rejects lower-information prints and keeps high-con
|
||||||
- `view=raw` — audit/debug path that preserves every stored print.
|
- `view=raw` — audit/debug path that preserves every stored print.
|
||||||
- The default Tape page options/packets posture is now stock-only, hides `B` / `BB`, keeps calls and puts visible, and applies in-memory min-notional controls immediately.
|
- The default Tape page options/packets posture is now stock-only, hides `B` / `BB`, keeps calls and puts visible, and applies in-memory min-notional controls immediately.
|
||||||
- Live retention uses a two-tier model:
|
- Live retention uses a two-tier model:
|
||||||
- API/Redis maintain a bounded hot cache per live generic channel.
|
- ClickHouse is durable server history; Redis is a bounded hot cache per live generic channel.
|
||||||
|
- `LIVE_LIMIT_*` controls initial snapshot/hot-cache depth, not total persisted history.
|
||||||
|
- Browser state is only a rendering window and UI preferences, not a market-data database.
|
||||||
|
- Devices connected to the same API hydrate from the same server-seen history.
|
||||||
- UI keeps a bounded hot window for rendering performance around the signal view rather than raw noise.
|
- UI keeps a bounded hot window for rendering performance around the signal view rather than raw noise.
|
||||||
- Options prints can use a deeper dedicated cap via `NEXT_PUBLIC_LIVE_HOT_WINDOW_OPTIONS` without raising every other feed.
|
- Options prints can use a deeper dedicated cap via `NEXT_PUBLIC_LIVE_HOT_WINDOW_OPTIONS` without raising every other feed.
|
||||||
- Alert/drawer evidence is pinned and hydrated by id/trace so details remain inspectable after hot-window eviction.
|
- Alert/drawer evidence is pinned and hydrated by id/trace so details remain inspectable after hot-window eviction.
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,7 @@
|
||||||
import { ChartsRoute } from "../terminal";
|
import { redirect } from "next/navigation";
|
||||||
|
|
||||||
|
export const dynamic = "force-dynamic";
|
||||||
|
|
||||||
export default function Page() {
|
export default function Page() {
|
||||||
return <ChartsRoute />;
|
redirect("/");
|
||||||
}
|
}
|
||||||
|
|
|
||||||
2
apps/web/app/frontend-cooker/frontend-cooker.module.css
Normal file
2
apps/web/app/frontend-cooker/frontend-cooker.module.css
Normal file
|
|
@ -0,0 +1,2 @@
|
||||||
|
.cookerShell{min-height:100vh;display:grid;grid-template-columns:280px 1fr;background:#080806;color:#f4efe3}.chrome{position:sticky;top:0;height:100vh;padding:18px;display:flex;flex-direction:column;gap:22px;background:#111;border-right:1px solid #333;z-index:5}.chrome p{margin:0 0 8px;color:#d6a84f;text-transform:uppercase;letter-spacing:.18em;font-size:12px}.chrome h2{margin:0 0 8px;font-family:Georgia,serif;font-size:28px;line-height:1}.chrome small,.chrome footer{color:#aaa;line-height:1.45}.chrome footer{margin-top:auto;font-size:12px}.switcher{display:grid;gap:9px}.switcher button{display:grid;grid-template-columns:28px 1fr;gap:10px;align-items:center;text-align:left;padding:10px;border:1px solid #333;border-radius:14px;background:#191919;color:#ddd;cursor:pointer;transition:.18s}.switcher button:hover,.switcher .active{transform:translateX(3px);border-color:#d6a84f;background:#272111}.switcher b{display:grid;place-items:center;width:24px;height:24px;border-radius:50%;background:#333;color:#fff}.mock{min-height:100vh;padding:28px;font-family:var(--body,serif);transition:background .25s,color .25s}.productNav{display:flex;align-items:center;gap:18px;margin-bottom:28px}.productNav strong{margin-right:auto;letter-spacing:.16em}.productNav span{opacity:.75}.productNav button,.panelHead button{border:0;border-radius:999px;padding:10px 14px;cursor:pointer;background:var(--accent);color:var(--accentText)}.hero{display:grid;grid-template-columns:minmax(0,1.25fr)360px;gap:24px;align-items:stretch}.kicker{margin:0 0 10px;color:var(--accent);letter-spacing:.18em;text-transform:uppercase;font-size:12px}.hero h1{margin:0;font-family:var(--display,Georgia,serif);font-size:clamp(42px,6vw,92px);line-height:.9;letter-spacing:-.05em;text-transform:none}.copy{max-width:680px;font-size:18px;line-height:1.5;opacity:.78}.statusCard,.metrics article,.primaryPanel,.sidePanel,.tableWrap{border:1px solid var(--line);background:var(--panel);box-shadow:var(--shadow);border-radius:var(--radius)}.statusCard{padding:24px;font-size:15px}.statusCard b{display:block;margin:28px 0 4px;font-size:48px;font-family:var(--display)}.liveDot{display:inline-block;width:10px;height:10px;border-radius:50%;background:#28d77f;box-shadow:0 0 18px #28d77f;margin-right:8px}.metrics{display:grid;grid-template-columns:repeat(4,1fr);gap:14px;margin:20px 0}.metrics article{padding:18px;font-weight:700}.workspace{display:grid;grid-template-columns:1.45fr .75fr;gap:18px}.primaryPanel,.sidePanel{padding:18px}.panelHead{display:flex;justify-content:space-between;align-items:center;margin-bottom:12px}.panelHead h2,.sidePanel h2{margin:0;font-family:var(--display);font-size:24px}.chart{height:330px;position:relative;display:flex;align-items:flex-end;gap:1.8%;padding:24px;overflow:hidden;background:var(--chart);border-radius:calc(var(--radius) - 6px)}.chart i{flex:1;background:var(--bar);border-radius:99px 99px 0 0;animation:rise .7s both}.chart b{position:absolute;left:5%;right:5%;top:45%;height:3px;background:var(--accent);transform:rotate(-8deg);box-shadow:0 0 24px var(--accent)}.alert,.empty,.loading,.error{padding:14px;margin-top:12px;border-radius:14px;border:1px solid var(--line);background:rgba(255,255,255,.06)}.loading{background:repeating-linear-gradient(90deg,rgba(255,255,255,.08),rgba(255,255,255,.08) 12px,transparent 12px,transparent 24px)}.error{color:#ffb1a8}.tableWrap{margin-top:18px;overflow:auto}.tableWrap table{width:100%;border-collapse:collapse}.tableWrap th,.tableWrap td{padding:14px 16px;border-bottom:1px solid var(--line);text-align:left}.tableWrap tr:hover td{background:rgba(255,255,255,.08)}@keyframes rise{from{transform:scaleY(.25);opacity:.2}to{transform:scaleY(1);opacity:1}}
|
||||||
|
.pit{--display:Impact,Haettenschweiler,'Arial Narrow Bold',sans-serif;--body:'Trebuchet MS',sans-serif;--accent:#ffb000;--accentText:#1a0c00;--line:#3e321b;--panel:#16120b;--chart:#080602;--bar:linear-gradient(#ffcf52,#b35b00);--radius:4px;--shadow:inset 0 0 0 1px #000,0 18px 0 rgba(0,0,0,.25);background:radial-gradient(circle at 70% -10%,#5d2500,transparent 35%),#0b0905;color:#fff0c9}.pit .productNav{border-bottom:6px solid #ffb000;padding-bottom:12px}.atlas{--display:'Didot','Bodoni 72',serif;--body:'Avenir Next',Verdana,sans-serif;--accent:#00b894;--accentText:#001b15;--line:rgba(16,80,70,.28);--panel:rgba(235,255,250,.68);--chart:linear-gradient(135deg,#dff9ef,#b8d6e5);--bar:#0b8874;--radius:28px;--shadow:0 30px 80px rgba(30,90,90,.16);background:linear-gradient(120deg,#eef8f3,#cbdde1);color:#17322f}.ledger{--display:'Iowan Old Style',Georgia,serif;--body:Georgia,serif;--accent:#8b3f1f;--accentText:#fff8ee;--line:#d8c7a9;--panel:#fffaf0;--chart:#f7ecd8;--bar:#1f3f35;--radius:0;--shadow:8px 8px 0 #d8c7a9;background:#f4ead8;color:#24190f}.ledger.mock,.ledger .tableWrap table{font-size:17px}.neon{--display:'Courier New',monospace;--body:'Courier New',monospace;--accent:#39ff14;--accentText:#001400;--line:#263cff;--panel:rgba(4,8,28,.82);--chart:#03040f;--bar:linear-gradient(#ff2bd6,#263cff);--radius:18px;--shadow:0 0 32px rgba(57,255,20,.2),inset 0 0 24px rgba(38,60,255,.18);background:linear-gradient(180deg,#050718,#110014);color:#d6fff4}.neon .hero h1{text-shadow:0 0 20px #ff2bd6}.paper{--display:'Franklin Gothic Medium','Arial Narrow',sans-serif;--body:'Times New Roman',serif;--accent:#c5281c;--accentText:#fff;--line:#111;--panel:#f8f1df;--chart:repeating-linear-gradient(0deg,#efe4cc,#efe4cc 14px,#e2d3b8 15px);--bar:#111;--radius:0;--shadow:none;background:#eee2c8;color:#111}.paper .productNav,.paper .hero,.paper .metrics{border-bottom:3px double #111;padding-bottom:14px}@media(max-width:900px){.cookerShell{grid-template-columns:1fr}.chrome{height:auto;position:relative}.switcher{grid-template-columns:repeat(2,1fr)}.hero,.workspace,.metrics{grid-template-columns:1fr}.productNav{flex-wrap:wrap}.mock{padding:18px}}
|
||||||
55
apps/web/app/frontend-cooker/page.tsx
Normal file
55
apps/web/app/frontend-cooker/page.tsx
Normal file
|
|
@ -0,0 +1,55 @@
|
||||||
|
"use client";
|
||||||
|
|
||||||
|
import { useMemo, useState } from "react";
|
||||||
|
import styles from "./frontend-cooker.module.css";
|
||||||
|
|
||||||
|
const variations = [
|
||||||
|
{ id: "pit", name: "Open-Outcry Pit", rationale: "A loud exchange-floor command center optimized for immediate threat recognition and dense scan paths." },
|
||||||
|
{ id: "atlas", name: "Glass Atlas", rationale: "A calm geospatial intelligence room that makes flow feel mapped, layered, and explorable." },
|
||||||
|
{ id: "ledger", name: "Ivory Ledger", rationale: "A refined analyst notebook with editorial hierarchy for slower, higher-confidence review." },
|
||||||
|
{ id: "neon", name: "Neon Underpass", rationale: "A kinetic cyberpunk tape for traders who want momentum, heat, and speed above all." },
|
||||||
|
{ id: "paper", name: "Signal Gazette", rationale: "A newspaper-like briefing that turns raw options activity into a morning intelligence digest." }
|
||||||
|
];
|
||||||
|
|
||||||
|
const flowRows = [
|
||||||
|
["NVDA", "910C", "05-17", "$4.8M", "AA", "+92%", "Sweep"],
|
||||||
|
["TSLA", "175P", "05-10", "$2.1M", "BB", "−68%", "ISO"],
|
||||||
|
["AAPL", "205C", "06-21", "$1.4M", "A", "+41%", "Block"],
|
||||||
|
["SPY", "520P", "05-03", "$8.7M", "B", "−53%", "Split"],
|
||||||
|
["AMD", "162C", "05-24", "$910K", "AA", "+77%", "Sweep"]
|
||||||
|
];
|
||||||
|
|
||||||
|
function MiniChart({ variant }: { variant: string }) {
|
||||||
|
return <div className={`${styles.chart} ${styles[`chart_${variant}`]}`} aria-label="Mock price and flow chart">
|
||||||
|
{Array.from({ length: 22 }).map((_, i) => <i key={i} style={{ height: `${24 + ((i * 17) % 58)}%`, animationDelay: `${i * 35}ms` }} />)}
|
||||||
|
<b />
|
||||||
|
</div>;
|
||||||
|
}
|
||||||
|
|
||||||
|
function AppMock({ id }: { id: string }) {
|
||||||
|
return <main className={`${styles.mock} ${styles[id]}`}>
|
||||||
|
<nav className={styles.productNav}>
|
||||||
|
<strong>ISLANDFLOW</strong><span>Overview</span><span>Live Tape</span><span>Signals</span><span>Replay</span><button>Filter Flow</button>
|
||||||
|
</nav>
|
||||||
|
<section className={styles.hero}>
|
||||||
|
<div><p className={styles.kicker}>Live Options Intelligence</p><h1>Unusual flow surfaced before the crowd.</h1><p className={styles.copy}>Representative redesign of the IslandFlow terminal: live status, option sweeps, inferred dark activity, classifier hits, and replay controls.</p></div>
|
||||||
|
<div className={styles.statusCard}><span className={styles.liveDot}/>Connected · 1,284 msgs/min<br/><b>$42.6M</b><small> premium tracked in active window</small></div>
|
||||||
|
</section>
|
||||||
|
<section className={styles.metrics}>{["Alert score 87", "Bullish 62%", "Dark pool 14", "Stale feeds 0"].map(x => <article key={x}>{x}</article>)}</section>
|
||||||
|
<section className={styles.workspace}>
|
||||||
|
<div className={styles.primaryPanel}><div className={styles.panelHead}><h2>Flow Radar</h2><button>Pause Tape</button></div><MiniChart variant={id}/></div>
|
||||||
|
<div className={styles.sidePanel}><h2>Classifier Hits</h2><div className={styles.alert}>High conviction: NVDA call sweep above ask with confirming equity print.</div><div className={styles.empty}>Empty state: no stale NBBO quotes in the last 15s.</div><div className={styles.loading}>Loading replay baseline…</div><div className={styles.error}>Error state: dark inference source delayed.</div></div>
|
||||||
|
</section>
|
||||||
|
<section className={styles.tableWrap}><table><thead><tr>{["Ticker", "Contract", "Expiry", "Notional", "Side", "Delta", "Condition"].map(h => <th key={h}>{h}</th>)}</tr></thead><tbody>{flowRows.map((r) => <tr key={r.join("")}>{r.map((c, i) => <td key={i}>{c}</td>)}</tr>)}</tbody></table></section>
|
||||||
|
</main>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export default function FrontendCooker() {
|
||||||
|
const [active, setActive] = useState(0);
|
||||||
|
const current = variations[active];
|
||||||
|
const nav = useMemo(() => variations.slice(0, 5), []);
|
||||||
|
return <div className={styles.cookerShell}>
|
||||||
|
<aside className={styles.chrome}><div><p>Frontend Cooker</p><h2>{current.name}</h2><small>{current.rationale}</small></div><div className={styles.switcher}>{nav.map((v, i) => <button key={v.id} className={i === active ? styles.active : ""} onClick={() => setActive(i)}><b>{i + 1}</b><span>{v.name}</span></button>)}</div><footer>Target: IslandFlow trading terminal overview</footer></aside>
|
||||||
|
<AppMock id={current.id}/>
|
||||||
|
</div>;
|
||||||
|
}
|
||||||
|
|
@ -19,7 +19,7 @@
|
||||||
--blue: #4da3ff;
|
--blue: #4da3ff;
|
||||||
--blue-soft: rgba(77, 163, 255, 0.14);
|
--blue-soft: rgba(77, 163, 255, 0.14);
|
||||||
--rail-width: 236px;
|
--rail-width: 236px;
|
||||||
--topbar-height: 76px;
|
--topbar-height: 64px;
|
||||||
}
|
}
|
||||||
|
|
||||||
* {
|
* {
|
||||||
|
|
@ -166,38 +166,16 @@ input {
|
||||||
position: sticky;
|
position: sticky;
|
||||||
top: 0;
|
top: 0;
|
||||||
z-index: 20;
|
z-index: 20;
|
||||||
display: grid;
|
display: flex;
|
||||||
grid-template-columns: minmax(0, 1fr) auto;
|
align-items: center;
|
||||||
align-items: end;
|
justify-content: flex-end;
|
||||||
gap: 18px 24px;
|
gap: 12px;
|
||||||
padding: 16px 24px 14px;
|
padding: 10px 20px;
|
||||||
background: rgba(7, 10, 14, 0.92);
|
background: rgba(7, 10, 14, 0.92);
|
||||||
backdrop-filter: blur(12px);
|
backdrop-filter: blur(12px);
|
||||||
border-bottom: 1px solid var(--border);
|
border-bottom: 1px solid var(--border);
|
||||||
}
|
}
|
||||||
|
|
||||||
.feed-status-bar {
|
|
||||||
display: flex;
|
|
||||||
align-items: center;
|
|
||||||
gap: 10px;
|
|
||||||
flex-wrap: wrap;
|
|
||||||
min-width: 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
.feed-status {
|
|
||||||
display: inline-flex;
|
|
||||||
align-items: center;
|
|
||||||
gap: 8px;
|
|
||||||
padding: 8px 10px;
|
|
||||||
border-radius: 999px;
|
|
||||||
border: 1px solid var(--border);
|
|
||||||
background: rgba(255, 255, 255, 0.02);
|
|
||||||
color: var(--text-dim);
|
|
||||||
font-family: var(--font-mono), monospace;
|
|
||||||
font-size: 0.75rem;
|
|
||||||
}
|
|
||||||
|
|
||||||
.feed-status-dot,
|
|
||||||
.status-dot,
|
.status-dot,
|
||||||
.chart-dot {
|
.chart-dot {
|
||||||
width: 8px;
|
width: 8px;
|
||||||
|
|
@ -206,28 +184,24 @@ input {
|
||||||
background: var(--text-faint);
|
background: var(--text-faint);
|
||||||
}
|
}
|
||||||
|
|
||||||
.feed-status-connected .feed-status-dot,
|
|
||||||
.status-connected .status-dot,
|
.status-connected .status-dot,
|
||||||
.chart-status-connected .chart-dot {
|
.chart-status-connected .chart-dot {
|
||||||
background: var(--green);
|
background: var(--green);
|
||||||
box-shadow: 0 0 0 4px rgba(37, 193, 122, 0.14);
|
box-shadow: 0 0 0 4px rgba(37, 193, 122, 0.14);
|
||||||
}
|
}
|
||||||
|
|
||||||
.feed-status-connecting .feed-status-dot,
|
|
||||||
.chart-status-connecting .chart-dot {
|
.chart-status-connecting .chart-dot {
|
||||||
background: var(--accent);
|
background: var(--accent);
|
||||||
box-shadow: 0 0 0 4px rgba(245, 166, 35, 0.12);
|
box-shadow: 0 0 0 4px rgba(245, 166, 35, 0.12);
|
||||||
animation: pulse 1.3s ease-in-out infinite;
|
animation: pulse 1.3s ease-in-out infinite;
|
||||||
}
|
}
|
||||||
|
|
||||||
.feed-status-stale .feed-status-dot,
|
|
||||||
.status-stale .status-dot,
|
.status-stale .status-dot,
|
||||||
.chart-status-stale .chart-dot {
|
.chart-status-stale .chart-dot {
|
||||||
background: var(--accent);
|
background: var(--accent);
|
||||||
box-shadow: 0 0 0 4px rgba(245, 166, 35, 0.18);
|
box-shadow: 0 0 0 4px rgba(245, 166, 35, 0.18);
|
||||||
}
|
}
|
||||||
|
|
||||||
.feed-status-disconnected .feed-status-dot,
|
|
||||||
.status-disconnected .status-dot,
|
.status-disconnected .status-dot,
|
||||||
.chart-status-disconnected .chart-dot {
|
.chart-status-disconnected .chart-dot {
|
||||||
background: var(--red);
|
background: var(--red);
|
||||||
|
|
@ -236,33 +210,37 @@ input {
|
||||||
|
|
||||||
.terminal-topbar-actions {
|
.terminal-topbar-actions {
|
||||||
display: flex;
|
display: flex;
|
||||||
align-items: flex-end;
|
align-items: center;
|
||||||
justify-content: flex-end;
|
justify-content: flex-end;
|
||||||
gap: 20px;
|
gap: 12px;
|
||||||
min-width: 0;
|
min-width: 0;
|
||||||
|
width: auto;
|
||||||
|
margin-left: auto;
|
||||||
}
|
}
|
||||||
|
|
||||||
.terminal-topbar-controls {
|
.terminal-topbar-controls {
|
||||||
display: flex;
|
display: flex;
|
||||||
align-items: flex-end;
|
align-items: center;
|
||||||
justify-content: flex-end;
|
justify-content: flex-end;
|
||||||
gap: 12px;
|
gap: 10px;
|
||||||
min-width: 0;
|
min-width: 0;
|
||||||
|
flex: 0 1 auto;
|
||||||
}
|
}
|
||||||
|
|
||||||
.terminal-topbar-mode {
|
.terminal-topbar-mode {
|
||||||
display: flex;
|
display: flex;
|
||||||
align-items: flex-end;
|
align-items: center;
|
||||||
justify-content: flex-end;
|
justify-content: flex-end;
|
||||||
flex: 0 0 auto;
|
flex: 0 0 auto;
|
||||||
|
margin-left: auto;
|
||||||
}
|
}
|
||||||
|
|
||||||
.terminal-filter {
|
.terminal-filter {
|
||||||
display: flex;
|
display: flex;
|
||||||
flex-direction: column;
|
flex-direction: column;
|
||||||
gap: 6px;
|
gap: 4px;
|
||||||
min-width: clamp(280px, 26vw, 420px);
|
min-width: clamp(220px, 24vw, 360px);
|
||||||
flex: 0 1 clamp(280px, 26vw, 420px);
|
flex: 0 1 clamp(220px, 24vw, 360px);
|
||||||
}
|
}
|
||||||
|
|
||||||
.terminal-filter-label {
|
.terminal-filter-label {
|
||||||
|
|
@ -274,7 +252,7 @@ input {
|
||||||
position: relative;
|
position: relative;
|
||||||
display: flex;
|
display: flex;
|
||||||
align-items: center;
|
align-items: center;
|
||||||
min-height: 36px;
|
min-height: 32px;
|
||||||
}
|
}
|
||||||
|
|
||||||
.terminal-filter-field::before,
|
.terminal-filter-field::before,
|
||||||
|
|
@ -308,7 +286,7 @@ input {
|
||||||
.terminal-input {
|
.terminal-input {
|
||||||
min-width: 0;
|
min-width: 0;
|
||||||
width: 100%;
|
width: 100%;
|
||||||
padding: 0 0 8px;
|
padding: 0 0 6px;
|
||||||
border: none;
|
border: none;
|
||||||
border-radius: 0;
|
border-radius: 0;
|
||||||
background: transparent;
|
background: transparent;
|
||||||
|
|
@ -358,8 +336,8 @@ input {
|
||||||
.overlay-toggle,
|
.overlay-toggle,
|
||||||
.drawer-close {
|
.drawer-close {
|
||||||
border: 1px solid var(--border);
|
border: 1px solid var(--border);
|
||||||
border-radius: 10px;
|
border-radius: 8px;
|
||||||
padding: 10px 12px;
|
padding: 8px 10px;
|
||||||
background: rgba(255, 255, 255, 0.03);
|
background: rgba(255, 255, 255, 0.03);
|
||||||
color: var(--text);
|
color: var(--text);
|
||||||
cursor: pointer;
|
cursor: pointer;
|
||||||
|
|
@ -384,6 +362,59 @@ input {
|
||||||
color: #ffd89a;
|
color: #ffd89a;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.instrument-focus-chip {
|
||||||
|
display: inline-flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 8px;
|
||||||
|
min-height: 32px;
|
||||||
|
max-width: min(360px, 32vw);
|
||||||
|
padding: 5px 8px 5px 10px;
|
||||||
|
border: 1px solid rgba(255, 216, 154, 0.34);
|
||||||
|
border-radius: 8px;
|
||||||
|
background: rgba(245, 166, 35, 0.08);
|
||||||
|
color: #ffe2aa;
|
||||||
|
font-family: var(--font-mono), monospace;
|
||||||
|
font-size: 0.72rem;
|
||||||
|
font-weight: 700;
|
||||||
|
}
|
||||||
|
|
||||||
|
.instrument-focus-chip span {
|
||||||
|
overflow: hidden;
|
||||||
|
text-overflow: ellipsis;
|
||||||
|
white-space: nowrap;
|
||||||
|
}
|
||||||
|
|
||||||
|
.instrument-focus-chip button,
|
||||||
|
.instrument-cell-button {
|
||||||
|
border: 0;
|
||||||
|
background: transparent;
|
||||||
|
color: inherit;
|
||||||
|
font: inherit;
|
||||||
|
cursor: pointer;
|
||||||
|
}
|
||||||
|
|
||||||
|
.instrument-focus-chip button {
|
||||||
|
padding: 4px 6px;
|
||||||
|
color: var(--text-muted);
|
||||||
|
text-transform: uppercase;
|
||||||
|
letter-spacing: 0.08em;
|
||||||
|
font-size: 0.62rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.instrument-cell-button {
|
||||||
|
padding: 0;
|
||||||
|
text-align: inherit;
|
||||||
|
text-decoration: underline;
|
||||||
|
text-decoration-color: rgba(255, 216, 154, 0.36);
|
||||||
|
text-underline-offset: 3px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.instrument-cell-button:hover,
|
||||||
|
.instrument-cell-button:focus-visible {
|
||||||
|
color: #ffd89a;
|
||||||
|
outline: none;
|
||||||
|
}
|
||||||
|
|
||||||
.pause-button {
|
.pause-button {
|
||||||
padding: 7px 10px;
|
padding: 7px 10px;
|
||||||
font-size: 0.66rem;
|
font-size: 0.66rem;
|
||||||
|
|
@ -391,7 +422,7 @@ input {
|
||||||
|
|
||||||
.terminal-content {
|
.terminal-content {
|
||||||
min-width: 0;
|
min-width: 0;
|
||||||
padding: 34px 24px 28px;
|
padding: 24px 24px 24px;
|
||||||
}
|
}
|
||||||
|
|
||||||
.page-shell {
|
.page-shell {
|
||||||
|
|
@ -427,6 +458,27 @@ h3 {
|
||||||
position: relative;
|
position: relative;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.contract-filter-button {
|
||||||
|
display: inline-flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: center;
|
||||||
|
min-height: 32px;
|
||||||
|
min-width: 0;
|
||||||
|
max-width: min(440px, 42vw);
|
||||||
|
}
|
||||||
|
|
||||||
|
.contract-filter-button-label {
|
||||||
|
overflow: hidden;
|
||||||
|
text-overflow: ellipsis;
|
||||||
|
white-space: nowrap;
|
||||||
|
}
|
||||||
|
|
||||||
|
.contract-filter-button.is-active {
|
||||||
|
border-color: rgba(245, 166, 35, 0.55);
|
||||||
|
background: linear-gradient(180deg, rgba(245, 166, 35, 0.18), rgba(245, 166, 35, 0.07));
|
||||||
|
color: #ffe2aa;
|
||||||
|
}
|
||||||
|
|
||||||
.flow-filter-popover {
|
.flow-filter-popover {
|
||||||
position: relative;
|
position: relative;
|
||||||
}
|
}
|
||||||
|
|
@ -564,11 +616,10 @@ h3 {
|
||||||
color: #ffe4b3;
|
color: #ffe4b3;
|
||||||
}
|
}
|
||||||
|
|
||||||
.overview-strip,
|
|
||||||
.replay-matrix {
|
.replay-matrix {
|
||||||
display: grid;
|
display: grid;
|
||||||
gap: 12px;
|
gap: 12px;
|
||||||
grid-template-columns: repeat(6, minmax(0, 1fr));
|
grid-template-columns: repeat(4, minmax(0, 1fr));
|
||||||
}
|
}
|
||||||
|
|
||||||
.overview-cell {
|
.overview-cell {
|
||||||
|
|
@ -585,8 +636,8 @@ h3 {
|
||||||
align-items: stretch;
|
align-items: stretch;
|
||||||
}
|
}
|
||||||
|
|
||||||
.page-grid-overview {
|
.page-grid-home {
|
||||||
grid-template-columns: repeat(3, minmax(0, 1fr));
|
grid-template-columns: minmax(0, 2fr) minmax(320px, 1fr);
|
||||||
}
|
}
|
||||||
|
|
||||||
.page-grid-tape {
|
.page-grid-tape {
|
||||||
|
|
@ -608,7 +659,7 @@ h3 {
|
||||||
grid-template-columns: repeat(3, minmax(0, 1fr));
|
grid-template-columns: repeat(3, minmax(0, 1fr));
|
||||||
}
|
}
|
||||||
|
|
||||||
.page-grid-overview > :nth-child(1),
|
.page-grid-home > :nth-child(3),
|
||||||
.page-grid-tape > :nth-child(1),
|
.page-grid-tape > :nth-child(1),
|
||||||
.page-grid-replay > :nth-child(1) {
|
.page-grid-replay > :nth-child(1) {
|
||||||
grid-column: 1 / -1;
|
grid-column: 1 / -1;
|
||||||
|
|
@ -784,11 +835,26 @@ h3 {
|
||||||
}
|
}
|
||||||
|
|
||||||
.missed-count {
|
.missed-count {
|
||||||
width: 86px;
|
display: inline-flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: flex-start;
|
||||||
|
max-width: 0;
|
||||||
|
overflow: hidden;
|
||||||
|
white-space: nowrap;
|
||||||
font-size: 0.72rem;
|
font-size: 0.72rem;
|
||||||
color: var(--accent);
|
color: var(--accent);
|
||||||
text-align: right;
|
opacity: 0;
|
||||||
white-space: nowrap;
|
transform: translateX(6px);
|
||||||
|
transition:
|
||||||
|
max-width 180ms ease,
|
||||||
|
opacity 140ms ease,
|
||||||
|
transform 180ms ease;
|
||||||
|
}
|
||||||
|
|
||||||
|
.missed-count-visible {
|
||||||
|
max-width: 92px;
|
||||||
|
opacity: 1;
|
||||||
|
transform: translateX(0);
|
||||||
}
|
}
|
||||||
|
|
||||||
.list {
|
.list {
|
||||||
|
|
@ -811,7 +877,12 @@ h3 {
|
||||||
max-height: 260px;
|
max-height: 260px;
|
||||||
}
|
}
|
||||||
|
|
||||||
.page-grid-overview > :not(:first-child),
|
.page-grid-home > :nth-child(1),
|
||||||
|
.page-grid-home > :nth-child(2) {
|
||||||
|
height: clamp(430px, 56vh, 760px);
|
||||||
|
}
|
||||||
|
|
||||||
|
.page-grid-home > :nth-child(3),
|
||||||
.page-grid-replay > :not(:first-child) {
|
.page-grid-replay > :not(:first-child) {
|
||||||
height: clamp(430px, 58vh, 760px);
|
height: clamp(430px, 58vh, 760px);
|
||||||
}
|
}
|
||||||
|
|
@ -873,6 +944,287 @@ h3 {
|
||||||
background: linear-gradient(180deg, rgba(245, 166, 35, 0.07), rgba(255, 255, 255, 0.018));
|
background: linear-gradient(180deg, rgba(245, 166, 35, 0.07), rgba(255, 255, 255, 0.018));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.data-table-shell,
|
||||||
|
.options-table-wrap {
|
||||||
|
display: flex;
|
||||||
|
flex: 1 1 auto;
|
||||||
|
min-height: 0;
|
||||||
|
flex-direction: column;
|
||||||
|
overflow: hidden;
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table-wrap {
|
||||||
|
flex: 1 1 auto;
|
||||||
|
min-height: 0;
|
||||||
|
overflow: auto;
|
||||||
|
border-top: 1px solid var(--border);
|
||||||
|
border-bottom: 1px solid var(--border);
|
||||||
|
background: rgba(5, 8, 12, 0.42);
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table {
|
||||||
|
display: block;
|
||||||
|
min-width: 980px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table-options {
|
||||||
|
min-width: 1280px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table-equities {
|
||||||
|
min-width: 660px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table-flow {
|
||||||
|
min-width: 1260px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table-alerts {
|
||||||
|
min-width: 900px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table-classifier {
|
||||||
|
min-width: 760px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table-dark {
|
||||||
|
min-width: 820px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table-head,
|
||||||
|
.data-table-row {
|
||||||
|
display: grid;
|
||||||
|
align-items: center;
|
||||||
|
column-gap: 8px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table-head {
|
||||||
|
position: sticky;
|
||||||
|
top: 0;
|
||||||
|
z-index: 2;
|
||||||
|
min-height: 30px;
|
||||||
|
padding: 0 10px;
|
||||||
|
border-bottom: 1px solid rgba(255, 255, 255, 0.095);
|
||||||
|
background: rgba(8, 11, 16, 0.98);
|
||||||
|
color: var(--text-faint);
|
||||||
|
font-size: 0.64rem;
|
||||||
|
font-weight: 700;
|
||||||
|
letter-spacing: 0.08em;
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table-row {
|
||||||
|
width: 100%;
|
||||||
|
min-height: 40px;
|
||||||
|
padding: 0 10px;
|
||||||
|
border: 0;
|
||||||
|
border-bottom: 1px solid rgba(255, 255, 255, 0.055);
|
||||||
|
background: rgba(255, 255, 255, 0.008);
|
||||||
|
color: inherit;
|
||||||
|
font: inherit;
|
||||||
|
text-align: left;
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table-row:nth-child(even) {
|
||||||
|
background: rgba(255, 255, 255, 0.022);
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table-row:hover,
|
||||||
|
.data-table-row:focus-visible {
|
||||||
|
outline: none;
|
||||||
|
background: rgba(245, 166, 35, 0.055);
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table-row-button {
|
||||||
|
cursor: pointer;
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table-row-options {
|
||||||
|
min-height: 36px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table-row-equities {
|
||||||
|
min-height: 34px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table-row-flow,
|
||||||
|
.data-table-row-alerts,
|
||||||
|
.data-table-row-classifier,
|
||||||
|
.data-table-row-dark {
|
||||||
|
min-height: 44px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table-row-classified {
|
||||||
|
background:
|
||||||
|
linear-gradient(90deg, rgba(var(--classifier-rgb, 192, 200, 210), calc(0.02 + var(--classifier-intensity, 0) * 0.12)), transparent 62%),
|
||||||
|
rgba(255, 255, 255, 0.008);
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table-row-classified:hover,
|
||||||
|
.data-table-row-classified:focus-visible {
|
||||||
|
background:
|
||||||
|
linear-gradient(90deg, rgba(var(--classifier-rgb, 192, 200, 210), calc(0.04 + var(--classifier-intensity, 0) * 0.18)), transparent 68%),
|
||||||
|
rgba(245, 166, 35, 0.04);
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table-row-classified.is-classified {
|
||||||
|
border-left: 3px solid rgba(var(--classifier-rgb), calc(0.35 + var(--classifier-intensity) * 0.45));
|
||||||
|
padding-left: 7px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table-row-warn,
|
||||||
|
.data-table-row-severity-high,
|
||||||
|
.data-table-row-direction-bearish {
|
||||||
|
border-left: 3px solid rgba(255, 107, 95, 0.58);
|
||||||
|
padding-left: 7px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table-row-severity-medium,
|
||||||
|
.data-table-row-direction-neutral {
|
||||||
|
border-left: 3px solid rgba(77, 163, 255, 0.46);
|
||||||
|
padding-left: 7px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table-row-severity-low,
|
||||||
|
.data-table-row-direction-bullish {
|
||||||
|
border-left: 3px solid rgba(37, 193, 122, 0.5);
|
||||||
|
padding-left: 7px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table-options .data-table-head,
|
||||||
|
.data-table-options .data-table-row {
|
||||||
|
grid-template-columns: minmax(72px, 0.8fr) minmax(50px, 0.55fr) minmax(64px, 0.7fr) minmax(58px, 0.6fr) minmax(34px, 0.35fr) minmax(62px, 0.65fr) minmax(104px, 1fr) minmax(54px, 0.55fr) minmax(66px, 0.7fr) minmax(48px, 0.5fr) minmax(42px, 0.45fr) minmax(92px, 0.9fr);
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table-equities .data-table-head,
|
||||||
|
.data-table-equities .data-table-row {
|
||||||
|
grid-template-columns: minmax(76px, 0.9fr) minmax(70px, 0.8fr) minmax(76px, 0.8fr) minmax(70px, 0.75fr) minmax(80px, 0.8fr) minmax(76px, 0.75fr);
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table-flow .data-table-head,
|
||||||
|
.data-table-flow .data-table-row {
|
||||||
|
grid-template-columns: minmax(148px, 1.1fr) minmax(180px, 1.4fr) minmax(62px, 0.45fr) minmax(70px, 0.5fr) minmax(88px, 0.7fr) minmax(74px, 0.55fr) minmax(132px, 1fr) minmax(110px, 0.8fr) minmax(210px, 1.6fr);
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table-alerts .data-table-head,
|
||||||
|
.data-table-alerts .data-table-row {
|
||||||
|
grid-template-columns: minmax(76px, 0.75fr) minmax(170px, 1.4fr) minmax(52px, 0.45fr) minmax(58px, 0.45fr) minmax(52px, 0.4fr) minmax(66px, 0.55fr) minmax(260px, 2fr);
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table-classifier .data-table-head,
|
||||||
|
.data-table-classifier .data-table-row {
|
||||||
|
grid-template-columns: minmax(76px, 0.75fr) minmax(180px, 1.45fr) minmax(70px, 0.6fr) minmax(74px, 0.65fr) minmax(300px, 2.2fr);
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table-dark .data-table-head,
|
||||||
|
.data-table-dark .data-table-row {
|
||||||
|
grid-template-columns: minmax(76px, 0.75fr) minmax(170px, 1.35fr) minmax(76px, 0.65fr) minmax(74px, 0.65fr) minmax(74px, 0.65fr) minmax(260px, 2fr);
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table-cell {
|
||||||
|
min-width: 0;
|
||||||
|
overflow: hidden;
|
||||||
|
text-overflow: ellipsis;
|
||||||
|
white-space: nowrap;
|
||||||
|
font-size: 0.72rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table-cell-number {
|
||||||
|
font-family: var(--font-mono), monospace;
|
||||||
|
font-variant-numeric: tabular-nums;
|
||||||
|
}
|
||||||
|
|
||||||
|
.data-table-spacer {
|
||||||
|
min-width: 100%;
|
||||||
|
pointer-events: none;
|
||||||
|
}
|
||||||
|
|
||||||
|
.options-table {
|
||||||
|
display: flex;
|
||||||
|
min-height: 0;
|
||||||
|
flex: 1 1 auto;
|
||||||
|
flex-direction: column;
|
||||||
|
overflow: hidden;
|
||||||
|
}
|
||||||
|
|
||||||
|
.options-table-head,
|
||||||
|
.options-table-row {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: minmax(72px, 0.8fr) minmax(50px, 0.55fr) minmax(64px, 0.7fr) minmax(58px, 0.6fr) minmax(34px, 0.35fr) minmax(62px, 0.65fr) minmax(104px, 1fr) minmax(54px, 0.55fr) minmax(66px, 0.7fr) minmax(48px, 0.5fr) minmax(42px, 0.45fr) minmax(92px, 0.9fr);
|
||||||
|
align-items: center;
|
||||||
|
column-gap: 8px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.options-table-head {
|
||||||
|
flex: 0 0 auto;
|
||||||
|
height: 30px;
|
||||||
|
padding: 0 8px;
|
||||||
|
border-bottom: 1px solid var(--border);
|
||||||
|
background: rgba(8, 11, 16, 0.98);
|
||||||
|
color: var(--muted);
|
||||||
|
font-size: 0.64rem;
|
||||||
|
font-weight: 700;
|
||||||
|
letter-spacing: 0.08em;
|
||||||
|
}
|
||||||
|
|
||||||
|
.options-table-body {
|
||||||
|
flex: 1 1 auto;
|
||||||
|
min-height: 0;
|
||||||
|
overflow-y: auto;
|
||||||
|
overflow-x: hidden;
|
||||||
|
}
|
||||||
|
|
||||||
|
.options-table-row {
|
||||||
|
width: 100%;
|
||||||
|
min-height: 34px;
|
||||||
|
padding: 0 8px;
|
||||||
|
border: 0;
|
||||||
|
border-bottom: 1px solid rgba(255, 255, 255, 0.055);
|
||||||
|
background:
|
||||||
|
linear-gradient(90deg, rgba(var(--classifier-rgb, 192, 200, 210), calc(0.02 + var(--classifier-intensity, 0) * 0.12)), transparent 62%),
|
||||||
|
rgba(255, 255, 255, 0.012);
|
||||||
|
color: inherit;
|
||||||
|
font: inherit;
|
||||||
|
text-align: left;
|
||||||
|
}
|
||||||
|
|
||||||
|
.options-table-row:hover,
|
||||||
|
.options-table-row:focus-visible {
|
||||||
|
outline: none;
|
||||||
|
background:
|
||||||
|
linear-gradient(90deg, rgba(var(--classifier-rgb, 192, 200, 210), calc(0.04 + var(--classifier-intensity, 0) * 0.18)), transparent 68%),
|
||||||
|
rgba(255, 255, 255, 0.03);
|
||||||
|
}
|
||||||
|
|
||||||
|
.options-table-row.is-classified {
|
||||||
|
cursor: pointer;
|
||||||
|
border-left: 3px solid rgba(var(--classifier-rgb), calc(0.35 + var(--classifier-intensity) * 0.45));
|
||||||
|
padding-left: 7px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.options-table-row > span {
|
||||||
|
min-width: 0;
|
||||||
|
overflow: hidden;
|
||||||
|
text-overflow: ellipsis;
|
||||||
|
white-space: nowrap;
|
||||||
|
font-size: 0.72rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.mono {
|
||||||
|
font-variant-numeric: tabular-nums;
|
||||||
|
}
|
||||||
|
|
||||||
|
.classifier-green { --classifier-rgb: 37, 193, 122; }
|
||||||
|
.classifier-red { --classifier-rgb: 255, 107, 95; }
|
||||||
|
.classifier-amber { --classifier-rgb: 245, 166, 35; }
|
||||||
|
.classifier-copper { --classifier-rgb: 198, 122, 75; }
|
||||||
|
.classifier-blue { --classifier-rgb: 77, 163, 255; }
|
||||||
|
.classifier-teal { --classifier-rgb: 64, 210, 190; }
|
||||||
|
.classifier-yellowgreen { --classifier-rgb: 174, 210, 78; }
|
||||||
|
.classifier-violet { --classifier-rgb: 170, 130, 255; }
|
||||||
|
.classifier-cyan { --classifier-rgb: 94, 214, 255; }
|
||||||
|
.classifier-magenta { --classifier-rgb: 255, 92, 205; }
|
||||||
|
.classifier-neutral { --classifier-rgb: 192, 200, 210; }
|
||||||
|
|
||||||
.contract,
|
.contract,
|
||||||
.drawer-row-title {
|
.drawer-row-title {
|
||||||
margin-bottom: 6px;
|
margin-bottom: 6px;
|
||||||
|
|
@ -1212,25 +1564,26 @@ h3 {
|
||||||
}
|
}
|
||||||
|
|
||||||
@media (max-width: 980px) {
|
@media (max-width: 980px) {
|
||||||
.page-grid-overview,
|
.page-grid-home,
|
||||||
.page-grid-tape,
|
.page-grid-tape,
|
||||||
.page-grid-signals,
|
.page-grid-signals,
|
||||||
.page-grid-charts,
|
.page-grid-charts,
|
||||||
.page-grid-replay,
|
.page-grid-replay,
|
||||||
.overview-strip,
|
|
||||||
.replay-matrix,
|
.replay-matrix,
|
||||||
.shell-metrics {
|
.shell-metrics {
|
||||||
grid-template-columns: minmax(0, 1fr);
|
grid-template-columns: minmax(0, 1fr);
|
||||||
}
|
}
|
||||||
|
|
||||||
.page-grid-overview > :nth-child(1),
|
.page-grid-home > :nth-child(3),
|
||||||
.page-grid-tape > :nth-child(1),
|
.page-grid-tape > :nth-child(1),
|
||||||
.page-grid-replay > :nth-child(1) {
|
.page-grid-replay > :nth-child(1) {
|
||||||
grid-column: auto;
|
grid-column: auto;
|
||||||
grid-row: auto;
|
grid-row: auto;
|
||||||
}
|
}
|
||||||
|
|
||||||
.page-grid-overview > :not(:first-child),
|
.page-grid-home > :nth-child(1),
|
||||||
|
.page-grid-home > :nth-child(2),
|
||||||
|
.page-grid-home > :nth-child(3),
|
||||||
.page-grid-signals > .terminal-pane,
|
.page-grid-signals > .terminal-pane,
|
||||||
.page-grid-replay > :not(:first-child),
|
.page-grid-replay > :not(:first-child),
|
||||||
.page-grid-tape > :first-child,
|
.page-grid-tape > :first-child,
|
||||||
|
|
@ -1246,16 +1599,19 @@ h3 {
|
||||||
|
|
||||||
.terminal-topbar {
|
.terminal-topbar {
|
||||||
position: static;
|
position: static;
|
||||||
grid-template-columns: minmax(0, 1fr);
|
align-items: center;
|
||||||
align-items: stretch;
|
justify-content: flex-end;
|
||||||
|
padding: 10px 16px;
|
||||||
}
|
}
|
||||||
|
|
||||||
.terminal-topbar-actions {
|
.terminal-topbar-actions {
|
||||||
justify-content: space-between;
|
justify-content: flex-end;
|
||||||
|
margin-left: auto;
|
||||||
|
width: auto;
|
||||||
}
|
}
|
||||||
|
|
||||||
.terminal-topbar-controls {
|
.terminal-topbar-controls {
|
||||||
flex: 1 1 auto;
|
flex: 0 1 auto;
|
||||||
}
|
}
|
||||||
|
|
||||||
.flow-filter-popover-panel {
|
.flow-filter-popover-panel {
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,7 @@
|
||||||
import { OverviewRoute } from "./terminal";
|
import { OverviewRoute } from "./terminal";
|
||||||
|
|
||||||
|
export const dynamic = "force-dynamic";
|
||||||
|
|
||||||
export default function Page() {
|
export default function Page() {
|
||||||
return <OverviewRoute />;
|
return <OverviewRoute />;
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,7 @@
|
||||||
import { ReplayRoute } from "../terminal";
|
import { redirect } from "next/navigation";
|
||||||
|
|
||||||
|
export const dynamic = "force-dynamic";
|
||||||
|
|
||||||
export default function Page() {
|
export default function Page() {
|
||||||
return <ReplayRoute />;
|
redirect("/");
|
||||||
}
|
}
|
||||||
|
|
|
||||||
31
apps/web/app/routes.test.ts
Normal file
31
apps/web/app/routes.test.ts
Normal file
|
|
@ -0,0 +1,31 @@
|
||||||
|
import { beforeEach, describe, expect, it, mock } from "bun:test";
|
||||||
|
|
||||||
|
const redirect = mock((path: string) => {
|
||||||
|
throw new Error(`NEXT_REDIRECT:${path}`);
|
||||||
|
});
|
||||||
|
|
||||||
|
mock.module("next/navigation", () => ({ redirect }));
|
||||||
|
|
||||||
|
describe("legacy page redirects", () => {
|
||||||
|
beforeEach(() => {
|
||||||
|
redirect.mockClear();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("redirects /signals to home", async () => {
|
||||||
|
const mod = await import("./signals/page");
|
||||||
|
expect(() => mod.default()).toThrow("NEXT_REDIRECT:/");
|
||||||
|
expect(redirect).toHaveBeenCalledWith("/");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("redirects /charts to home", async () => {
|
||||||
|
const mod = await import("./charts/page");
|
||||||
|
expect(() => mod.default()).toThrow("NEXT_REDIRECT:/");
|
||||||
|
expect(redirect).toHaveBeenCalledWith("/");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("redirects /replay to home", async () => {
|
||||||
|
const mod = await import("./replay/page");
|
||||||
|
expect(() => mod.default()).toThrow("NEXT_REDIRECT:/");
|
||||||
|
expect(redirect).toHaveBeenCalledWith("/");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
@ -1,5 +1,7 @@
|
||||||
import { SignalsRoute } from "../terminal";
|
import { redirect } from "next/navigation";
|
||||||
|
|
||||||
|
export const dynamic = "force-dynamic";
|
||||||
|
|
||||||
export default function Page() {
|
export default function Page() {
|
||||||
return <SignalsRoute />;
|
redirect("/");
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,7 @@
|
||||||
import { TapeRoute } from "../terminal";
|
import { TapeRoute } from "../terminal";
|
||||||
|
|
||||||
|
export const dynamic = "force-dynamic";
|
||||||
|
|
||||||
export default function Page() {
|
export default function Page() {
|
||||||
return <TapeRoute />;
|
return <TapeRoute />;
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -1,19 +1,24 @@
|
||||||
import { describe, expect, it } from "bun:test";
|
import { describe, expect, it } from "bun:test";
|
||||||
import {
|
import {
|
||||||
|
NAV_ITEMS,
|
||||||
buildDefaultFlowFilters,
|
buildDefaultFlowFilters,
|
||||||
|
classifierToneForFamily,
|
||||||
deriveAlertDirection,
|
deriveAlertDirection,
|
||||||
countActiveFlowFilterGroups,
|
countActiveFlowFilterGroups,
|
||||||
formatCompactUsd,
|
formatCompactUsd,
|
||||||
formatOptionContractLabel,
|
formatOptionContractLabel,
|
||||||
flushPausableTapeData,
|
flushPausableTapeData,
|
||||||
getAlertWindowAnchorTs,
|
getAlertWindowAnchorTs,
|
||||||
|
getOptionTableSnapshot,
|
||||||
getLiveFeedStatus,
|
getLiveFeedStatus,
|
||||||
|
getLiveManifest,
|
||||||
normalizeAlertSeverity,
|
normalizeAlertSeverity,
|
||||||
nextFlowFilterPopoverState,
|
nextFlowFilterPopoverState,
|
||||||
projectPausableTapeState,
|
projectPausableTapeState,
|
||||||
reducePausableTapeData,
|
reducePausableTapeData,
|
||||||
shouldRetainLiveSnapshotHistory,
|
shouldRetainLiveSnapshotHistory,
|
||||||
shouldShowEquitiesSilentFeedWarning,
|
shouldShowEquitiesSilentFeedWarning,
|
||||||
|
selectPrimaryClassifierHit,
|
||||||
statusLabel,
|
statusLabel,
|
||||||
toggleFilterValue
|
toggleFilterValue
|
||||||
} from "./terminal";
|
} from "./terminal";
|
||||||
|
|
@ -35,6 +40,98 @@ const makeAlert = (overrides: Record<string, unknown> = {}) =>
|
||||||
...overrides
|
...overrides
|
||||||
}) as any;
|
}) as any;
|
||||||
|
|
||||||
|
describe("live manifest", () => {
|
||||||
|
it("includes options on home and tape", () => {
|
||||||
|
const filters = buildDefaultFlowFilters();
|
||||||
|
for (const pathname of ["/", "/tape"]) {
|
||||||
|
expect(
|
||||||
|
getLiveManifest(pathname, "SPY", 60000, filters).some(
|
||||||
|
(subscription) => subscription.channel === "options"
|
||||||
|
)
|
||||||
|
).toBe(true);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
it("dedupes tape options subscription", () => {
|
||||||
|
const tapeOptionsSubscriptions = getLiveManifest(
|
||||||
|
"/tape",
|
||||||
|
"SPY",
|
||||||
|
60000,
|
||||||
|
buildDefaultFlowFilters()
|
||||||
|
).filter((subscription) => subscription.channel === "options");
|
||||||
|
expect(tapeOptionsSubscriptions).toHaveLength(1);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("keeps option filters on baseline subscription across page changes", () => {
|
||||||
|
const filters = {
|
||||||
|
...buildDefaultFlowFilters(),
|
||||||
|
minNotional: 125_000
|
||||||
|
};
|
||||||
|
|
||||||
|
const homeOptionsSubscription = getLiveManifest("/", "SPY", 60000, filters).find(
|
||||||
|
(subscription) => subscription.channel === "options"
|
||||||
|
);
|
||||||
|
const tapeOptionsSubscription = getLiveManifest("/tape", "SPY", 60000, filters).find(
|
||||||
|
(subscription) => subscription.channel === "options"
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(homeOptionsSubscription?.filters).toBe(filters);
|
||||||
|
expect(tapeOptionsSubscription?.filters).toBe(filters);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("applies global flow filters to flow subscriptions on home and tape", () => {
|
||||||
|
const filters = {
|
||||||
|
...buildDefaultFlowFilters(),
|
||||||
|
minNotional: 50_000
|
||||||
|
};
|
||||||
|
|
||||||
|
const homeFlowSubscription = getLiveManifest("/", "SPY", 60000, filters).find(
|
||||||
|
(subscription) => subscription.channel === "flow"
|
||||||
|
);
|
||||||
|
const tapeFlowSubscription = getLiveManifest("/tape", "SPY", 60000, filters).find(
|
||||||
|
(subscription) => subscription.channel === "flow"
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(homeFlowSubscription?.filters).toBe(filters);
|
||||||
|
expect(tapeFlowSubscription?.filters).toBe(filters);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("includes scoped option and equity subscriptions", () => {
|
||||||
|
const manifest = getLiveManifest(
|
||||||
|
"/tape",
|
||||||
|
"AAPL",
|
||||||
|
60000,
|
||||||
|
buildDefaultFlowFilters(),
|
||||||
|
{
|
||||||
|
underlying_ids: ["AAPL"],
|
||||||
|
option_contract_id: "AAPL-2025-01-17-200-C"
|
||||||
|
},
|
||||||
|
{ underlying_ids: ["AAPL"] }
|
||||||
|
);
|
||||||
|
const optionsSubscription = manifest.find(
|
||||||
|
(subscription): subscription is Extract<(typeof manifest)[number], { channel: "options" }> =>
|
||||||
|
subscription.channel === "options"
|
||||||
|
);
|
||||||
|
const equitiesSubscription = manifest.find(
|
||||||
|
(subscription): subscription is Extract<(typeof manifest)[number], { channel: "equities" }> =>
|
||||||
|
subscription.channel === "equities"
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(optionsSubscription?.underlying_ids).toEqual(["AAPL"]);
|
||||||
|
expect(optionsSubscription?.option_contract_id).toBe("AAPL-2025-01-17-200-C");
|
||||||
|
expect(equitiesSubscription?.underlying_ids).toEqual(["AAPL"]);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("terminal navigation", () => {
|
||||||
|
it("exposes only Home and Tape as top-level destinations", () => {
|
||||||
|
expect(NAV_ITEMS).toEqual([
|
||||||
|
{ href: "/", label: "Home" },
|
||||||
|
{ href: "/tape", label: "Tape" }
|
||||||
|
]);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
describe("live tape pausable helpers", () => {
|
describe("live tape pausable helpers", () => {
|
||||||
it("queues new items while paused and flushes them on resume", () => {
|
it("queues new items while paused and flushes them on resume", () => {
|
||||||
let state = reducePausableTapeData(
|
let state = reducePausableTapeData(
|
||||||
|
|
@ -171,6 +268,54 @@ describe("options display formatters", () => {
|
||||||
expect(formatCompactUsd(1_250_000)).toBe("1.3M");
|
expect(formatCompactUsd(1_250_000)).toBe("1.3M");
|
||||||
expect(formatCompactUsd(Number.NaN)).toBe("0.00");
|
expect(formatCompactUsd(Number.NaN)).toBe("0.00");
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("renders options table snapshot values from preserved spot and IV", () => {
|
||||||
|
expect(
|
||||||
|
getOptionTableSnapshot({
|
||||||
|
price: 1.25,
|
||||||
|
size: 10,
|
||||||
|
notional: 12_500,
|
||||||
|
execution_nbbo_side: "A",
|
||||||
|
execution_underlying_spot: 450.05,
|
||||||
|
execution_iv: 0.42
|
||||||
|
})
|
||||||
|
).toEqual({
|
||||||
|
spot: "450.05",
|
||||||
|
iv: "42%",
|
||||||
|
side: "A",
|
||||||
|
details: "10@1.25_A",
|
||||||
|
value: "12.5K"
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it("renders legacy options table snapshot spot and IV as dashes", () => {
|
||||||
|
const snapshot = getOptionTableSnapshot({
|
||||||
|
price: 1,
|
||||||
|
size: 2
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(snapshot.spot).toBe("--");
|
||||||
|
expect(snapshot.iv).toBe("--");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("classifier row decoration helpers", () => {
|
||||||
|
it("maps classifier families to row tones", () => {
|
||||||
|
expect(classifierToneForFamily("large_bullish_call_sweep")).toBe("green");
|
||||||
|
expect(classifierToneForFamily("large_bearish_put_sweep")).toBe("red");
|
||||||
|
expect(classifierToneForFamily("straddle")).toBe("blue");
|
||||||
|
expect(classifierToneForFamily("unknown_family")).toBe("neutral");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("selects primary hits by confidence, source timestamp, then seq", () => {
|
||||||
|
const hit = selectPrimaryClassifierHit([
|
||||||
|
{ ...makeAlert({ classifier_id: "old", confidence: 0.9, source_ts: 1_000, seq: 1 }), direction: "bullish", explanations: [] },
|
||||||
|
{ ...makeAlert({ classifier_id: "new", confidence: 0.9, source_ts: 2_000, seq: 1 }), direction: "bullish", explanations: [] },
|
||||||
|
{ ...makeAlert({ classifier_id: "low", confidence: 0.5, source_ts: 3_000, seq: 9 }), direction: "bullish", explanations: [] }
|
||||||
|
]);
|
||||||
|
|
||||||
|
expect(hit?.classifier_id).toBe("new");
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
describe("flow filter popup helpers", () => {
|
describe("flow filter popup helpers", () => {
|
||||||
|
|
|
||||||
File diff suppressed because it is too large
Load diff
20
options-overhaul-phase1.md
Normal file
20
options-overhaul-phase1.md
Normal file
|
|
@ -0,0 +1,20 @@
|
||||||
|
# Options Overhaul Phase 1: Snapshot Tape Table
|
||||||
|
|
||||||
|
Implemented Phase 1 snapshot semantics for the Options tape.
|
||||||
|
|
||||||
|
## Completed
|
||||||
|
|
||||||
|
- Added flat execution snapshot fields to `OptionPrintSchema` / `OptionPrint`.
|
||||||
|
- Added ClickHouse columns and migrations for execution NBBO, underlying spot, and IV context.
|
||||||
|
- Added ingest enrichment that selects option NBBO and equity quote context at or before the option print timestamp.
|
||||||
|
- New enriched prints mirror `nbbo_side` from `execution_nbbo_side`.
|
||||||
|
- Added synthetic per-contract IV state with pressure, decay, and clamps.
|
||||||
|
- Redesigned the Options pane as a dense table using preserved spot/IV/NBBO side first.
|
||||||
|
- Added classifier-hit row color mapping and click/keyboard drawer interaction for classified rows.
|
||||||
|
- Updated `/tape` live subscriptions to include `classifier-hits`.
|
||||||
|
- Added focused tests for schema, storage, enrichment, synthetic IV, and frontend table/classifier helpers.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
- `bun test packages/types/tests/events.test.ts packages/storage/tests/option-prints.test.ts services/ingest-options/tests/enrichment.test.ts services/ingest-options/tests/synthetic.test.ts apps/web/app/terminal.test.ts`
|
||||||
|
- `bun run build` from `apps/web`
|
||||||
|
|
@ -512,7 +512,23 @@ const normalizeOptionRow = (row: unknown): unknown => {
|
||||||
"ts",
|
"ts",
|
||||||
"price",
|
"price",
|
||||||
"size",
|
"size",
|
||||||
"notional"
|
"notional",
|
||||||
|
"execution_nbbo_bid",
|
||||||
|
"execution_nbbo_ask",
|
||||||
|
"execution_nbbo_mid",
|
||||||
|
"execution_nbbo_spread",
|
||||||
|
"execution_nbbo_bid_size",
|
||||||
|
"execution_nbbo_ask_size",
|
||||||
|
"execution_nbbo_ts",
|
||||||
|
"execution_nbbo_age_ms",
|
||||||
|
"execution_underlying_spot",
|
||||||
|
"execution_underlying_bid",
|
||||||
|
"execution_underlying_ask",
|
||||||
|
"execution_underlying_mid",
|
||||||
|
"execution_underlying_spread",
|
||||||
|
"execution_underlying_ts",
|
||||||
|
"execution_underlying_age_ms",
|
||||||
|
"execution_iv"
|
||||||
]);
|
]);
|
||||||
|
|
||||||
if ("is_etf" in record) {
|
if ("is_etf" in record) {
|
||||||
|
|
@ -536,6 +552,14 @@ export type OptionPrintQueryFilters = {
|
||||||
security?: "stock" | "etf" | "all";
|
security?: "stock" | "etf" | "all";
|
||||||
optionTypes?: string[];
|
optionTypes?: string[];
|
||||||
nbboSides?: string[];
|
nbboSides?: string[];
|
||||||
|
underlyingIds?: string[];
|
||||||
|
optionContractId?: string;
|
||||||
|
sinceTs?: number;
|
||||||
|
};
|
||||||
|
|
||||||
|
export type EquityPrintQueryFilters = {
|
||||||
|
underlyingIds?: string[];
|
||||||
|
sinceTs?: number;
|
||||||
};
|
};
|
||||||
|
|
||||||
const buildOptionPrintFilterConditions = (
|
const buildOptionPrintFilterConditions = (
|
||||||
|
|
@ -574,6 +598,32 @@ const buildOptionPrintFilterConditions = (
|
||||||
conditions.push(`nbbo_side IN (${buildStringList(filters.nbboSides)})`);
|
conditions.push(`nbbo_side IN (${buildStringList(filters.nbboSides)})`);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (filters.underlyingIds && filters.underlyingIds.length > 0) {
|
||||||
|
conditions.push(`underlying_id IN (${buildStringList(filters.underlyingIds)})`);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (filters.optionContractId) {
|
||||||
|
conditions.push(`option_contract_id = ${quoteString(filters.optionContractId)}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (typeof filters.sinceTs === "number" && Number.isFinite(filters.sinceTs)) {
|
||||||
|
conditions.push(`ts >= ${clampCursor(filters.sinceTs)}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
return conditions;
|
||||||
|
};
|
||||||
|
|
||||||
|
const buildEquityPrintFilterConditions = (filters?: EquityPrintQueryFilters): string[] => {
|
||||||
|
const conditions: string[] = [];
|
||||||
|
if (!filters) {
|
||||||
|
return conditions;
|
||||||
|
}
|
||||||
|
if (filters.underlyingIds && filters.underlyingIds.length > 0) {
|
||||||
|
conditions.push(`underlying_id IN (${buildStringList(filters.underlyingIds)})`);
|
||||||
|
}
|
||||||
|
if (typeof filters.sinceTs === "number" && Number.isFinite(filters.sinceTs)) {
|
||||||
|
conditions.push(`ts >= ${clampCursor(filters.sinceTs)}`);
|
||||||
|
}
|
||||||
return conditions;
|
return conditions;
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
@ -782,11 +832,14 @@ export const fetchRecentOptionNBBO = async (
|
||||||
|
|
||||||
export const fetchRecentEquityPrints = async (
|
export const fetchRecentEquityPrints = async (
|
||||||
client: ClickHouseClient,
|
client: ClickHouseClient,
|
||||||
limit: number
|
limit: number,
|
||||||
|
filters?: EquityPrintQueryFilters
|
||||||
): Promise<EquityPrint[]> => {
|
): Promise<EquityPrint[]> => {
|
||||||
const safeLimit = clampLimit(limit);
|
const safeLimit = clampLimit(limit);
|
||||||
|
const conditions = buildEquityPrintFilterConditions(filters);
|
||||||
|
const whereClause = conditions.length > 0 ? ` WHERE ${conditions.join(" AND ")}` : "";
|
||||||
const result = await client.query({
|
const result = await client.query({
|
||||||
query: `SELECT * FROM ${EQUITY_PRINTS_TABLE} ORDER BY ts DESC, seq DESC LIMIT ${safeLimit}`,
|
query: `SELECT * FROM ${EQUITY_PRINTS_TABLE}${whereClause} ORDER BY ts DESC, seq DESC LIMIT ${safeLimit}`,
|
||||||
format: "JSONEachRow"
|
format: "JSONEachRow"
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|
@ -967,14 +1020,20 @@ export const fetchEquityPrintsAfter = async (
|
||||||
client: ClickHouseClient,
|
client: ClickHouseClient,
|
||||||
afterTs: number,
|
afterTs: number,
|
||||||
afterSeq: number,
|
afterSeq: number,
|
||||||
limit: number
|
limit: number,
|
||||||
|
filters?: EquityPrintQueryFilters
|
||||||
): Promise<EquityPrint[]> => {
|
): Promise<EquityPrint[]> => {
|
||||||
const safeLimit = clampLimit(limit);
|
const safeLimit = clampLimit(limit);
|
||||||
const safeAfterTs = clampCursor(afterTs);
|
const safeAfterTs = clampCursor(afterTs);
|
||||||
const safeAfterSeq = clampCursor(afterSeq);
|
const safeAfterSeq = clampCursor(afterSeq);
|
||||||
|
|
||||||
|
const conditions = [
|
||||||
|
`((ts, seq) > (${safeAfterTs}, ${safeAfterSeq}))`,
|
||||||
|
...buildEquityPrintFilterConditions(filters)
|
||||||
|
];
|
||||||
|
|
||||||
const result = await client.query({
|
const result = await client.query({
|
||||||
query: `SELECT * FROM ${EQUITY_PRINTS_TABLE} WHERE (ts, seq) > (${safeAfterTs}, ${safeAfterSeq}) ORDER BY ts ASC, seq ASC LIMIT ${safeLimit}`,
|
query: `SELECT * FROM ${EQUITY_PRINTS_TABLE} WHERE ${conditions.join(" AND ")} ORDER BY ts ASC, seq ASC LIMIT ${safeLimit}`,
|
||||||
format: "JSONEachRow"
|
format: "JSONEachRow"
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|
@ -1236,11 +1295,16 @@ export const fetchEquityPrintsBefore = async (
|
||||||
client: ClickHouseClient,
|
client: ClickHouseClient,
|
||||||
beforeTs: number,
|
beforeTs: number,
|
||||||
beforeSeq: number,
|
beforeSeq: number,
|
||||||
limit: number
|
limit: number,
|
||||||
|
filters?: EquityPrintQueryFilters
|
||||||
): Promise<EquityPrint[]> => {
|
): Promise<EquityPrint[]> => {
|
||||||
const safeLimit = clampLimit(limit);
|
const safeLimit = clampLimit(limit);
|
||||||
|
const conditions = [
|
||||||
|
buildBeforeTupleCondition("ts", "seq", beforeTs, beforeSeq),
|
||||||
|
...buildEquityPrintFilterConditions(filters)
|
||||||
|
];
|
||||||
const result = await client.query({
|
const result = await client.query({
|
||||||
query: `SELECT * FROM ${EQUITY_PRINTS_TABLE} WHERE ${buildBeforeTupleCondition("ts", "seq", beforeTs, beforeSeq)} ORDER BY ts DESC, seq DESC LIMIT ${safeLimit}`,
|
query: `SELECT * FROM ${EQUITY_PRINTS_TABLE} WHERE ${conditions.join(" AND ")} ORDER BY ts DESC, seq DESC LIMIT ${safeLimit}`,
|
||||||
format: "JSONEachRow"
|
format: "JSONEachRow"
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|
@ -1248,6 +1312,22 @@ export const fetchEquityPrintsBefore = async (
|
||||||
return EquityPrintSchema.array().parse(rows.map(normalizeEquityRow));
|
return EquityPrintSchema.array().parse(rows.map(normalizeEquityRow));
|
||||||
};
|
};
|
||||||
|
|
||||||
|
export const fetchEquityQuotesBefore = async (
|
||||||
|
client: ClickHouseClient,
|
||||||
|
beforeTs: number,
|
||||||
|
beforeSeq: number,
|
||||||
|
limit: number
|
||||||
|
): Promise<EquityQuote[]> => {
|
||||||
|
const safeLimit = clampLimit(limit);
|
||||||
|
const result = await client.query({
|
||||||
|
query: `SELECT * FROM ${EQUITY_QUOTES_TABLE} WHERE ${buildBeforeTupleCondition("ts", "seq", beforeTs, beforeSeq)} ORDER BY ts DESC, seq DESC LIMIT ${safeLimit}`,
|
||||||
|
format: "JSONEachRow"
|
||||||
|
});
|
||||||
|
|
||||||
|
const rows = await result.json<unknown[]>();
|
||||||
|
return EquityQuoteSchema.array().parse(rows.map(normalizeEquityQuoteRow));
|
||||||
|
};
|
||||||
|
|
||||||
export const fetchEquityPrintJoinsBefore = async (
|
export const fetchEquityPrintJoinsBefore = async (
|
||||||
client: ClickHouseClient,
|
client: ClickHouseClient,
|
||||||
beforeTs: number,
|
beforeTs: number,
|
||||||
|
|
|
||||||
|
|
@ -19,6 +19,25 @@ CREATE TABLE IF NOT EXISTS ${OPTION_PRINTS_TABLE} (
|
||||||
option_type Nullable(String),
|
option_type Nullable(String),
|
||||||
notional Nullable(Float64),
|
notional Nullable(Float64),
|
||||||
nbbo_side Nullable(String),
|
nbbo_side Nullable(String),
|
||||||
|
execution_nbbo_bid Nullable(Float64),
|
||||||
|
execution_nbbo_ask Nullable(Float64),
|
||||||
|
execution_nbbo_mid Nullable(Float64),
|
||||||
|
execution_nbbo_spread Nullable(Float64),
|
||||||
|
execution_nbbo_bid_size Nullable(UInt32),
|
||||||
|
execution_nbbo_ask_size Nullable(UInt32),
|
||||||
|
execution_nbbo_ts Nullable(UInt64),
|
||||||
|
execution_nbbo_age_ms Nullable(Float64),
|
||||||
|
execution_nbbo_side Nullable(String),
|
||||||
|
execution_underlying_spot Nullable(Float64),
|
||||||
|
execution_underlying_bid Nullable(Float64),
|
||||||
|
execution_underlying_ask Nullable(Float64),
|
||||||
|
execution_underlying_mid Nullable(Float64),
|
||||||
|
execution_underlying_spread Nullable(Float64),
|
||||||
|
execution_underlying_ts Nullable(UInt64),
|
||||||
|
execution_underlying_age_ms Nullable(Float64),
|
||||||
|
execution_underlying_source Nullable(String),
|
||||||
|
execution_iv Nullable(Float64),
|
||||||
|
execution_iv_source Nullable(String),
|
||||||
is_etf Nullable(Bool),
|
is_etf Nullable(Bool),
|
||||||
signal_pass Nullable(Bool),
|
signal_pass Nullable(Bool),
|
||||||
signal_reasons Array(String) DEFAULT [],
|
signal_reasons Array(String) DEFAULT [],
|
||||||
|
|
@ -35,6 +54,25 @@ export const optionPrintsTableMigrations = (): string[] => {
|
||||||
`ALTER TABLE ${OPTION_PRINTS_TABLE} ADD COLUMN IF NOT EXISTS option_type Nullable(String)`,
|
`ALTER TABLE ${OPTION_PRINTS_TABLE} ADD COLUMN IF NOT EXISTS option_type Nullable(String)`,
|
||||||
`ALTER TABLE ${OPTION_PRINTS_TABLE} ADD COLUMN IF NOT EXISTS notional Nullable(Float64)`,
|
`ALTER TABLE ${OPTION_PRINTS_TABLE} ADD COLUMN IF NOT EXISTS notional Nullable(Float64)`,
|
||||||
`ALTER TABLE ${OPTION_PRINTS_TABLE} ADD COLUMN IF NOT EXISTS nbbo_side Nullable(String)`,
|
`ALTER TABLE ${OPTION_PRINTS_TABLE} ADD COLUMN IF NOT EXISTS nbbo_side Nullable(String)`,
|
||||||
|
`ALTER TABLE ${OPTION_PRINTS_TABLE} ADD COLUMN IF NOT EXISTS execution_nbbo_bid Nullable(Float64)`,
|
||||||
|
`ALTER TABLE ${OPTION_PRINTS_TABLE} ADD COLUMN IF NOT EXISTS execution_nbbo_ask Nullable(Float64)`,
|
||||||
|
`ALTER TABLE ${OPTION_PRINTS_TABLE} ADD COLUMN IF NOT EXISTS execution_nbbo_mid Nullable(Float64)`,
|
||||||
|
`ALTER TABLE ${OPTION_PRINTS_TABLE} ADD COLUMN IF NOT EXISTS execution_nbbo_spread Nullable(Float64)`,
|
||||||
|
`ALTER TABLE ${OPTION_PRINTS_TABLE} ADD COLUMN IF NOT EXISTS execution_nbbo_bid_size Nullable(UInt32)`,
|
||||||
|
`ALTER TABLE ${OPTION_PRINTS_TABLE} ADD COLUMN IF NOT EXISTS execution_nbbo_ask_size Nullable(UInt32)`,
|
||||||
|
`ALTER TABLE ${OPTION_PRINTS_TABLE} ADD COLUMN IF NOT EXISTS execution_nbbo_ts Nullable(UInt64)`,
|
||||||
|
`ALTER TABLE ${OPTION_PRINTS_TABLE} ADD COLUMN IF NOT EXISTS execution_nbbo_age_ms Nullable(Float64)`,
|
||||||
|
`ALTER TABLE ${OPTION_PRINTS_TABLE} ADD COLUMN IF NOT EXISTS execution_nbbo_side Nullable(String)`,
|
||||||
|
`ALTER TABLE ${OPTION_PRINTS_TABLE} ADD COLUMN IF NOT EXISTS execution_underlying_spot Nullable(Float64)`,
|
||||||
|
`ALTER TABLE ${OPTION_PRINTS_TABLE} ADD COLUMN IF NOT EXISTS execution_underlying_bid Nullable(Float64)`,
|
||||||
|
`ALTER TABLE ${OPTION_PRINTS_TABLE} ADD COLUMN IF NOT EXISTS execution_underlying_ask Nullable(Float64)`,
|
||||||
|
`ALTER TABLE ${OPTION_PRINTS_TABLE} ADD COLUMN IF NOT EXISTS execution_underlying_mid Nullable(Float64)`,
|
||||||
|
`ALTER TABLE ${OPTION_PRINTS_TABLE} ADD COLUMN IF NOT EXISTS execution_underlying_spread Nullable(Float64)`,
|
||||||
|
`ALTER TABLE ${OPTION_PRINTS_TABLE} ADD COLUMN IF NOT EXISTS execution_underlying_ts Nullable(UInt64)`,
|
||||||
|
`ALTER TABLE ${OPTION_PRINTS_TABLE} ADD COLUMN IF NOT EXISTS execution_underlying_age_ms Nullable(Float64)`,
|
||||||
|
`ALTER TABLE ${OPTION_PRINTS_TABLE} ADD COLUMN IF NOT EXISTS execution_underlying_source Nullable(String)`,
|
||||||
|
`ALTER TABLE ${OPTION_PRINTS_TABLE} ADD COLUMN IF NOT EXISTS execution_iv Nullable(Float64)`,
|
||||||
|
`ALTER TABLE ${OPTION_PRINTS_TABLE} ADD COLUMN IF NOT EXISTS execution_iv_source Nullable(String)`,
|
||||||
`ALTER TABLE ${OPTION_PRINTS_TABLE} ADD COLUMN IF NOT EXISTS is_etf Nullable(Bool)`,
|
`ALTER TABLE ${OPTION_PRINTS_TABLE} ADD COLUMN IF NOT EXISTS is_etf Nullable(Bool)`,
|
||||||
`ALTER TABLE ${OPTION_PRINTS_TABLE} ADD COLUMN IF NOT EXISTS signal_pass Nullable(Bool)`,
|
`ALTER TABLE ${OPTION_PRINTS_TABLE} ADD COLUMN IF NOT EXISTS signal_pass Nullable(Bool)`,
|
||||||
`ALTER TABLE ${OPTION_PRINTS_TABLE} ADD COLUMN IF NOT EXISTS signal_reasons Array(String) DEFAULT []`,
|
`ALTER TABLE ${OPTION_PRINTS_TABLE} ADD COLUMN IF NOT EXISTS signal_reasons Array(String) DEFAULT []`,
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,10 @@
|
||||||
import { describe, expect, it } from "bun:test";
|
import { describe, expect, it } from "bun:test";
|
||||||
|
import {
|
||||||
|
createClickHouseClient,
|
||||||
|
fetchEquityPrintsAfter,
|
||||||
|
fetchEquityPrintsBefore,
|
||||||
|
fetchRecentEquityPrints
|
||||||
|
} from "../src/clickhouse";
|
||||||
import { equityPrintsTableDDL, EQUITY_PRINTS_TABLE } from "../src/equity-prints";
|
import { equityPrintsTableDDL, EQUITY_PRINTS_TABLE } from "../src/equity-prints";
|
||||||
|
|
||||||
const basePrint = {
|
const basePrint = {
|
||||||
|
|
@ -24,4 +30,39 @@ describe("equity-prints storage helpers", () => {
|
||||||
expect(ddl).toContain(EQUITY_PRINTS_TABLE);
|
expect(ddl).toContain(EQUITY_PRINTS_TABLE);
|
||||||
expect(ddl).toContain("CREATE TABLE IF NOT EXISTS");
|
expect(ddl).toContain("CREATE TABLE IF NOT EXISTS");
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("builds scoped recent, before, and after queries", async () => {
|
||||||
|
const queries: string[] = [];
|
||||||
|
const client = createClickHouseClient({ url: "http://127.0.0.1:8123" });
|
||||||
|
client.query = async ({ query }) => {
|
||||||
|
queries.push(query);
|
||||||
|
return {
|
||||||
|
async json<T>() {
|
||||||
|
return [] as T;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
|
await fetchRecentEquityPrints(client, 25, {
|
||||||
|
underlyingIds: ["AAPL", "NVDA"],
|
||||||
|
sinceTs: 123
|
||||||
|
});
|
||||||
|
await fetchEquityPrintsBefore(client, 100, 5, 20, {
|
||||||
|
underlyingIds: ["AAPL"],
|
||||||
|
sinceTs: 50
|
||||||
|
});
|
||||||
|
await fetchEquityPrintsAfter(client, 100, 5, 20, {
|
||||||
|
underlyingIds: ["NVDA"],
|
||||||
|
sinceTs: 50
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(queries[0]).toContain("underlying_id IN ('AAPL', 'NVDA')");
|
||||||
|
expect(queries[0]).toContain("ts >= 123");
|
||||||
|
expect(queries[1]).toContain("(ts, seq) < (100, 5)");
|
||||||
|
expect(queries[1]).toContain("underlying_id IN ('AAPL')");
|
||||||
|
expect(queries[1]).toContain("ts >= 50");
|
||||||
|
expect(queries[2]).toContain("((ts, seq) > (100, 5))");
|
||||||
|
expect(queries[2]).toContain("underlying_id IN ('NVDA')");
|
||||||
|
expect(queries[2]).toContain("ts >= 50");
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|
|
||||||
|
|
@ -4,6 +4,7 @@ import {
|
||||||
EQUITY_QUOTES_TABLE,
|
EQUITY_QUOTES_TABLE,
|
||||||
normalizeEquityQuote
|
normalizeEquityQuote
|
||||||
} from "../src/equity-quotes";
|
} from "../src/equity-quotes";
|
||||||
|
import { fetchEquityQuotesBefore, type ClickHouseClient } from "../src/clickhouse";
|
||||||
|
|
||||||
const baseQuote = {
|
const baseQuote = {
|
||||||
source_ts: 100,
|
source_ts: 100,
|
||||||
|
|
@ -27,4 +28,35 @@ describe("equity-quotes storage helpers", () => {
|
||||||
expect(ddl).toContain(EQUITY_QUOTES_TABLE);
|
expect(ddl).toContain(EQUITY_QUOTES_TABLE);
|
||||||
expect(ddl).toContain("CREATE TABLE IF NOT EXISTS");
|
expect(ddl).toContain("CREATE TABLE IF NOT EXISTS");
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("fetches older quotes with tuple cursor ordering", async () => {
|
||||||
|
let queryText = "";
|
||||||
|
const client = {
|
||||||
|
query: async ({ query }: { query: string }) => {
|
||||||
|
queryText = query;
|
||||||
|
return {
|
||||||
|
async json<T>() {
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
...baseQuote,
|
||||||
|
source_ts: 90,
|
||||||
|
ingest_ts: 201,
|
||||||
|
seq: 2,
|
||||||
|
trace_id: "trace-2",
|
||||||
|
ts: 90
|
||||||
|
}
|
||||||
|
] as T;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
}
|
||||||
|
} as unknown as ClickHouseClient;
|
||||||
|
|
||||||
|
const rows = await fetchEquityQuotesBefore(client, 100, 3, 25);
|
||||||
|
|
||||||
|
expect(rows).toHaveLength(1);
|
||||||
|
expect(rows[0]?.trace_id).toBe("trace-2");
|
||||||
|
expect(queryText).toContain(EQUITY_QUOTES_TABLE);
|
||||||
|
expect(queryText).toContain("WHERE (ts, seq) < (100, 3)");
|
||||||
|
expect(queryText).toContain("ORDER BY ts DESC, seq DESC LIMIT 25");
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|
|
||||||
|
|
@ -25,10 +25,20 @@ describe("option-prints storage helpers", () => {
|
||||||
expect(normalized.conditions).toEqual([]);
|
expect(normalized.conditions).toEqual([]);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("normalizes legacy rows with missing execution context", () => {
|
||||||
|
const normalized = normalizeOptionPrint(basePrint);
|
||||||
|
expect(normalized.execution_nbbo_bid).toBeUndefined();
|
||||||
|
expect(normalized.execution_underlying_spot).toBeUndefined();
|
||||||
|
expect(normalized.execution_iv).toBeUndefined();
|
||||||
|
});
|
||||||
|
|
||||||
it("includes the correct table name in the DDL", () => {
|
it("includes the correct table name in the DDL", () => {
|
||||||
const ddl = optionPrintsTableDDL();
|
const ddl = optionPrintsTableDDL();
|
||||||
expect(ddl).toContain(OPTION_PRINTS_TABLE);
|
expect(ddl).toContain(OPTION_PRINTS_TABLE);
|
||||||
expect(ddl).toContain("CREATE TABLE IF NOT EXISTS");
|
expect(ddl).toContain("CREATE TABLE IF NOT EXISTS");
|
||||||
|
expect(ddl).toContain("execution_nbbo_bid Nullable(Float64)");
|
||||||
|
expect(ddl).toContain("execution_underlying_spot Nullable(Float64)");
|
||||||
|
expect(ddl).toContain("execution_iv Nullable(Float64)");
|
||||||
});
|
});
|
||||||
|
|
||||||
it("builds before/history and trace lookup queries", async () => {
|
it("builds before/history and trace lookup queries", async () => {
|
||||||
|
|
@ -48,7 +58,10 @@ describe("option-prints storage helpers", () => {
|
||||||
security: "stock",
|
security: "stock",
|
||||||
nbboSides: ["AA", "A"],
|
nbboSides: ["AA", "A"],
|
||||||
optionTypes: ["call"],
|
optionTypes: ["call"],
|
||||||
minNotional: 25_000
|
minNotional: 25_000,
|
||||||
|
underlyingIds: ["AAPL", "NVDA"],
|
||||||
|
optionContractId: "AAPL-2025-01-17-200-C",
|
||||||
|
sinceTs: 123
|
||||||
});
|
});
|
||||||
await fetchOptionPrintsBefore(client, 100, 5, 20, "alpaca");
|
await fetchOptionPrintsBefore(client, 100, 5, 20, "alpaca");
|
||||||
await fetchOptionPrintsByTraceIds(client, ["trace-1", "trace-2"]);
|
await fetchOptionPrintsByTraceIds(client, ["trace-1", "trace-2"]);
|
||||||
|
|
@ -58,6 +71,9 @@ describe("option-prints storage helpers", () => {
|
||||||
expect(queries[0]).toContain("nbbo_side IN ('AA', 'A')");
|
expect(queries[0]).toContain("nbbo_side IN ('AA', 'A')");
|
||||||
expect(queries[0]).toContain("option_type IN ('call')");
|
expect(queries[0]).toContain("option_type IN ('call')");
|
||||||
expect(queries[0]).toContain("notional >= 25000");
|
expect(queries[0]).toContain("notional >= 25000");
|
||||||
|
expect(queries[0]).toContain("underlying_id IN ('AAPL', 'NVDA')");
|
||||||
|
expect(queries[0]).toContain("option_contract_id = 'AAPL-2025-01-17-200-C'");
|
||||||
|
expect(queries[0]).toContain("ts >= 123");
|
||||||
expect(queries[1]).toContain("(ts, seq) < (100, 5)");
|
expect(queries[1]).toContain("(ts, seq) < (100, 5)");
|
||||||
expect(queries[1]).toContain("startsWith(trace_id, 'alpaca')");
|
expect(queries[1]).toContain("startsWith(trace_id, 'alpaca')");
|
||||||
expect(queries[1]).toContain("ORDER BY ts DESC, seq DESC LIMIT 20");
|
expect(queries[1]).toContain("ORDER BY ts DESC, seq DESC LIMIT 20");
|
||||||
|
|
|
||||||
|
|
@ -22,6 +22,31 @@ export const OptionPrintSchema = EventMetaSchema.merge(
|
||||||
option_type: z.preprocess((value) => (value === null ? undefined : value), OptionTypeSchema.optional()),
|
option_type: z.preprocess((value) => (value === null ? undefined : value), OptionTypeSchema.optional()),
|
||||||
notional: z.preprocess((value) => (value === null ? undefined : value), z.number().nonnegative().optional()),
|
notional: z.preprocess((value) => (value === null ? undefined : value), z.number().nonnegative().optional()),
|
||||||
nbbo_side: z.preprocess((value) => (value === null ? undefined : value), OptionNbboSideSchema.optional()),
|
nbbo_side: z.preprocess((value) => (value === null ? undefined : value), OptionNbboSideSchema.optional()),
|
||||||
|
execution_nbbo_bid: z.preprocess((value) => (value === null ? undefined : value), z.number().optional()),
|
||||||
|
execution_nbbo_ask: z.preprocess((value) => (value === null ? undefined : value), z.number().optional()),
|
||||||
|
execution_nbbo_mid: z.preprocess((value) => (value === null ? undefined : value), z.number().optional()),
|
||||||
|
execution_nbbo_spread: z.preprocess((value) => (value === null ? undefined : value), z.number().optional()),
|
||||||
|
execution_nbbo_bid_size: z.preprocess((value) => (value === null ? undefined : value), z.number().int().nonnegative().optional()),
|
||||||
|
execution_nbbo_ask_size: z.preprocess((value) => (value === null ? undefined : value), z.number().int().nonnegative().optional()),
|
||||||
|
execution_nbbo_ts: z.preprocess((value) => (value === null ? undefined : value), z.number().int().nonnegative().optional()),
|
||||||
|
execution_nbbo_age_ms: z.preprocess((value) => (value === null ? undefined : value), z.number().nonnegative().optional()),
|
||||||
|
execution_nbbo_side: z.preprocess((value) => (value === null ? undefined : value), OptionNbboSideSchema.optional()),
|
||||||
|
execution_underlying_spot: z.preprocess((value) => (value === null ? undefined : value), z.number().optional()),
|
||||||
|
execution_underlying_bid: z.preprocess((value) => (value === null ? undefined : value), z.number().optional()),
|
||||||
|
execution_underlying_ask: z.preprocess((value) => (value === null ? undefined : value), z.number().optional()),
|
||||||
|
execution_underlying_mid: z.preprocess((value) => (value === null ? undefined : value), z.number().optional()),
|
||||||
|
execution_underlying_spread: z.preprocess((value) => (value === null ? undefined : value), z.number().optional()),
|
||||||
|
execution_underlying_ts: z.preprocess((value) => (value === null ? undefined : value), z.number().int().nonnegative().optional()),
|
||||||
|
execution_underlying_age_ms: z.preprocess((value) => (value === null ? undefined : value), z.number().nonnegative().optional()),
|
||||||
|
execution_underlying_source: z.preprocess(
|
||||||
|
(value) => (value === null ? undefined : value),
|
||||||
|
z.literal("equity_quote_mid").optional()
|
||||||
|
),
|
||||||
|
execution_iv: z.preprocess((value) => (value === null ? undefined : value), z.number().nonnegative().optional()),
|
||||||
|
execution_iv_source: z.preprocess(
|
||||||
|
(value) => (value === null ? undefined : value),
|
||||||
|
z.enum(["provider", "synthetic_pressure_model"]).optional()
|
||||||
|
),
|
||||||
is_etf: z.preprocess((value) => (value === null ? undefined : value), z.boolean().optional()),
|
is_etf: z.preprocess((value) => (value === null ? undefined : value), z.boolean().optional()),
|
||||||
signal_pass: z.preprocess((value) => (value === null ? undefined : value), z.boolean().optional()),
|
signal_pass: z.preprocess((value) => (value === null ? undefined : value), z.boolean().optional()),
|
||||||
signal_reasons: z.array(z.string().min(1)).optional(),
|
signal_reasons: z.array(z.string().min(1)).optional(),
|
||||||
|
|
|
||||||
|
|
@ -5,6 +5,7 @@ import {
|
||||||
EquityCandleSchema,
|
EquityCandleSchema,
|
||||||
EquityPrintJoinSchema,
|
EquityPrintJoinSchema,
|
||||||
EquityPrintSchema,
|
EquityPrintSchema,
|
||||||
|
EquityQuoteSchema,
|
||||||
FlowPacketSchema,
|
FlowPacketSchema,
|
||||||
InferredDarkEventSchema,
|
InferredDarkEventSchema,
|
||||||
OptionNBBOSchema,
|
OptionNBBOSchema,
|
||||||
|
|
@ -26,6 +27,7 @@ export const LiveGenericChannelSchema = z.enum([
|
||||||
"options",
|
"options",
|
||||||
"nbbo",
|
"nbbo",
|
||||||
"equities",
|
"equities",
|
||||||
|
"equity-quotes",
|
||||||
"equity-joins",
|
"equity-joins",
|
||||||
"flow",
|
"flow",
|
||||||
"classifier-hits",
|
"classifier-hits",
|
||||||
|
|
@ -37,6 +39,7 @@ export const LiveChannelSchema = z.enum([
|
||||||
"options",
|
"options",
|
||||||
"nbbo",
|
"nbbo",
|
||||||
"equities",
|
"equities",
|
||||||
|
"equity-quotes",
|
||||||
"equity-joins",
|
"equity-joins",
|
||||||
"flow",
|
"flow",
|
||||||
"classifier-hits",
|
"classifier-hits",
|
||||||
|
|
@ -52,14 +55,20 @@ export type LiveGenericChannel = z.infer<typeof LiveGenericChannelSchema>;
|
||||||
export const LiveSubscriptionSchema = z.discriminatedUnion("channel", [
|
export const LiveSubscriptionSchema = z.discriminatedUnion("channel", [
|
||||||
z.object({
|
z.object({
|
||||||
channel: z.literal("options"),
|
channel: z.literal("options"),
|
||||||
filters: OptionFlowFiltersSchema.optional()
|
filters: OptionFlowFiltersSchema.optional(),
|
||||||
|
underlying_ids: z.array(z.string().min(1)).optional(),
|
||||||
|
option_contract_id: z.string().min(1).optional()
|
||||||
}),
|
}),
|
||||||
z.object({
|
z.object({
|
||||||
channel: z.literal("flow"),
|
channel: z.literal("flow"),
|
||||||
filters: OptionFlowFiltersSchema.optional()
|
filters: OptionFlowFiltersSchema.optional()
|
||||||
}),
|
}),
|
||||||
z.object({
|
z.object({
|
||||||
channel: z.enum(["nbbo", "equities", "equity-joins", "classifier-hits", "alerts", "inferred-dark"])
|
channel: z.enum(["nbbo", "equity-quotes", "equity-joins", "classifier-hits", "alerts", "inferred-dark"])
|
||||||
|
}),
|
||||||
|
z.object({
|
||||||
|
channel: z.literal("equities"),
|
||||||
|
underlying_ids: z.array(z.string().min(1)).optional()
|
||||||
}),
|
}),
|
||||||
z.object({
|
z.object({
|
||||||
channel: z.literal("equity-candles"),
|
channel: z.literal("equity-candles"),
|
||||||
|
|
@ -78,6 +87,7 @@ const livePayloadSchemas = {
|
||||||
options: OptionPrintSchema,
|
options: OptionPrintSchema,
|
||||||
nbbo: OptionNBBOSchema,
|
nbbo: OptionNBBOSchema,
|
||||||
equities: EquityPrintSchema,
|
equities: EquityPrintSchema,
|
||||||
|
"equity-quotes": EquityQuoteSchema,
|
||||||
"equity-joins": EquityPrintJoinSchema,
|
"equity-joins": EquityPrintJoinSchema,
|
||||||
flow: FlowPacketSchema,
|
flow: FlowPacketSchema,
|
||||||
"classifier-hits": ClassifierHitEventSchema,
|
"classifier-hits": ClassifierHitEventSchema,
|
||||||
|
|
@ -177,9 +187,23 @@ export type LiveServerMessage = z.infer<typeof LiveServerMessageSchema>;
|
||||||
|
|
||||||
export const getSubscriptionKey = (subscription: LiveSubscription): string => {
|
export const getSubscriptionKey = (subscription: LiveSubscription): string => {
|
||||||
switch (subscription.channel) {
|
switch (subscription.channel) {
|
||||||
case "options":
|
case "options": {
|
||||||
|
const underlyings = subscription.underlying_ids?.length
|
||||||
|
? `|underlyings:${[...subscription.underlying_ids].sort().join(",")}`
|
||||||
|
: "";
|
||||||
|
const contract = subscription.option_contract_id
|
||||||
|
? `|contract:${subscription.option_contract_id}`
|
||||||
|
: "";
|
||||||
|
return `${subscription.channel}|${optionFlowFilterKey(subscription.filters)}${underlyings}${contract}`;
|
||||||
|
}
|
||||||
case "flow":
|
case "flow":
|
||||||
return `${subscription.channel}|${optionFlowFilterKey(subscription.filters)}`;
|
return `${subscription.channel}|${optionFlowFilterKey(subscription.filters)}`;
|
||||||
|
case "equities": {
|
||||||
|
const underlyings = subscription.underlying_ids?.length
|
||||||
|
? `|underlyings:${[...subscription.underlying_ids].sort().join(",")}`
|
||||||
|
: "";
|
||||||
|
return `${subscription.channel}${underlyings}`;
|
||||||
|
}
|
||||||
case "equity-candles":
|
case "equity-candles":
|
||||||
return `${subscription.channel}|${subscription.underlying_id}|${subscription.interval_ms}`;
|
return `${subscription.channel}|${subscription.underlying_id}|${subscription.interval_ms}`;
|
||||||
case "equity-overlay":
|
case "equity-overlay":
|
||||||
|
|
|
||||||
41
packages/types/tests/events.test.ts
Normal file
41
packages/types/tests/events.test.ts
Normal file
|
|
@ -0,0 +1,41 @@
|
||||||
|
import { describe, expect, it } from "bun:test";
|
||||||
|
import { OptionPrintSchema } from "../src/events";
|
||||||
|
|
||||||
|
describe("event schemas", () => {
|
||||||
|
it("accepts option print execution context fields", () => {
|
||||||
|
const parsed = OptionPrintSchema.parse({
|
||||||
|
source_ts: 100,
|
||||||
|
ingest_ts: 101,
|
||||||
|
seq: 1,
|
||||||
|
trace_id: "trace-1",
|
||||||
|
ts: 100,
|
||||||
|
option_contract_id: "SPY-2025-01-17-450-C",
|
||||||
|
price: 1.25,
|
||||||
|
size: 10,
|
||||||
|
exchange: "TEST",
|
||||||
|
execution_nbbo_bid: 1.2,
|
||||||
|
execution_nbbo_ask: 1.3,
|
||||||
|
execution_nbbo_mid: 1.25,
|
||||||
|
execution_nbbo_spread: 0.1,
|
||||||
|
execution_nbbo_bid_size: 20,
|
||||||
|
execution_nbbo_ask_size: 30,
|
||||||
|
execution_nbbo_ts: 99,
|
||||||
|
execution_nbbo_age_ms: 1,
|
||||||
|
execution_nbbo_side: "MID",
|
||||||
|
execution_underlying_spot: 450.05,
|
||||||
|
execution_underlying_bid: 450,
|
||||||
|
execution_underlying_ask: 450.1,
|
||||||
|
execution_underlying_mid: 450.05,
|
||||||
|
execution_underlying_spread: 0.1,
|
||||||
|
execution_underlying_ts: 98,
|
||||||
|
execution_underlying_age_ms: 2,
|
||||||
|
execution_underlying_source: "equity_quote_mid",
|
||||||
|
execution_iv: 0.42,
|
||||||
|
execution_iv_source: "synthetic_pressure_model"
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(parsed.execution_nbbo_side).toBe("MID");
|
||||||
|
expect(parsed.execution_underlying_spot).toBe(450.05);
|
||||||
|
expect(parsed.execution_iv_source).toBe("synthetic_pressure_model");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
@ -23,6 +23,19 @@ describe("live protocol types", () => {
|
||||||
).toBe(
|
).toBe(
|
||||||
'options|{"view":"signal","securityTypes":["stock"],"nbboSides":["A","AA"],"optionTypes":["call","put"],"minNotional":25000}'
|
'options|{"view":"signal","securityTypes":["stock"],"nbboSides":["A","AA"],"optionTypes":["call","put"],"minNotional":25000}'
|
||||||
);
|
);
|
||||||
|
expect(
|
||||||
|
getSubscriptionKey({
|
||||||
|
channel: "options",
|
||||||
|
filters: { view: "signal" },
|
||||||
|
underlying_ids: ["NVDA", "AAPL"],
|
||||||
|
option_contract_id: "AAPL-2025-01-17-200-C"
|
||||||
|
})
|
||||||
|
).toBe(
|
||||||
|
'options|{"view":"signal"}|underlyings:AAPL,NVDA|contract:AAPL-2025-01-17-200-C'
|
||||||
|
);
|
||||||
|
expect(getSubscriptionKey({ channel: "equities", underlying_ids: ["NVDA", "AAPL"] })).toBe(
|
||||||
|
"equities|underlyings:AAPL,NVDA"
|
||||||
|
);
|
||||||
expect(
|
expect(
|
||||||
getSubscriptionKey({
|
getSubscriptionKey({
|
||||||
channel: "equity-candles",
|
channel: "equity-candles",
|
||||||
|
|
|
||||||
|
|
@ -60,6 +60,7 @@ import {
|
||||||
fetchEquityPrintsBefore,
|
fetchEquityPrintsBefore,
|
||||||
fetchEquityPrintsRange,
|
fetchEquityPrintsRange,
|
||||||
fetchEquityPrintJoinsAfter,
|
fetchEquityPrintJoinsAfter,
|
||||||
|
fetchEquityQuotesBefore,
|
||||||
fetchEquityQuotesAfter,
|
fetchEquityQuotesAfter,
|
||||||
fetchInferredDarkBefore,
|
fetchInferredDarkBefore,
|
||||||
fetchInferredDarkAfter,
|
fetchInferredDarkAfter,
|
||||||
|
|
@ -71,6 +72,7 @@ import {
|
||||||
fetchOptionPrintsByTraceIds,
|
fetchOptionPrintsByTraceIds,
|
||||||
fetchRecentOptionPrints
|
fetchRecentOptionPrints
|
||||||
} from "@islandflow/storage";
|
} from "@islandflow/storage";
|
||||||
|
import type { EquityPrintQueryFilters, OptionPrintQueryFilters } from "@islandflow/storage";
|
||||||
import {
|
import {
|
||||||
AlertEventSchema,
|
AlertEventSchema,
|
||||||
ClassifierHitEventSchema,
|
ClassifierHitEventSchema,
|
||||||
|
|
@ -99,7 +101,7 @@ import {
|
||||||
} from "@islandflow/types";
|
} from "@islandflow/types";
|
||||||
import { createClient } from "redis";
|
import { createClient } from "redis";
|
||||||
import { z } from "zod";
|
import { z } from "zod";
|
||||||
import { LiveStateManager, isLiveItemFresh } from "./live";
|
import { LIVE_FEED_LOOKBACK_MS, LiveStateManager, shouldFanoutLiveEvent } from "./live";
|
||||||
|
|
||||||
const service = "api";
|
const service = "api";
|
||||||
const logger = createLogger({ service });
|
const logger = createLogger({ service });
|
||||||
|
|
@ -114,7 +116,8 @@ const envSchema = z.object({
|
||||||
REDIS_URL: z.string().default("redis://127.0.0.1:6379"),
|
REDIS_URL: z.string().default("redis://127.0.0.1:6379"),
|
||||||
REST_DEFAULT_LIMIT: z.coerce.number().int().positive().default(200),
|
REST_DEFAULT_LIMIT: z.coerce.number().int().positive().default(200),
|
||||||
API_DELIVER_POLICY: DeliverPolicySchema.default("new"),
|
API_DELIVER_POLICY: DeliverPolicySchema.default("new"),
|
||||||
API_CONSUMER_RESET: z.coerce.boolean().default(false)
|
API_CONSUMER_RESET: z.coerce.boolean().default(false),
|
||||||
|
LIVE_LAG_WARN_MS: z.coerce.number().int().positive().default(120_000)
|
||||||
});
|
});
|
||||||
|
|
||||||
const env = readEnv(envSchema);
|
const env = readEnv(envSchema);
|
||||||
|
|
@ -124,6 +127,13 @@ const state = {
|
||||||
shutdownPromise: null as Promise<void> | null
|
shutdownPromise: null as Promise<void> | null
|
||||||
};
|
};
|
||||||
|
|
||||||
|
const HOT_LIVE_REDIS_KEYS = {
|
||||||
|
options: "live:options",
|
||||||
|
equities: "live:equities",
|
||||||
|
flow: "live:flow",
|
||||||
|
nbbo: "live:nbbo"
|
||||||
|
} as const;
|
||||||
|
|
||||||
const getErrorMessage = (error: unknown): string => {
|
const getErrorMessage = (error: unknown): string => {
|
||||||
return error instanceof Error ? error.message : String(error);
|
return error instanceof Error ? error.message : String(error);
|
||||||
};
|
};
|
||||||
|
|
@ -557,6 +567,62 @@ const buildHistoryResponse = <T extends { seq: number }>(
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
|
const parseScopeList = (url: URL, ...keys: string[]): string[] | undefined => {
|
||||||
|
const values = keys
|
||||||
|
.flatMap((key) => url.searchParams.getAll(key))
|
||||||
|
.flatMap((value) => value.split(","))
|
||||||
|
.map((value) => value.trim().toUpperCase())
|
||||||
|
.filter(Boolean);
|
||||||
|
const unique = Array.from(new Set(values));
|
||||||
|
return unique.length > 0 ? unique : undefined;
|
||||||
|
};
|
||||||
|
|
||||||
|
const parseLiveOptionPrintFilters = (url: URL): OptionPrintQueryFilters => {
|
||||||
|
const { storageFilters } = parseOptionPrintFilters(url);
|
||||||
|
return {
|
||||||
|
...storageFilters,
|
||||||
|
underlyingIds: parseScopeList(url, "underlying_id", "underlying_ids"),
|
||||||
|
optionContractId: url.searchParams.get("option_contract_id") ?? undefined,
|
||||||
|
sinceTs: Date.now() - LIVE_FEED_LOOKBACK_MS
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
|
const parseLiveEquityPrintFilters = (url: URL): EquityPrintQueryFilters => ({
|
||||||
|
underlyingIds: parseScopeList(url, "underlying_id", "underlying_ids"),
|
||||||
|
sinceTs: Date.now() - LIVE_FEED_LOOKBACK_MS
|
||||||
|
});
|
||||||
|
|
||||||
|
const matchesScopedOptionSubscription = (
|
||||||
|
print: { underlying_id?: string; option_contract_id: string },
|
||||||
|
subscription: LiveSubscription
|
||||||
|
): boolean => {
|
||||||
|
if (subscription.channel !== "options") {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (subscription.option_contract_id && subscription.option_contract_id !== print.option_contract_id) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (subscription.underlying_ids?.length) {
|
||||||
|
const underlying = (print.underlying_id ?? "").toUpperCase();
|
||||||
|
return subscription.underlying_ids.map((value) => value.toUpperCase()).includes(underlying);
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
};
|
||||||
|
|
||||||
|
const matchesScopedEquitySubscription = (
|
||||||
|
print: { underlying_id: string },
|
||||||
|
subscription: LiveSubscription
|
||||||
|
): boolean => {
|
||||||
|
if (subscription.channel !== "equities") {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (!subscription.underlying_ids?.length) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
const underlying = print.underlying_id.toUpperCase();
|
||||||
|
return subscription.underlying_ids.map((value) => value.toUpperCase()).includes(underlying);
|
||||||
|
};
|
||||||
|
|
||||||
const buildCandleCacheKey = (underlyingId: string, intervalMs: number): string => {
|
const buildCandleCacheKey = (underlyingId: string, intervalMs: number): string => {
|
||||||
return `candles:equity:${intervalMs}:${underlyingId}`;
|
return `candles:equity:${intervalMs}:${underlyingId}`;
|
||||||
};
|
};
|
||||||
|
|
@ -777,9 +843,38 @@ const run = async () => {
|
||||||
|
|
||||||
const liveState = new LiveStateManager(clickhouse, redis);
|
const liveState = new LiveStateManager(clickhouse, redis);
|
||||||
await liveState.hydrate();
|
await liveState.hydrate();
|
||||||
|
const warnLiveLag = (
|
||||||
|
channel: keyof typeof HOT_LIVE_REDIS_KEYS,
|
||||||
|
ageMs: number | null | undefined
|
||||||
|
) => {
|
||||||
|
if (typeof ageMs !== "number" || !Number.isFinite(ageMs)) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (ageMs < env.LIVE_LAG_WARN_MS) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
logger.warn("live feed lag exceeded threshold", {
|
||||||
|
channel,
|
||||||
|
age_ms: ageMs,
|
||||||
|
threshold_ms: env.LIVE_LAG_WARN_MS
|
||||||
|
});
|
||||||
|
};
|
||||||
const liveStateMetricsTimer = setInterval(() => {
|
const liveStateMetricsTimer = setInterval(() => {
|
||||||
const snapshot = liveState.getStatsSnapshot();
|
const snapshot = liveState.getStatsSnapshot();
|
||||||
logger.info("live cache metrics", snapshot);
|
const hotFeedLagMs = {
|
||||||
|
options: snapshot.freshnessAgeMsByKey[HOT_LIVE_REDIS_KEYS.options] ?? null,
|
||||||
|
equities: snapshot.freshnessAgeMsByKey[HOT_LIVE_REDIS_KEYS.equities] ?? null,
|
||||||
|
flow: snapshot.freshnessAgeMsByKey[HOT_LIVE_REDIS_KEYS.flow] ?? null,
|
||||||
|
nbbo: snapshot.freshnessAgeMsByKey[HOT_LIVE_REDIS_KEYS.nbbo] ?? null
|
||||||
|
};
|
||||||
|
logger.info("live cache metrics", {
|
||||||
|
...snapshot,
|
||||||
|
hotFeedLagMs
|
||||||
|
});
|
||||||
|
warnLiveLag("options", hotFeedLagMs.options);
|
||||||
|
warnLiveLag("equities", hotFeedLagMs.equities);
|
||||||
|
warnLiveLag("flow", hotFeedLagMs.flow);
|
||||||
|
warnLiveLag("nbbo", hotFeedLagMs.nbbo);
|
||||||
}, 60000);
|
}, 60000);
|
||||||
|
|
||||||
const consumerBindings = [
|
const consumerBindings = [
|
||||||
|
|
@ -977,21 +1072,16 @@ const run = async () => {
|
||||||
const fanoutLive = async (
|
const fanoutLive = async (
|
||||||
subscription: LiveSubscription,
|
subscription: LiveSubscription,
|
||||||
item: unknown,
|
item: unknown,
|
||||||
ingestChannel: "options" | "nbbo" | "equities" | "equity-candles" | "equity-overlay" | "equity-joins" | "flow" | "classifier-hits" | "alerts" | "inferred-dark"
|
ingestChannel: "options" | "nbbo" | "equities" | "equity-quotes" | "equity-candles" | "equity-overlay" | "equity-joins" | "flow" | "classifier-hits" | "alerts" | "inferred-dark"
|
||||||
) => {
|
) => {
|
||||||
if (
|
const watermark = await liveState.ingest(ingestChannel, item);
|
||||||
(ingestChannel === "options" ||
|
|
||||||
ingestChannel === "nbbo" ||
|
if (!shouldFanoutLiveEvent(ingestChannel, item)) {
|
||||||
ingestChannel === "equities" ||
|
|
||||||
ingestChannel === "flow") &&
|
|
||||||
!isLiveItemFresh(ingestChannel, item)
|
|
||||||
) {
|
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
const watermark = await liveState.ingest(ingestChannel, item);
|
|
||||||
const matchingSubscriptions =
|
const matchingSubscriptions =
|
||||||
subscription.channel === "options" || subscription.channel === "flow"
|
subscription.channel === "options" || subscription.channel === "flow" || subscription.channel === "equities"
|
||||||
? [...subscriptionDefinitions.entries()].filter(([, candidate]) => candidate.channel === subscription.channel)
|
? [...subscriptionDefinitions.entries()].filter(([, candidate]) => candidate.channel === subscription.channel)
|
||||||
: [[getSubscriptionKey(subscription), subscription] as const];
|
: [[getSubscriptionKey(subscription), subscription] as const];
|
||||||
|
|
||||||
|
|
@ -1007,7 +1097,15 @@ const run = async () => {
|
||||||
|
|
||||||
if (
|
if (
|
||||||
candidate.channel === "options" &&
|
candidate.channel === "options" &&
|
||||||
!matchesOptionPrintFilters(OptionPrintSchema.parse(item), candidate.filters)
|
(!matchesOptionPrintFilters(OptionPrintSchema.parse(item), candidate.filters) ||
|
||||||
|
!matchesScopedOptionSubscription(OptionPrintSchema.parse(item), candidate))
|
||||||
|
) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (
|
||||||
|
candidate.channel === "equities" &&
|
||||||
|
!matchesScopedEquitySubscription(EquityPrintSchema.parse(item), candidate)
|
||||||
) {
|
) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
@ -1088,6 +1186,7 @@ const run = async () => {
|
||||||
try {
|
try {
|
||||||
const payload = EquityQuoteSchema.parse(equityQuoteSubscription.decode(msg));
|
const payload = EquityQuoteSchema.parse(equityQuoteSubscription.decode(msg));
|
||||||
broadcast(equityQuoteSockets, { type: "equity-quote", payload });
|
broadcast(equityQuoteSockets, { type: "equity-quote", payload });
|
||||||
|
await fanoutLive({ channel: "equity-quotes" }, payload, "equity-quotes");
|
||||||
msg.ack();
|
msg.ack();
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
logger.error("failed to process equity quote", {
|
logger.error("failed to process equity quote", {
|
||||||
|
|
@ -1346,7 +1445,7 @@ const run = async () => {
|
||||||
try {
|
try {
|
||||||
const { beforeTs, beforeSeq, limit } = parseBeforeParams(url);
|
const { beforeTs, beforeSeq, limit } = parseBeforeParams(url);
|
||||||
const source = parseReplaySource(url) ?? undefined;
|
const source = parseReplaySource(url) ?? undefined;
|
||||||
const { storageFilters } = parseOptionPrintFilters(url);
|
const storageFilters = parseLiveOptionPrintFilters(url);
|
||||||
const data = await fetchOptionPrintsBefore(
|
const data = await fetchOptionPrintsBefore(
|
||||||
clickhouse,
|
clickhouse,
|
||||||
beforeTs,
|
beforeTs,
|
||||||
|
|
@ -1376,7 +1475,19 @@ const run = async () => {
|
||||||
|
|
||||||
if (req.method === "GET" && url.pathname === "/history/equities") {
|
if (req.method === "GET" && url.pathname === "/history/equities") {
|
||||||
const { beforeTs, beforeSeq, limit } = parseBeforeParams(url);
|
const { beforeTs, beforeSeq, limit } = parseBeforeParams(url);
|
||||||
const data = await fetchEquityPrintsBefore(clickhouse, beforeTs, beforeSeq, limit);
|
const data = await fetchEquityPrintsBefore(
|
||||||
|
clickhouse,
|
||||||
|
beforeTs,
|
||||||
|
beforeSeq,
|
||||||
|
limit,
|
||||||
|
parseLiveEquityPrintFilters(url)
|
||||||
|
);
|
||||||
|
return jsonResponse(buildHistoryResponse(data, (item) => ({ ts: item.ts, seq: item.seq })));
|
||||||
|
}
|
||||||
|
|
||||||
|
if (req.method === "GET" && url.pathname === "/history/equity-quotes") {
|
||||||
|
const { beforeTs, beforeSeq, limit } = parseBeforeParams(url);
|
||||||
|
const data = await fetchEquityQuotesBefore(clickhouse, beforeTs, beforeSeq, limit);
|
||||||
return jsonResponse(buildHistoryResponse(data, (item) => ({ ts: item.ts, seq: item.seq })));
|
return jsonResponse(buildHistoryResponse(data, (item) => ({ ts: item.ts, seq: item.seq })));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -5,12 +5,14 @@ import {
|
||||||
fetchRecentEquityCandles,
|
fetchRecentEquityCandles,
|
||||||
fetchRecentEquityPrintJoins,
|
fetchRecentEquityPrintJoins,
|
||||||
fetchRecentEquityPrints,
|
fetchRecentEquityPrints,
|
||||||
|
fetchRecentEquityQuotes,
|
||||||
fetchRecentFlowPackets,
|
fetchRecentFlowPackets,
|
||||||
fetchRecentInferredDark,
|
fetchRecentInferredDark,
|
||||||
fetchRecentOptionNBBO,
|
fetchRecentOptionNBBO,
|
||||||
type ClickHouseClient
|
type ClickHouseClient
|
||||||
} from "@islandflow/storage";
|
} from "@islandflow/storage";
|
||||||
import type { OptionPrintQueryFilters } from "@islandflow/storage";
|
import type { OptionPrintQueryFilters } from "@islandflow/storage";
|
||||||
|
import type { EquityPrintQueryFilters } from "@islandflow/storage";
|
||||||
import {
|
import {
|
||||||
AlertEventSchema,
|
AlertEventSchema,
|
||||||
ClassifierHitEventSchema,
|
ClassifierHitEventSchema,
|
||||||
|
|
@ -18,6 +20,7 @@ import {
|
||||||
EquityCandleSchema,
|
EquityCandleSchema,
|
||||||
EquityPrintJoinSchema,
|
EquityPrintJoinSchema,
|
||||||
EquityPrintSchema,
|
EquityPrintSchema,
|
||||||
|
EquityQuoteSchema,
|
||||||
FeedSnapshot,
|
FeedSnapshot,
|
||||||
FlowPacketSchema,
|
FlowPacketSchema,
|
||||||
InferredDarkEventSchema,
|
InferredDarkEventSchema,
|
||||||
|
|
@ -36,6 +39,7 @@ import {
|
||||||
import type { RedisClientType } from "redis";
|
import type { RedisClientType } from "redis";
|
||||||
|
|
||||||
const CURSOR_HASH_KEY = "live:cursors";
|
const CURSOR_HASH_KEY = "live:cursors";
|
||||||
|
export const LIVE_FEED_LOOKBACK_MS = 24 * 60 * 60 * 1000;
|
||||||
|
|
||||||
const DEFAULT_GENERIC_LIMIT = 10000;
|
const DEFAULT_GENERIC_LIMIT = 10000;
|
||||||
const MAX_GENERIC_LIMIT = 100000;
|
const MAX_GENERIC_LIMIT = 100000;
|
||||||
|
|
@ -44,6 +48,7 @@ const GENERIC_LIMIT_ENV_KEYS: Record<LiveGenericChannel, string> = {
|
||||||
options: "LIVE_LIMIT_OPTIONS",
|
options: "LIVE_LIMIT_OPTIONS",
|
||||||
nbbo: "LIVE_LIMIT_NBBO",
|
nbbo: "LIVE_LIMIT_NBBO",
|
||||||
equities: "LIVE_LIMIT_EQUITIES",
|
equities: "LIVE_LIMIT_EQUITIES",
|
||||||
|
"equity-quotes": "LIVE_LIMIT_EQUITY_QUOTES",
|
||||||
"equity-joins": "LIVE_LIMIT_EQUITY_JOINS",
|
"equity-joins": "LIVE_LIMIT_EQUITY_JOINS",
|
||||||
flow: "LIVE_LIMIT_FLOW",
|
flow: "LIVE_LIMIT_FLOW",
|
||||||
"classifier-hits": "LIVE_LIMIT_CLASSIFIER_HITS",
|
"classifier-hits": "LIVE_LIMIT_CLASSIFIER_HITS",
|
||||||
|
|
@ -69,6 +74,7 @@ export const LIVE_FRESHNESS_THRESHOLDS: Partial<Record<LiveGenericChannel, numbe
|
||||||
options: 15_000,
|
options: 15_000,
|
||||||
nbbo: 15_000,
|
nbbo: 15_000,
|
||||||
equities: 15_000,
|
equities: 15_000,
|
||||||
|
"equity-quotes": 15_000,
|
||||||
flow: 30_000
|
flow: 30_000
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
@ -102,6 +108,7 @@ export const resolveGenericLiveLimits = (env: NodeJS.ProcessEnv = process.env):
|
||||||
options: parseGenericLimit(env, "options", DEFAULT_GENERIC_LIMIT),
|
options: parseGenericLimit(env, "options", DEFAULT_GENERIC_LIMIT),
|
||||||
nbbo: parseGenericLimit(env, "nbbo", DEFAULT_GENERIC_LIMIT),
|
nbbo: parseGenericLimit(env, "nbbo", DEFAULT_GENERIC_LIMIT),
|
||||||
equities: parseGenericLimit(env, "equities", DEFAULT_GENERIC_LIMIT),
|
equities: parseGenericLimit(env, "equities", DEFAULT_GENERIC_LIMIT),
|
||||||
|
"equity-quotes": parseGenericLimit(env, "equity-quotes", DEFAULT_GENERIC_LIMIT),
|
||||||
"equity-joins": parseGenericLimit(env, "equity-joins", DEFAULT_GENERIC_LIMIT),
|
"equity-joins": parseGenericLimit(env, "equity-joins", DEFAULT_GENERIC_LIMIT),
|
||||||
flow: parseGenericLimit(env, "flow", DEFAULT_GENERIC_LIMIT),
|
flow: parseGenericLimit(env, "flow", DEFAULT_GENERIC_LIMIT),
|
||||||
"classifier-hits": parseGenericLimit(env, "classifier-hits", DEFAULT_GENERIC_LIMIT),
|
"classifier-hits": parseGenericLimit(env, "classifier-hits", DEFAULT_GENERIC_LIMIT),
|
||||||
|
|
@ -154,6 +161,14 @@ const getGenericConfig = (limits: GenericLiveLimits): {
|
||||||
cursor: (item) => ({ ts: item.ts, seq: item.seq }),
|
cursor: (item) => ({ ts: item.ts, seq: item.seq }),
|
||||||
fetchRecent: fetchRecentEquityPrints
|
fetchRecent: fetchRecentEquityPrints
|
||||||
},
|
},
|
||||||
|
"equity-quotes": {
|
||||||
|
redisKey: "live:equity-quotes",
|
||||||
|
cursorField: "equity-quotes",
|
||||||
|
limit: limits["equity-quotes"],
|
||||||
|
parse: (value) => EquityQuoteSchema.parse(value),
|
||||||
|
cursor: (item) => ({ ts: item.ts, seq: item.seq }),
|
||||||
|
fetchRecent: fetchRecentEquityQuotes
|
||||||
|
},
|
||||||
"equity-joins": {
|
"equity-joins": {
|
||||||
redisKey: "live:equity-joins",
|
redisKey: "live:equity-joins",
|
||||||
cursorField: "equity-joins",
|
cursorField: "equity-joins",
|
||||||
|
|
@ -251,14 +266,27 @@ const extractFreshnessTs = (channel: LiveGenericChannel, item: any): number | nu
|
||||||
case "options":
|
case "options":
|
||||||
case "nbbo":
|
case "nbbo":
|
||||||
case "equities":
|
case "equities":
|
||||||
|
case "equity-quotes":
|
||||||
return typeof item.ts === "number" ? item.ts : null;
|
return typeof item.ts === "number" ? item.ts : null;
|
||||||
case "flow":
|
case "flow":
|
||||||
|
case "classifier-hits":
|
||||||
|
case "alerts":
|
||||||
|
case "inferred-dark":
|
||||||
return typeof item.source_ts === "number" ? item.source_ts : null;
|
return typeof item.source_ts === "number" ? item.source_ts : null;
|
||||||
default:
|
default:
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
|
const isWithinLiveFeedLookback = (
|
||||||
|
channel: LiveGenericChannel,
|
||||||
|
item: unknown,
|
||||||
|
now = Date.now()
|
||||||
|
): boolean => {
|
||||||
|
const ts = extractFreshnessTs(channel, item);
|
||||||
|
return ts !== null && now - ts <= LIVE_FEED_LOOKBACK_MS;
|
||||||
|
};
|
||||||
|
|
||||||
export const isLiveItemFresh = (
|
export const isLiveItemFresh = (
|
||||||
channel: LiveGenericChannel,
|
channel: LiveGenericChannel,
|
||||||
item: unknown,
|
item: unknown,
|
||||||
|
|
@ -275,17 +303,11 @@ export const isLiveItemFresh = (
|
||||||
return now - ts <= thresholdMs;
|
return now - ts <= thresholdMs;
|
||||||
};
|
};
|
||||||
|
|
||||||
const filterFreshGenericItems = <T>(
|
export const shouldFanoutLiveEvent = (channel: LiveChannel, item: unknown): boolean => {
|
||||||
channel: LiveGenericChannel,
|
if (channel === "equity-candles" || channel === "equity-overlay") {
|
||||||
items: T[],
|
return true;
|
||||||
now = Date.now()
|
|
||||||
): T[] => {
|
|
||||||
const thresholdMs = LIVE_FRESHNESS_THRESHOLDS[channel];
|
|
||||||
if (!thresholdMs) {
|
|
||||||
return items;
|
|
||||||
}
|
}
|
||||||
|
return isWithinLiveFeedLookback(channel, item);
|
||||||
return items.filter((item) => isLiveItemFresh(channel, item, now));
|
|
||||||
};
|
};
|
||||||
|
|
||||||
const nextBeforeForItems = <T>(items: T[], cursorOf: (item: T) => Cursor): Cursor | null => {
|
const nextBeforeForItems = <T>(items: T[], cursorOf: (item: T) => Cursor): Cursor | null => {
|
||||||
|
|
@ -316,7 +338,8 @@ export class LiveStateManager {
|
||||||
genericHydrateFromRedis: 0,
|
genericHydrateFromRedis: 0,
|
||||||
genericHydrateFromClickHouse: 0,
|
genericHydrateFromClickHouse: 0,
|
||||||
trimOperations: 0,
|
trimOperations: 0,
|
||||||
cacheDepthByKey: new Map<string, number>()
|
cacheDepthByKey: new Map<string, number>(),
|
||||||
|
freshnessAgeMsByKey: new Map<string, number>()
|
||||||
};
|
};
|
||||||
|
|
||||||
constructor(
|
constructor(
|
||||||
|
|
@ -332,15 +355,30 @@ export class LiveStateManager {
|
||||||
genericHydrateFromClickHouse: number;
|
genericHydrateFromClickHouse: number;
|
||||||
trimOperations: number;
|
trimOperations: number;
|
||||||
cacheDepthByKey: Record<string, number>;
|
cacheDepthByKey: Record<string, number>;
|
||||||
|
freshnessAgeMsByKey: Record<string, number>;
|
||||||
} {
|
} {
|
||||||
return {
|
return {
|
||||||
genericHydrateFromRedis: this.stats.genericHydrateFromRedis,
|
genericHydrateFromRedis: this.stats.genericHydrateFromRedis,
|
||||||
genericHydrateFromClickHouse: this.stats.genericHydrateFromClickHouse,
|
genericHydrateFromClickHouse: this.stats.genericHydrateFromClickHouse,
|
||||||
trimOperations: this.stats.trimOperations,
|
trimOperations: this.stats.trimOperations,
|
||||||
cacheDepthByKey: Object.fromEntries(this.stats.cacheDepthByKey)
|
cacheDepthByKey: Object.fromEntries(this.stats.cacheDepthByKey),
|
||||||
|
freshnessAgeMsByKey: Object.fromEntries(this.stats.freshnessAgeMsByKey)
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private updateFreshnessMetric(listKey: string, channel: LiveChannel, item: unknown, now = Date.now()): void {
|
||||||
|
const ts =
|
||||||
|
channel === "equity-candles" || channel === "equity-overlay"
|
||||||
|
? typeof (item as { ts?: unknown })?.ts === "number"
|
||||||
|
? ((item as { ts: number }).ts as number)
|
||||||
|
: null
|
||||||
|
: extractFreshnessTs(channel, item);
|
||||||
|
|
||||||
|
if (typeof ts === "number" && Number.isFinite(ts)) {
|
||||||
|
this.stats.freshnessAgeMsByKey.set(listKey, Math.max(0, now - ts));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
async hydrate(): Promise<void> {
|
async hydrate(): Promise<void> {
|
||||||
const channels = Object.keys(this.generic) as LiveGenericChannel[];
|
const channels = Object.keys(this.generic) as LiveGenericChannel[];
|
||||||
await Promise.all(channels.map((channel) => this.hydrateGeneric(channel)));
|
await Promise.all(channels.map((channel) => this.hydrateGeneric(channel)));
|
||||||
|
|
@ -350,11 +388,18 @@ export class LiveStateManager {
|
||||||
const config = this.generic[channel];
|
const config = this.generic[channel];
|
||||||
if (this.redis?.isOpen) {
|
if (this.redis?.isOpen) {
|
||||||
const payloads = await this.redis.lRange(config.redisKey, 0, config.limit - 1);
|
const payloads = await this.redis.lRange(config.redisKey, 0, config.limit - 1);
|
||||||
const cached = normalizeGenericItems(channel, parseJsonList(payloads, config.parse), config);
|
const cached = normalizeGenericItems(
|
||||||
|
channel,
|
||||||
|
parseJsonList(payloads, config.parse).filter((item) =>
|
||||||
|
isWithinLiveFeedLookback(channel, item)
|
||||||
|
),
|
||||||
|
config
|
||||||
|
);
|
||||||
if (cached.length > 0) {
|
if (cached.length > 0) {
|
||||||
this.genericItems.set(channel, cached);
|
this.genericItems.set(channel, cached);
|
||||||
this.stats.genericHydrateFromRedis += 1;
|
this.stats.genericHydrateFromRedis += 1;
|
||||||
this.stats.cacheDepthByKey.set(config.redisKey, cached.length);
|
this.stats.cacheDepthByKey.set(config.redisKey, cached.length);
|
||||||
|
this.updateFreshnessMetric(config.redisKey, channel, cached[0]);
|
||||||
this.genericCursors.set(config.cursorField, parseCursor(await this.redis.hGet(CURSOR_HASH_KEY, config.cursorField)));
|
this.genericCursors.set(config.cursorField, parseCursor(await this.redis.hGet(CURSOR_HASH_KEY, config.cursorField)));
|
||||||
await this.persistList(
|
await this.persistList(
|
||||||
config.redisKey,
|
config.redisKey,
|
||||||
|
|
@ -367,10 +412,19 @@ export class LiveStateManager {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
const fresh = normalizeGenericItems(channel, await config.fetchRecent(this.clickhouse, config.limit), config);
|
const fresh = normalizeGenericItems(
|
||||||
|
channel,
|
||||||
|
(await config.fetchRecent(this.clickhouse, config.limit)).filter((item) =>
|
||||||
|
isWithinLiveFeedLookback(channel, item)
|
||||||
|
),
|
||||||
|
config
|
||||||
|
);
|
||||||
this.stats.genericHydrateFromClickHouse += 1;
|
this.stats.genericHydrateFromClickHouse += 1;
|
||||||
this.stats.cacheDepthByKey.set(config.redisKey, fresh.length);
|
this.stats.cacheDepthByKey.set(config.redisKey, fresh.length);
|
||||||
this.genericItems.set(channel, fresh);
|
this.genericItems.set(channel, fresh);
|
||||||
|
if (fresh.length > 0) {
|
||||||
|
this.updateFreshnessMetric(config.redisKey, channel, fresh[0]);
|
||||||
|
}
|
||||||
const watermark = fresh[0] ? config.cursor(fresh[0]) : null;
|
const watermark = fresh[0] ? config.cursor(fresh[0]) : null;
|
||||||
this.genericCursors.set(config.cursorField, watermark);
|
this.genericCursors.set(config.cursorField, watermark);
|
||||||
await this.persistList(config.redisKey, config.cursorField, fresh, config.limit, watermark);
|
await this.persistList(config.redisKey, config.cursorField, fresh, config.limit, watermark);
|
||||||
|
|
@ -379,16 +433,21 @@ export class LiveStateManager {
|
||||||
async getSnapshot(subscription: LiveSubscription): Promise<FeedSnapshot<unknown>> {
|
async getSnapshot(subscription: LiveSubscription): Promise<FeedSnapshot<unknown>> {
|
||||||
switch (subscription.channel) {
|
switch (subscription.channel) {
|
||||||
case "options": {
|
case "options": {
|
||||||
if (subscription.filters?.view === "raw") {
|
const scoped =
|
||||||
|
Boolean(subscription.underlying_ids?.length) || Boolean(subscription.option_contract_id);
|
||||||
|
if (subscription.filters?.view === "raw" || scoped) {
|
||||||
const storageFilters: OptionPrintQueryFilters = {
|
const storageFilters: OptionPrintQueryFilters = {
|
||||||
view: "raw",
|
view: subscription.filters?.view ?? "signal",
|
||||||
security:
|
security:
|
||||||
subscription.filters.securityTypes?.length === 1
|
subscription.filters?.securityTypes?.length === 1
|
||||||
? subscription.filters.securityTypes[0]
|
? subscription.filters.securityTypes[0]
|
||||||
: "all",
|
: "all",
|
||||||
nbboSides: subscription.filters.nbboSides,
|
nbboSides: subscription.filters?.nbboSides,
|
||||||
optionTypes: subscription.filters.optionTypes,
|
optionTypes: subscription.filters?.optionTypes,
|
||||||
minNotional: subscription.filters.minNotional
|
minNotional: subscription.filters?.minNotional,
|
||||||
|
underlyingIds: subscription.underlying_ids,
|
||||||
|
optionContractId: subscription.option_contract_id,
|
||||||
|
sinceTs: Date.now() - LIVE_FEED_LOOKBACK_MS
|
||||||
};
|
};
|
||||||
const items = await fetchRecentOptionPrints(
|
const items = await fetchRecentOptionPrints(
|
||||||
this.clickhouse,
|
this.clickhouse,
|
||||||
|
|
@ -396,21 +455,18 @@ export class LiveStateManager {
|
||||||
undefined,
|
undefined,
|
||||||
storageFilters
|
storageFilters
|
||||||
);
|
);
|
||||||
const freshItems = filterFreshGenericItems("options", items);
|
|
||||||
return {
|
return {
|
||||||
subscription,
|
subscription,
|
||||||
items: freshItems,
|
items,
|
||||||
watermark: items[0] ? { ts: items[0].ts, seq: items[0].seq } : null,
|
watermark: items[0] ? { ts: items[0].ts, seq: items[0].seq } : null,
|
||||||
next_before: nextBeforeForItems(freshItems, (item) => ({ ts: item.ts, seq: item.seq }))
|
next_before: nextBeforeForItems(items, (item) => ({ ts: item.ts, seq: item.seq }))
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
const config = this.generic.options;
|
const config = this.generic.options;
|
||||||
const items = filterFreshGenericItems(
|
const items = (this.genericItems.get("options") ?? []).filter((item) =>
|
||||||
"options",
|
isWithinLiveFeedLookback("options", item) &&
|
||||||
(this.genericItems.get("options") ?? []).filter((item) =>
|
|
||||||
matchesOptionPrintFilters(item, subscription.filters)
|
matchesOptionPrintFilters(item, subscription.filters)
|
||||||
)
|
|
||||||
);
|
);
|
||||||
return {
|
return {
|
||||||
subscription,
|
subscription,
|
||||||
|
|
@ -421,11 +477,34 @@ export class LiveStateManager {
|
||||||
}
|
}
|
||||||
case "flow": {
|
case "flow": {
|
||||||
const config = this.generic.flow;
|
const config = this.generic.flow;
|
||||||
const items = filterFreshGenericItems(
|
const items = (this.genericItems.get("flow") ?? []).filter((item) =>
|
||||||
"flow",
|
isWithinLiveFeedLookback("flow", item) &&
|
||||||
(this.genericItems.get("flow") ?? []).filter((item) =>
|
|
||||||
matchesFlowPacketFilters(item, subscription.filters)
|
matchesFlowPacketFilters(item, subscription.filters)
|
||||||
)
|
);
|
||||||
|
return {
|
||||||
|
subscription,
|
||||||
|
items,
|
||||||
|
watermark: this.genericCursors.get(config.cursorField) ?? null,
|
||||||
|
next_before: nextBeforeForItems(items, config.cursor)
|
||||||
|
};
|
||||||
|
}
|
||||||
|
case "equities": {
|
||||||
|
const config = this.generic.equities;
|
||||||
|
if (subscription.underlying_ids?.length) {
|
||||||
|
const filters: EquityPrintQueryFilters = {
|
||||||
|
underlyingIds: subscription.underlying_ids,
|
||||||
|
sinceTs: Date.now() - LIVE_FEED_LOOKBACK_MS
|
||||||
|
};
|
||||||
|
const items = await fetchRecentEquityPrints(this.clickhouse, config.limit, filters);
|
||||||
|
return {
|
||||||
|
subscription,
|
||||||
|
items,
|
||||||
|
watermark: items[0] ? { ts: items[0].ts, seq: items[0].seq } : null,
|
||||||
|
next_before: nextBeforeForItems(items, config.cursor)
|
||||||
|
};
|
||||||
|
}
|
||||||
|
const items = (this.genericItems.get("equities") ?? []).filter((item) =>
|
||||||
|
isWithinLiveFeedLookback("equities", item)
|
||||||
);
|
);
|
||||||
return {
|
return {
|
||||||
subscription,
|
subscription,
|
||||||
|
|
@ -464,9 +543,8 @@ export class LiveStateManager {
|
||||||
}
|
}
|
||||||
default: {
|
default: {
|
||||||
const config = this.generic[subscription.channel];
|
const config = this.generic[subscription.channel];
|
||||||
const items = filterFreshGenericItems(
|
const items = (this.genericItems.get(subscription.channel) ?? []).filter((item) =>
|
||||||
subscription.channel,
|
isWithinLiveFeedLookback(subscription.channel, item)
|
||||||
this.genericItems.get(subscription.channel) ?? []
|
|
||||||
);
|
);
|
||||||
return {
|
return {
|
||||||
subscription,
|
subscription,
|
||||||
|
|
@ -484,6 +562,7 @@ export class LiveStateManager {
|
||||||
const candle = EquityCandleSchema.parse(item);
|
const candle = EquityCandleSchema.parse(item);
|
||||||
const key = candleRedisKey(candle.underlying_id, candle.interval_ms);
|
const key = candleRedisKey(candle.underlying_id, candle.interval_ms);
|
||||||
const cursorField = candleCursorField(candle.underlying_id, candle.interval_ms);
|
const cursorField = candleCursorField(candle.underlying_id, candle.interval_ms);
|
||||||
|
const previousCursor = this.candleCursors.get(cursorField) ?? null;
|
||||||
const items = this.candleItems.get(key) ?? [];
|
const items = this.candleItems.get(key) ?? [];
|
||||||
const next = [candle, ...items]
|
const next = [candle, ...items]
|
||||||
.sort((a, b) => (b.ts - a.ts) || (b.seq - a.seq))
|
.sort((a, b) => (b.ts - a.ts) || (b.seq - a.seq))
|
||||||
|
|
@ -492,13 +571,22 @@ export class LiveStateManager {
|
||||||
this.stats.cacheDepthByKey.set(key, next.length);
|
this.stats.cacheDepthByKey.set(key, next.length);
|
||||||
const cursor = { ts: candle.ts, seq: candle.seq };
|
const cursor = { ts: candle.ts, seq: candle.seq };
|
||||||
this.candleCursors.set(cursorField, cursor);
|
this.candleCursors.set(cursorField, cursor);
|
||||||
|
if (next.length > 0) {
|
||||||
|
this.updateFreshnessMetric(key, "equity-candles", next[0]);
|
||||||
|
}
|
||||||
|
const outOfOrder = previousCursor ? compareCursors(cursor, previousCursor) > 0 : false;
|
||||||
|
if (outOfOrder) {
|
||||||
await this.persistList(key, cursorField, next, CHART_LIMITS.candles, cursor);
|
await this.persistList(key, cursorField, next, CHART_LIMITS.candles, cursor);
|
||||||
|
} else {
|
||||||
|
await this.persistItem(key, cursorField, candle, CHART_LIMITS.candles, cursor, next.length);
|
||||||
|
}
|
||||||
return cursor;
|
return cursor;
|
||||||
}
|
}
|
||||||
case "equity-overlay": {
|
case "equity-overlay": {
|
||||||
const print = EquityPrintSchema.parse(item);
|
const print = EquityPrintSchema.parse(item);
|
||||||
const key = overlayRedisKey(print.underlying_id);
|
const key = overlayRedisKey(print.underlying_id);
|
||||||
const cursorField = overlayCursorField(print.underlying_id);
|
const cursorField = overlayCursorField(print.underlying_id);
|
||||||
|
const previousCursor = this.overlayCursors.get(cursorField) ?? null;
|
||||||
const items = this.overlayItems.get(key) ?? [];
|
const items = this.overlayItems.get(key) ?? [];
|
||||||
const next = [print, ...items]
|
const next = [print, ...items]
|
||||||
.sort((a, b) => (b.ts - a.ts) || (b.seq - a.seq))
|
.sort((a, b) => (b.ts - a.ts) || (b.seq - a.seq))
|
||||||
|
|
@ -507,22 +595,39 @@ export class LiveStateManager {
|
||||||
this.stats.cacheDepthByKey.set(key, next.length);
|
this.stats.cacheDepthByKey.set(key, next.length);
|
||||||
const cursor = { ts: print.ts, seq: print.seq };
|
const cursor = { ts: print.ts, seq: print.seq };
|
||||||
this.overlayCursors.set(cursorField, cursor);
|
this.overlayCursors.set(cursorField, cursor);
|
||||||
|
if (next.length > 0) {
|
||||||
|
this.updateFreshnessMetric(key, "equity-overlay", next[0]);
|
||||||
|
}
|
||||||
|
const outOfOrder = previousCursor ? compareCursors(cursor, previousCursor) > 0 : false;
|
||||||
|
if (outOfOrder) {
|
||||||
await this.persistList(key, cursorField, next, CHART_LIMITS.overlay, cursor);
|
await this.persistList(key, cursorField, next, CHART_LIMITS.overlay, cursor);
|
||||||
|
} else {
|
||||||
|
await this.persistItem(key, cursorField, print, CHART_LIMITS.overlay, cursor, next.length);
|
||||||
|
}
|
||||||
return cursor;
|
return cursor;
|
||||||
}
|
}
|
||||||
default: {
|
default: {
|
||||||
const config = this.generic[channel];
|
const config = this.generic[channel];
|
||||||
const parsed = config.parse(item);
|
const parsed = config.parse(item);
|
||||||
if (!isLiveItemFresh(channel, parsed)) {
|
if (!isWithinLiveFeedLookback(channel, parsed)) {
|
||||||
return this.genericCursors.get(config.cursorField) ?? null;
|
return null;
|
||||||
}
|
}
|
||||||
|
const previousCursor = this.genericCursors.get(config.cursorField) ?? null;
|
||||||
const items = this.genericItems.get(channel) ?? [];
|
const items = this.genericItems.get(channel) ?? [];
|
||||||
const next = normalizeGenericItems(channel, [parsed, ...items], config);
|
const next = normalizeGenericItems(channel, [parsed, ...items], config);
|
||||||
this.genericItems.set(channel, next);
|
this.genericItems.set(channel, next);
|
||||||
this.stats.cacheDepthByKey.set(config.redisKey, next.length);
|
this.stats.cacheDepthByKey.set(config.redisKey, next.length);
|
||||||
const cursor = config.cursor(parsed);
|
const cursor = config.cursor(parsed);
|
||||||
this.genericCursors.set(config.cursorField, cursor);
|
this.genericCursors.set(config.cursorField, cursor);
|
||||||
|
if (next.length > 0) {
|
||||||
|
this.updateFreshnessMetric(config.redisKey, channel, next[0]);
|
||||||
|
}
|
||||||
|
const outOfOrder = previousCursor ? compareCursors(cursor, previousCursor) > 0 : false;
|
||||||
|
if (channel === "nbbo" || outOfOrder) {
|
||||||
await this.persistList(config.redisKey, config.cursorField, next, config.limit, cursor);
|
await this.persistList(config.redisKey, config.cursorField, next, config.limit, cursor);
|
||||||
|
} else {
|
||||||
|
await this.persistItem(config.redisKey, config.cursorField, parsed, config.limit, cursor, next.length);
|
||||||
|
}
|
||||||
return cursor;
|
return cursor;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -537,6 +642,7 @@ export class LiveStateManager {
|
||||||
if (cached.length > 0) {
|
if (cached.length > 0) {
|
||||||
this.candleItems.set(key, cached);
|
this.candleItems.set(key, cached);
|
||||||
this.stats.cacheDepthByKey.set(key, cached.length);
|
this.stats.cacheDepthByKey.set(key, cached.length);
|
||||||
|
this.updateFreshnessMetric(key, "equity-candles", cached[0]);
|
||||||
this.candleCursors.set(cursorField, parseCursor(await this.redis.hGet(CURSOR_HASH_KEY, cursorField)));
|
this.candleCursors.set(cursorField, parseCursor(await this.redis.hGet(CURSOR_HASH_KEY, cursorField)));
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
@ -545,6 +651,9 @@ export class LiveStateManager {
|
||||||
const fresh = await fetchRecentEquityCandles(this.clickhouse, underlyingId, intervalMs, CHART_LIMITS.candles);
|
const fresh = await fetchRecentEquityCandles(this.clickhouse, underlyingId, intervalMs, CHART_LIMITS.candles);
|
||||||
this.candleItems.set(key, fresh);
|
this.candleItems.set(key, fresh);
|
||||||
this.stats.cacheDepthByKey.set(key, fresh.length);
|
this.stats.cacheDepthByKey.set(key, fresh.length);
|
||||||
|
if (fresh.length > 0) {
|
||||||
|
this.updateFreshnessMetric(key, "equity-candles", fresh[0]);
|
||||||
|
}
|
||||||
const watermark = fresh[0] ? { ts: fresh[0].ts, seq: fresh[0].seq } : null;
|
const watermark = fresh[0] ? { ts: fresh[0].ts, seq: fresh[0].seq } : null;
|
||||||
this.candleCursors.set(cursorField, watermark);
|
this.candleCursors.set(cursorField, watermark);
|
||||||
await this.persistList(key, cursorField, fresh, CHART_LIMITS.candles, watermark);
|
await this.persistList(key, cursorField, fresh, CHART_LIMITS.candles, watermark);
|
||||||
|
|
@ -559,6 +668,7 @@ export class LiveStateManager {
|
||||||
if (cached.length > 0) {
|
if (cached.length > 0) {
|
||||||
this.overlayItems.set(key, cached);
|
this.overlayItems.set(key, cached);
|
||||||
this.stats.cacheDepthByKey.set(key, cached.length);
|
this.stats.cacheDepthByKey.set(key, cached.length);
|
||||||
|
this.updateFreshnessMetric(key, "equity-overlay", cached[0]);
|
||||||
this.overlayCursors.set(cursorField, parseCursor(await this.redis.hGet(CURSOR_HASH_KEY, cursorField)));
|
this.overlayCursors.set(cursorField, parseCursor(await this.redis.hGet(CURSOR_HASH_KEY, cursorField)));
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
@ -569,11 +679,33 @@ export class LiveStateManager {
|
||||||
);
|
);
|
||||||
this.overlayItems.set(key, fresh);
|
this.overlayItems.set(key, fresh);
|
||||||
this.stats.cacheDepthByKey.set(key, fresh.length);
|
this.stats.cacheDepthByKey.set(key, fresh.length);
|
||||||
|
if (fresh.length > 0) {
|
||||||
|
this.updateFreshnessMetric(key, "equity-overlay", fresh[0]);
|
||||||
|
}
|
||||||
const watermark = fresh[0] ? { ts: fresh[0].ts, seq: fresh[0].seq } : null;
|
const watermark = fresh[0] ? { ts: fresh[0].ts, seq: fresh[0].seq } : null;
|
||||||
this.overlayCursors.set(cursorField, watermark);
|
this.overlayCursors.set(cursorField, watermark);
|
||||||
await this.persistList(key, cursorField, fresh, CHART_LIMITS.overlay, watermark);
|
await this.persistList(key, cursorField, fresh, CHART_LIMITS.overlay, watermark);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private async persistItem<T>(
|
||||||
|
listKey: string,
|
||||||
|
cursorField: string,
|
||||||
|
item: T,
|
||||||
|
limit: number,
|
||||||
|
cursor: Cursor | null,
|
||||||
|
depth: number
|
||||||
|
): Promise<void> {
|
||||||
|
if (!this.redis?.isOpen) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
await this.redis.lPush(listKey, JSON.stringify(item));
|
||||||
|
await this.redis.lTrim(listKey, 0, limit - 1);
|
||||||
|
this.stats.trimOperations += 1;
|
||||||
|
this.stats.cacheDepthByKey.set(listKey, Math.min(depth, limit));
|
||||||
|
await this.redis.hSet(CURSOR_HASH_KEY, cursorField, JSON.stringify(cursor));
|
||||||
|
}
|
||||||
|
|
||||||
private async persistList<T>(
|
private async persistList<T>(
|
||||||
listKey: string,
|
listKey: string,
|
||||||
cursorField: string,
|
cursorField: string,
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,11 @@
|
||||||
import { describe, expect, it } from "bun:test";
|
import { describe, expect, it } from "bun:test";
|
||||||
import type { ClickHouseClient } from "@islandflow/storage";
|
import type { ClickHouseClient } from "@islandflow/storage";
|
||||||
import { LiveStateManager, isLiveItemFresh, resolveGenericLiveLimits } from "../src/live";
|
import {
|
||||||
|
LiveStateManager,
|
||||||
|
isLiveItemFresh,
|
||||||
|
resolveGenericLiveLimits,
|
||||||
|
shouldFanoutLiveEvent
|
||||||
|
} from "../src/live";
|
||||||
|
|
||||||
const makeClickHouse = (): ClickHouseClient =>
|
const makeClickHouse = (): ClickHouseClient =>
|
||||||
({
|
({
|
||||||
|
|
@ -58,6 +63,7 @@ describe("LiveStateManager", () => {
|
||||||
expect(limits.options).toBe(777);
|
expect(limits.options).toBe(777);
|
||||||
expect(limits.nbbo).toBe(100000);
|
expect(limits.nbbo).toBe(100000);
|
||||||
expect(limits.flow).toBe(10000);
|
expect(limits.flow).toBe(10000);
|
||||||
|
expect(limits["equity-quotes"]).toBe(10000);
|
||||||
expect(limits.alerts).toBe(10000);
|
expect(limits.alerts).toBe(10000);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|
@ -145,6 +151,7 @@ describe("LiveStateManager", () => {
|
||||||
options: 10000,
|
options: 10000,
|
||||||
nbbo: 10000,
|
nbbo: 10000,
|
||||||
equities: 10000,
|
equities: 10000,
|
||||||
|
"equity-quotes": 10000,
|
||||||
"equity-joins": 10000,
|
"equity-joins": 10000,
|
||||||
flow: 2,
|
flow: 2,
|
||||||
"classifier-hits": 10000,
|
"classifier-hits": 10000,
|
||||||
|
|
@ -277,7 +284,7 @@ describe("LiveStateManager", () => {
|
||||||
expect(flowSnapshot.items).toHaveLength(1);
|
expect(flowSnapshot.items).toHaveLength(1);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("suppresses stale items from live snapshots while preserving fresh ones", async () => {
|
it("keeps stale persisted items in live snapshots", async () => {
|
||||||
const manager = new LiveStateManager(makeClickHouse(), null);
|
const manager = new LiveStateManager(makeClickHouse(), null);
|
||||||
const now = Date.now();
|
const now = Date.now();
|
||||||
|
|
||||||
|
|
@ -383,16 +390,20 @@ describe("LiveStateManager", () => {
|
||||||
]);
|
]);
|
||||||
|
|
||||||
expect((optionsSnapshot.items as Array<{ trace_id: string }>).map((item) => item.trace_id)).toEqual([
|
expect((optionsSnapshot.items as Array<{ trace_id: string }>).map((item) => item.trace_id)).toEqual([
|
||||||
"opt-fresh"
|
"opt-fresh",
|
||||||
|
"opt-stale"
|
||||||
]);
|
]);
|
||||||
expect((nbboSnapshot.items as Array<{ trace_id: string }>).map((item) => item.trace_id)).toEqual([
|
expect((nbboSnapshot.items as Array<{ trace_id: string }>).map((item) => item.trace_id)).toEqual([
|
||||||
"nbbo-fresh"
|
"nbbo-fresh",
|
||||||
|
"nbbo-stale"
|
||||||
]);
|
]);
|
||||||
expect((equitiesSnapshot.items as Array<{ trace_id: string }>).map((item) => item.trace_id)).toEqual([
|
expect((equitiesSnapshot.items as Array<{ trace_id: string }>).map((item) => item.trace_id)).toEqual([
|
||||||
"eq-fresh"
|
"eq-fresh",
|
||||||
|
"eq-stale"
|
||||||
]);
|
]);
|
||||||
expect((flowSnapshot.items as Array<{ id: string }>).map((item) => item.id)).toEqual([
|
expect((flowSnapshot.items as Array<{ id: string }>).map((item) => item.id)).toEqual([
|
||||||
"flow-fresh"
|
"flow-fresh",
|
||||||
|
"flow-stale"
|
||||||
]);
|
]);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|
@ -476,7 +487,7 @@ describe("LiveStateManager", () => {
|
||||||
]);
|
]);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("rejects stale ingest for freshness-gated channels", async () => {
|
it("stores older valid ingest for freshness-gated channels", async () => {
|
||||||
const manager = new LiveStateManager(makeClickHouse(), null);
|
const manager = new LiveStateManager(makeClickHouse(), null);
|
||||||
const now = Date.now();
|
const now = Date.now();
|
||||||
|
|
||||||
|
|
@ -494,12 +505,84 @@ describe("LiveStateManager", () => {
|
||||||
});
|
});
|
||||||
|
|
||||||
const snapshot = await manager.getSnapshot({ channel: "equities" });
|
const snapshot = await manager.getSnapshot({ channel: "equities" });
|
||||||
expect(snapshot.items).toHaveLength(0);
|
expect(snapshot.items).toHaveLength(1);
|
||||||
|
expect(snapshot.next_before).toEqual({ ts: now - 60_000, seq: 1 });
|
||||||
});
|
});
|
||||||
|
|
||||||
it("exposes freshness helper for event fanout gating", () => {
|
it("hydrates equity quotes from redis", async () => {
|
||||||
|
const redis = makeRedis();
|
||||||
|
const now = Date.now();
|
||||||
|
await redis.lPush(
|
||||||
|
"live:equity-quotes",
|
||||||
|
JSON.stringify({
|
||||||
|
source_ts: now,
|
||||||
|
ingest_ts: now + 1,
|
||||||
|
seq: 1,
|
||||||
|
trace_id: "quote-1",
|
||||||
|
ts: now,
|
||||||
|
underlying_id: "SPY",
|
||||||
|
bid: 450,
|
||||||
|
ask: 450.01
|
||||||
|
})
|
||||||
|
);
|
||||||
|
await redis.hSet("live:cursors", "equity-quotes", JSON.stringify({ ts: now, seq: 1 }));
|
||||||
|
|
||||||
|
const manager = new LiveStateManager(makeClickHouse(), redis as never);
|
||||||
|
await manager.hydrate();
|
||||||
|
const snapshot = await manager.getSnapshot({ channel: "equity-quotes" });
|
||||||
|
|
||||||
|
expect(snapshot.items).toHaveLength(1);
|
||||||
|
expect(snapshot.watermark).toEqual({ ts: now, seq: 1 });
|
||||||
|
expect(snapshot.next_before).toEqual({ ts: now, seq: 1 });
|
||||||
|
});
|
||||||
|
|
||||||
|
it("hydrates equity quotes from clickhouse when redis is empty and persists hot cache", async () => {
|
||||||
|
const redis = makeRedis();
|
||||||
|
const now = Date.now();
|
||||||
|
const clickhouse = {
|
||||||
|
...makeClickHouse(),
|
||||||
|
query: async ({ query }: { query: string }) => ({
|
||||||
|
async json<T>() {
|
||||||
|
if (query.includes("equity_quotes")) {
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
source_ts: now,
|
||||||
|
ingest_ts: now + 1,
|
||||||
|
seq: 2,
|
||||||
|
trace_id: "quote-2",
|
||||||
|
ts: now,
|
||||||
|
underlying_id: "SPY",
|
||||||
|
bid: 451,
|
||||||
|
ask: 451.01
|
||||||
|
}
|
||||||
|
] as T;
|
||||||
|
}
|
||||||
|
return [] as T;
|
||||||
|
}
|
||||||
|
})
|
||||||
|
} as ClickHouseClient;
|
||||||
|
|
||||||
|
const manager = new LiveStateManager(clickhouse, redis as never);
|
||||||
|
await manager.hydrate();
|
||||||
|
const snapshot = await manager.getSnapshot({ channel: "equity-quotes" });
|
||||||
|
const persisted = await redis.lRange("live:equity-quotes", 0, 10);
|
||||||
|
|
||||||
|
expect(snapshot.items).toHaveLength(1);
|
||||||
|
expect(snapshot.watermark).toEqual({ ts: now, seq: 2 });
|
||||||
|
expect(persisted).toHaveLength(1);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("exposes freshness helper for feed status", () => {
|
||||||
expect(isLiveItemFresh("options", { ts: 1000 }, 1010)).toBe(true);
|
expect(isLiveItemFresh("options", { ts: 1000 }, 1010)).toBe(true);
|
||||||
expect(isLiveItemFresh("options", { ts: 1000 }, 20_001)).toBe(false);
|
expect(isLiveItemFresh("options", { ts: 1000 }, 20_001)).toBe(false);
|
||||||
expect(isLiveItemFresh("equity-joins", { source_ts: 1 }, 1_000_000)).toBe(true);
|
expect(isLiveItemFresh("equity-joins", { source_ts: 1 }, 1_000_000)).toBe(true);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("gates live feed fanout to the rolling visibility window", () => {
|
||||||
|
const now = Date.now();
|
||||||
|
expect(shouldFanoutLiveEvent("options", { ts: now })).toBe(true);
|
||||||
|
expect(shouldFanoutLiveEvent("equities", { ts: now - 25 * 60 * 60 * 1000 })).toBe(false);
|
||||||
|
expect(shouldFanoutLiveEvent("flow", { source_ts: now - 25 * 60 * 60 * 1000 })).toBe(false);
|
||||||
|
expect(shouldFanoutLiveEvent("equity-candles", { ts: 1000 })).toBe(true);
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|
|
||||||
|
|
@ -13,6 +13,9 @@ type SyntheticOptionsAdapterConfig = {
|
||||||
|
|
||||||
type Burst = {
|
type Burst = {
|
||||||
contractId: string;
|
contractId: string;
|
||||||
|
underlying: number;
|
||||||
|
expiryOffsetDays: number;
|
||||||
|
strike: number;
|
||||||
basePrice: number;
|
basePrice: number;
|
||||||
baseSize: number;
|
baseSize: number;
|
||||||
exchange: string;
|
exchange: string;
|
||||||
|
|
@ -23,7 +26,16 @@ type Burst = {
|
||||||
seed: number;
|
seed: number;
|
||||||
};
|
};
|
||||||
|
|
||||||
|
export type SyntheticContractIvState = {
|
||||||
|
iv: number;
|
||||||
|
pressure: number;
|
||||||
|
lastTs: number;
|
||||||
|
};
|
||||||
|
|
||||||
const OPTION_CONTRACT_MULTIPLIER = 100;
|
const OPTION_CONTRACT_MULTIPLIER = 100;
|
||||||
|
const IV_MIN = 0.05;
|
||||||
|
const IV_MAX = 2.5;
|
||||||
|
const IV_DECAY_HALF_LIFE_MS = 60_000;
|
||||||
|
|
||||||
const SYNTHETIC_SYMBOLS = ["SPY", ...(SP500_SYMBOLS as readonly string[])];
|
const SYNTHETIC_SYMBOLS = ["SPY", ...(SP500_SYMBOLS as readonly string[])];
|
||||||
const MS_PER_DAY = 24 * 60 * 60 * 1000;
|
const MS_PER_DAY = 24 * 60 * 60 * 1000;
|
||||||
|
|
@ -36,7 +48,7 @@ type SyntheticOptionsProfile = {
|
||||||
pricePlacements: Record<string, WeightedValue<PricePlacement>[]>;
|
pricePlacements: Record<string, WeightedValue<PricePlacement>[]>;
|
||||||
};
|
};
|
||||||
|
|
||||||
type PricePlacement = "AA" | "A" | "MID" | "B" | "BB";
|
export type PricePlacement = "AA" | "A" | "MID" | "B" | "BB";
|
||||||
|
|
||||||
type WeightedValue<T> = {
|
type WeightedValue<T> = {
|
||||||
value: T;
|
value: T;
|
||||||
|
|
@ -347,6 +359,55 @@ const formatExpiry = (now: number, offsetDays: number): string => {
|
||||||
return expiryDate.toISOString().slice(0, 10);
|
return expiryDate.toISOString().slice(0, 10);
|
||||||
};
|
};
|
||||||
|
|
||||||
|
const clampValue = (value: number, min: number, max: number): number => {
|
||||||
|
if (!Number.isFinite(value)) {
|
||||||
|
return min;
|
||||||
|
}
|
||||||
|
return Math.max(min, Math.min(max, value));
|
||||||
|
};
|
||||||
|
|
||||||
|
const initializeSyntheticIv = (dteDays: number, moneyness: number): number => {
|
||||||
|
const dteBoost = dteDays <= 0 ? 0.22 : dteDays <= 7 ? 0.14 : dteDays <= 30 ? 0.06 : 0;
|
||||||
|
const moneynessBoost = clampValue(Math.abs(moneyness - 1) * 0.8, 0, 0.2);
|
||||||
|
return clampValue(0.24 + dteBoost + moneynessBoost, 0.18, 0.65);
|
||||||
|
};
|
||||||
|
|
||||||
|
export const updateSyntheticIvForTest = (
|
||||||
|
state: SyntheticContractIvState | undefined,
|
||||||
|
input: {
|
||||||
|
ts: number;
|
||||||
|
placement: PricePlacement;
|
||||||
|
size: number;
|
||||||
|
notional: number;
|
||||||
|
dteDays: number;
|
||||||
|
moneyness: number;
|
||||||
|
}
|
||||||
|
): SyntheticContractIvState => {
|
||||||
|
const previous = state ?? {
|
||||||
|
iv: initializeSyntheticIv(input.dteDays, input.moneyness),
|
||||||
|
pressure: 0,
|
||||||
|
lastTs: input.ts
|
||||||
|
};
|
||||||
|
const elapsed = Math.max(0, input.ts - previous.lastTs);
|
||||||
|
const decay = Math.pow(0.5, elapsed / IV_DECAY_HALF_LIFE_MS);
|
||||||
|
let pressure = previous.pressure * decay;
|
||||||
|
|
||||||
|
if (input.placement === "AA" || input.placement === "A") {
|
||||||
|
const sizeImpact = Math.log10(Math.max(10, input.size)) * 0.012;
|
||||||
|
const notionalImpact = Math.log10(Math.max(1_000, input.notional)) * 0.01;
|
||||||
|
pressure += input.placement === "AA" ? sizeImpact + notionalImpact : (sizeImpact + notionalImpact) * 0.65;
|
||||||
|
} else if (input.placement === "MID") {
|
||||||
|
pressure += 0.001;
|
||||||
|
} else {
|
||||||
|
pressure -= input.placement === "BB" ? 0.018 : 0.01;
|
||||||
|
}
|
||||||
|
|
||||||
|
pressure = clampValue(pressure, -0.25, 1.85);
|
||||||
|
const baseline = initializeSyntheticIv(input.dteDays, input.moneyness);
|
||||||
|
const iv = clampValue(baseline + pressure * 0.42, IV_MIN, IV_MAX);
|
||||||
|
return { iv: Number(iv.toFixed(4)), pressure, lastTs: input.ts };
|
||||||
|
};
|
||||||
|
|
||||||
const buildBurst = (burstIndex: number, now: number, profile: SyntheticOptionsProfile): Burst => {
|
const buildBurst = (burstIndex: number, now: number, profile: SyntheticOptionsProfile): Burst => {
|
||||||
const symbol = SYNTHETIC_SYMBOLS[burstIndex % SYNTHETIC_SYMBOLS.length];
|
const symbol = SYNTHETIC_SYMBOLS[burstIndex % SYNTHETIC_SYMBOLS.length];
|
||||||
const symbolHash = hashSymbol(symbol);
|
const symbolHash = hashSymbol(symbol);
|
||||||
|
|
@ -392,6 +453,9 @@ const buildBurst = (burstIndex: number, now: number, profile: SyntheticOptionsPr
|
||||||
|
|
||||||
return {
|
return {
|
||||||
contractId,
|
contractId,
|
||||||
|
underlying: baseUnderlying,
|
||||||
|
expiryOffsetDays: expiryOffset,
|
||||||
|
strike,
|
||||||
basePrice: basePricePer,
|
basePrice: basePricePer,
|
||||||
baseSize,
|
baseSize,
|
||||||
exchange,
|
exchange,
|
||||||
|
|
@ -420,6 +484,7 @@ export const createSyntheticOptionsAdapter = (
|
||||||
let nbboSeq = 0;
|
let nbboSeq = 0;
|
||||||
let burstIndex = 0;
|
let burstIndex = 0;
|
||||||
let currentBurst: Burst | null = null;
|
let currentBurst: Burst | null = null;
|
||||||
|
const ivByContract = new Map<string, SyntheticContractIvState>();
|
||||||
let remainingRuns = 0;
|
let remainingRuns = 0;
|
||||||
let timer: ReturnType<typeof setInterval> | null = null;
|
let timer: ReturnType<typeof setInterval> | null = null;
|
||||||
let stopped = false;
|
let stopped = false;
|
||||||
|
|
@ -448,12 +513,28 @@ export const createSyntheticOptionsAdapter = (
|
||||||
const priceJitter = ((i % 3) - 1) * 0.004;
|
const priceJitter = ((i % 3) - 1) * 0.004;
|
||||||
const sizeJitter = ((i % 3) - 1) * 0.08;
|
const sizeJitter = ((i % 3) - 1) * 0.08;
|
||||||
const priceMultiplier = 1 + burst.priceStep * i + priceJitter;
|
const priceMultiplier = 1 + burst.priceStep * i + priceJitter;
|
||||||
const mid = Math.max(0.05, Number((burst.basePrice * priceMultiplier).toFixed(2)));
|
const placement = pickPlacement(burst, i, profile);
|
||||||
const spread = Math.max(0.02, Number((mid * 0.02).toFixed(2)));
|
const size = Math.max(1, Math.round(burst.baseSize * (1 + sizeJitter)));
|
||||||
|
const previousIv = ivByContract.get(burst.contractId);
|
||||||
|
const provisionalNotional = burst.basePrice * size * OPTION_CONTRACT_MULTIPLIER;
|
||||||
|
const ivState = updateSyntheticIvForTest(previousIv, {
|
||||||
|
ts: now + i * 5,
|
||||||
|
placement,
|
||||||
|
size,
|
||||||
|
notional: provisionalNotional,
|
||||||
|
dteDays: burst.expiryOffsetDays,
|
||||||
|
moneyness: burst.strike / burst.underlying
|
||||||
|
});
|
||||||
|
ivByContract.set(burst.contractId, ivState);
|
||||||
|
const ivDrift = Math.max(0, ivState.iv - initializeSyntheticIv(burst.expiryOffsetDays, burst.strike / burst.underlying));
|
||||||
|
const mid = Math.max(
|
||||||
|
0.05,
|
||||||
|
Number((burst.basePrice * priceMultiplier * (1 + ivDrift * 1.15)).toFixed(2))
|
||||||
|
);
|
||||||
|
const spread = Math.max(0.02, Number((mid * (0.02 + Math.min(0.035, ivState.iv * 0.01))).toFixed(2)));
|
||||||
const bid = Math.max(0.01, Number((mid - spread / 2).toFixed(2)));
|
const bid = Math.max(0.01, Number((mid - spread / 2).toFixed(2)));
|
||||||
const ask = Math.max(bid + 0.01, Number((mid + spread / 2).toFixed(2)));
|
const ask = Math.max(bid + 0.01, Number((mid + spread / 2).toFixed(2)));
|
||||||
const tick = Math.max(0.01, Number((spread * 0.25).toFixed(2)));
|
const tick = Math.max(0.01, Number((spread * 0.25).toFixed(2)));
|
||||||
const placement = pickPlacement(burst, i, profile);
|
|
||||||
let tradePrice = mid;
|
let tradePrice = mid;
|
||||||
|
|
||||||
if (placement === "AA") {
|
if (placement === "AA") {
|
||||||
|
|
@ -476,9 +557,11 @@ export const createSyntheticOptionsAdapter = (
|
||||||
ts: now + i * 5,
|
ts: now + i * 5,
|
||||||
option_contract_id: burst.contractId,
|
option_contract_id: burst.contractId,
|
||||||
price: tradePrice,
|
price: tradePrice,
|
||||||
size: Math.max(1, Math.round(burst.baseSize * (1 + sizeJitter))),
|
size,
|
||||||
exchange: burst.exchange,
|
exchange: burst.exchange,
|
||||||
conditions: burst.conditions
|
conditions: burst.conditions,
|
||||||
|
execution_iv: ivState.iv,
|
||||||
|
execution_iv_source: "synthetic_pressure_model"
|
||||||
};
|
};
|
||||||
|
|
||||||
if (handlers.onNBBO) {
|
if (handlers.onNBBO) {
|
||||||
|
|
|
||||||
125
services/ingest-options/src/enrichment.ts
Normal file
125
services/ingest-options/src/enrichment.ts
Normal file
|
|
@ -0,0 +1,125 @@
|
||||||
|
import {
|
||||||
|
OptionPrintSchema,
|
||||||
|
classifyOptionNbboSide,
|
||||||
|
deriveOptionPrintMetadata,
|
||||||
|
evaluateOptionSignal,
|
||||||
|
type EquityQuote,
|
||||||
|
type OptionNBBO,
|
||||||
|
type OptionPrint,
|
||||||
|
type OptionsSignalConfig
|
||||||
|
} from "@islandflow/types";
|
||||||
|
|
||||||
|
export const MAX_CONTEXT_HISTORY = 64;
|
||||||
|
|
||||||
|
export type ContextHistory<T extends { ts: number; seq: number }> = Map<string, T[]>;
|
||||||
|
|
||||||
|
export const rememberContext = <T extends { ts: number; seq: number }>(
|
||||||
|
history: ContextHistory<T>,
|
||||||
|
key: string,
|
||||||
|
value: T
|
||||||
|
): void => {
|
||||||
|
const bucket = history.get(key) ?? [];
|
||||||
|
const existingIndex = bucket.findIndex((item) => item.ts === value.ts && item.seq === value.seq);
|
||||||
|
if (existingIndex >= 0) {
|
||||||
|
bucket[existingIndex] = value;
|
||||||
|
} else {
|
||||||
|
bucket.push(value);
|
||||||
|
}
|
||||||
|
bucket.sort((a, b) => {
|
||||||
|
const delta = a.ts - b.ts;
|
||||||
|
return delta !== 0 ? delta : a.seq - b.seq;
|
||||||
|
});
|
||||||
|
if (bucket.length > MAX_CONTEXT_HISTORY) {
|
||||||
|
bucket.splice(0, bucket.length - MAX_CONTEXT_HISTORY);
|
||||||
|
}
|
||||||
|
history.set(key, bucket);
|
||||||
|
};
|
||||||
|
|
||||||
|
export const selectAtOrBefore = <T extends { ts: number; seq: number }>(
|
||||||
|
items: readonly T[] | undefined,
|
||||||
|
ts: number
|
||||||
|
): T | null => {
|
||||||
|
if (!items?.length) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
let selected: T | null = null;
|
||||||
|
for (const item of items) {
|
||||||
|
if (item.ts > ts) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (!selected || item.ts > selected.ts || (item.ts === selected.ts && item.seq >= selected.seq)) {
|
||||||
|
selected = item;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return selected;
|
||||||
|
};
|
||||||
|
|
||||||
|
export const enrichOptionPrint = (
|
||||||
|
rawPrint: OptionPrint,
|
||||||
|
optionQuote: OptionNBBO | null | undefined,
|
||||||
|
equityQuote: EquityQuote | null | undefined,
|
||||||
|
config: OptionsSignalConfig
|
||||||
|
): OptionPrint => {
|
||||||
|
const derived = deriveOptionPrintMetadata(rawPrint, optionQuote, config);
|
||||||
|
const executionNbboSide = optionQuote
|
||||||
|
? classifyOptionNbboSide(rawPrint.price, optionQuote, rawPrint.ts, config.nbboMaxAgeMs)
|
||||||
|
: undefined;
|
||||||
|
const nbboMid =
|
||||||
|
optionQuote && Number.isFinite(optionQuote.bid) && Number.isFinite(optionQuote.ask)
|
||||||
|
? Number(((optionQuote.bid + optionQuote.ask) / 2).toFixed(4))
|
||||||
|
: undefined;
|
||||||
|
const nbboSpread =
|
||||||
|
optionQuote && Number.isFinite(optionQuote.bid) && Number.isFinite(optionQuote.ask)
|
||||||
|
? Number(Math.max(0, optionQuote.ask - optionQuote.bid).toFixed(4))
|
||||||
|
: undefined;
|
||||||
|
const underlyingMid =
|
||||||
|
equityQuote && Number.isFinite(equityQuote.bid) && Number.isFinite(equityQuote.ask)
|
||||||
|
? Number(((equityQuote.bid + equityQuote.ask) / 2).toFixed(4))
|
||||||
|
: undefined;
|
||||||
|
const underlyingSpread =
|
||||||
|
equityQuote && Number.isFinite(equityQuote.bid) && Number.isFinite(equityQuote.ask)
|
||||||
|
? Number(Math.max(0, equityQuote.ask - equityQuote.bid).toFixed(4))
|
||||||
|
: undefined;
|
||||||
|
|
||||||
|
const enrichedForSignal: OptionPrint = {
|
||||||
|
...rawPrint,
|
||||||
|
...derived,
|
||||||
|
nbbo_side: executionNbboSide ?? derived.nbbo_side,
|
||||||
|
...(optionQuote
|
||||||
|
? {
|
||||||
|
execution_nbbo_bid: optionQuote.bid,
|
||||||
|
execution_nbbo_ask: optionQuote.ask,
|
||||||
|
execution_nbbo_mid: nbboMid,
|
||||||
|
execution_nbbo_spread: nbboSpread,
|
||||||
|
execution_nbbo_bid_size: optionQuote.bidSize,
|
||||||
|
execution_nbbo_ask_size: optionQuote.askSize,
|
||||||
|
execution_nbbo_ts: optionQuote.ts,
|
||||||
|
execution_nbbo_age_ms: rawPrint.ts - optionQuote.ts,
|
||||||
|
execution_nbbo_side: executionNbboSide,
|
||||||
|
nbbo_side: executionNbboSide
|
||||||
|
}
|
||||||
|
: {}),
|
||||||
|
...(equityQuote && underlyingMid !== undefined
|
||||||
|
? {
|
||||||
|
execution_underlying_spot: underlyingMid,
|
||||||
|
execution_underlying_bid: equityQuote.bid,
|
||||||
|
execution_underlying_ask: equityQuote.ask,
|
||||||
|
execution_underlying_mid: underlyingMid,
|
||||||
|
execution_underlying_spread: underlyingSpread,
|
||||||
|
execution_underlying_ts: equityQuote.ts,
|
||||||
|
execution_underlying_age_ms: rawPrint.ts - equityQuote.ts,
|
||||||
|
execution_underlying_source: "equity_quote_mid" as const
|
||||||
|
}
|
||||||
|
: {}),
|
||||||
|
signal_profile: config.mode
|
||||||
|
};
|
||||||
|
|
||||||
|
const signalDecision = evaluateOptionSignal(enrichedForSignal, config);
|
||||||
|
return OptionPrintSchema.parse({
|
||||||
|
...enrichedForSignal,
|
||||||
|
signal_pass: signalDecision.signalPass,
|
||||||
|
signal_reasons: signalDecision.signalReasons,
|
||||||
|
signal_profile: signalDecision.signalProfile
|
||||||
|
});
|
||||||
|
};
|
||||||
|
|
@ -4,12 +4,16 @@ import {
|
||||||
SUBJECT_OPTION_NBBO,
|
SUBJECT_OPTION_NBBO,
|
||||||
SUBJECT_OPTION_PRINTS,
|
SUBJECT_OPTION_PRINTS,
|
||||||
SUBJECT_OPTION_SIGNAL_PRINTS,
|
SUBJECT_OPTION_SIGNAL_PRINTS,
|
||||||
|
SUBJECT_EQUITY_QUOTES,
|
||||||
|
STREAM_EQUITY_QUOTES,
|
||||||
STREAM_OPTION_NBBO,
|
STREAM_OPTION_NBBO,
|
||||||
STREAM_OPTION_PRINTS,
|
STREAM_OPTION_PRINTS,
|
||||||
STREAM_OPTION_SIGNAL_PRINTS,
|
STREAM_OPTION_SIGNAL_PRINTS,
|
||||||
|
buildDurableConsumer,
|
||||||
connectJetStreamWithRetry,
|
connectJetStreamWithRetry,
|
||||||
ensureStream,
|
ensureStream,
|
||||||
publishJson
|
publishJson,
|
||||||
|
subscribeJson
|
||||||
} from "@islandflow/bus";
|
} from "@islandflow/bus";
|
||||||
import {
|
import {
|
||||||
createClickHouseClient,
|
createClickHouseClient,
|
||||||
|
|
@ -21,9 +25,10 @@ import {
|
||||||
import {
|
import {
|
||||||
OptionNBBOSchema,
|
OptionNBBOSchema,
|
||||||
OptionPrintSchema,
|
OptionPrintSchema,
|
||||||
evaluateOptionSignal,
|
EquityQuoteSchema,
|
||||||
deriveOptionPrintMetadata,
|
deriveOptionPrintMetadata,
|
||||||
resolveSyntheticMarketModes,
|
resolveSyntheticMarketModes,
|
||||||
|
type EquityQuote,
|
||||||
type OptionNBBO,
|
type OptionNBBO,
|
||||||
type OptionPrint,
|
type OptionPrint,
|
||||||
type OptionsSignalConfig
|
type OptionsSignalConfig
|
||||||
|
|
@ -33,6 +38,7 @@ import { createDatabentoOptionsAdapter } from "./adapters/databento";
|
||||||
import { createIbkrOptionsAdapter } from "./adapters/ibkr";
|
import { createIbkrOptionsAdapter } from "./adapters/ibkr";
|
||||||
import { createSyntheticOptionsAdapter } from "./adapters/synthetic";
|
import { createSyntheticOptionsAdapter } from "./adapters/synthetic";
|
||||||
import type { OptionIngestAdapter, StopHandler } from "./adapters/types";
|
import type { OptionIngestAdapter, StopHandler } from "./adapters/types";
|
||||||
|
import { enrichOptionPrint, rememberContext, selectAtOrBefore, type ContextHistory } from "./enrichment";
|
||||||
import { z } from "zod";
|
import { z } from "zod";
|
||||||
|
|
||||||
const service = "ingest-options";
|
const service = "ingest-options";
|
||||||
|
|
@ -135,7 +141,9 @@ const state = {
|
||||||
shuttingDown: false,
|
shuttingDown: false,
|
||||||
shutdownPromise: null as Promise<void> | null
|
shutdownPromise: null as Promise<void> | null
|
||||||
};
|
};
|
||||||
const latestNbboByContract = new Map<string, OptionNBBO>();
|
|
||||||
|
const nbboHistoryByContract: ContextHistory<OptionNBBO> = new Map();
|
||||||
|
const equityQuoteHistoryByUnderlying: ContextHistory<EquityQuote> = new Map();
|
||||||
|
|
||||||
const getErrorMessage = (error: unknown): string => {
|
const getErrorMessage = (error: unknown): string => {
|
||||||
return error instanceof Error ? error.message : String(error);
|
return error instanceof Error ? error.message : String(error);
|
||||||
|
|
@ -338,6 +346,19 @@ const run = async () => {
|
||||||
num_replicas: 1
|
num_replicas: 1
|
||||||
});
|
});
|
||||||
|
|
||||||
|
await ensureStream(jsm, {
|
||||||
|
name: STREAM_EQUITY_QUOTES,
|
||||||
|
subjects: [SUBJECT_EQUITY_QUOTES],
|
||||||
|
retention: "limits",
|
||||||
|
storage: "file",
|
||||||
|
discard: "old",
|
||||||
|
max_msgs_per_subject: -1,
|
||||||
|
max_msgs: -1,
|
||||||
|
max_bytes: -1,
|
||||||
|
max_age: 0,
|
||||||
|
num_replicas: 1
|
||||||
|
});
|
||||||
|
|
||||||
const clickhouse = createClickHouseClient({
|
const clickhouse = createClickHouseClient({
|
||||||
url: env.CLICKHOUSE_URL,
|
url: env.CLICKHOUSE_URL,
|
||||||
database: env.CLICKHOUSE_DATABASE
|
database: env.CLICKHOUSE_DATABASE
|
||||||
|
|
@ -365,26 +386,15 @@ const run = async () => {
|
||||||
}
|
}
|
||||||
|
|
||||||
const rawPrint = OptionPrintSchema.parse(candidate);
|
const rawPrint = OptionPrintSchema.parse(candidate);
|
||||||
const derived = deriveOptionPrintMetadata(
|
const parsedMetadata = deriveOptionPrintMetadata(rawPrint, null, optionsSignalConfig);
|
||||||
rawPrint,
|
const optionQuote = selectAtOrBefore(
|
||||||
latestNbboByContract.get(rawPrint.option_contract_id),
|
nbboHistoryByContract.get(rawPrint.option_contract_id),
|
||||||
optionsSignalConfig
|
rawPrint.ts
|
||||||
);
|
);
|
||||||
const signalDecision = evaluateOptionSignal(
|
const equityQuote = parsedMetadata.underlying_id
|
||||||
{
|
? selectAtOrBefore(equityQuoteHistoryByUnderlying.get(parsedMetadata.underlying_id), rawPrint.ts)
|
||||||
...rawPrint,
|
: null;
|
||||||
...derived,
|
const print = enrichOptionPrint(rawPrint, optionQuote, equityQuote, optionsSignalConfig);
|
||||||
signal_profile: optionsSignalConfig.mode
|
|
||||||
},
|
|
||||||
optionsSignalConfig
|
|
||||||
);
|
|
||||||
const print = OptionPrintSchema.parse({
|
|
||||||
...rawPrint,
|
|
||||||
...derived,
|
|
||||||
signal_pass: signalDecision.signalPass,
|
|
||||||
signal_reasons: signalDecision.signalReasons,
|
|
||||||
signal_profile: signalDecision.signalProfile
|
|
||||||
});
|
|
||||||
|
|
||||||
try {
|
try {
|
||||||
await insertOptionPrint(clickhouse, print);
|
await insertOptionPrint(clickhouse, print);
|
||||||
|
|
@ -422,14 +432,7 @@ const run = async () => {
|
||||||
}
|
}
|
||||||
|
|
||||||
const nbbo = OptionNBBOSchema.parse(candidate);
|
const nbbo = OptionNBBOSchema.parse(candidate);
|
||||||
const existing = latestNbboByContract.get(nbbo.option_contract_id);
|
rememberContext(nbboHistoryByContract, nbbo.option_contract_id, nbbo);
|
||||||
if (
|
|
||||||
!existing ||
|
|
||||||
nbbo.ts > existing.ts ||
|
|
||||||
(nbbo.ts === existing.ts && nbbo.seq >= existing.seq)
|
|
||||||
) {
|
|
||||||
latestNbboByContract.set(nbbo.option_contract_id, nbbo);
|
|
||||||
}
|
|
||||||
|
|
||||||
try {
|
try {
|
||||||
await insertOptionNBBO(clickhouse, nbbo);
|
await insertOptionNBBO(clickhouse, nbbo);
|
||||||
|
|
@ -447,6 +450,33 @@ const run = async () => {
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
const equityQuoteConsumer = buildDurableConsumer("ingest-options-equity-quotes");
|
||||||
|
equityQuoteConsumer.deliverAll();
|
||||||
|
const equityQuoteSubscription = await subscribeJson<EquityQuote>(
|
||||||
|
js,
|
||||||
|
SUBJECT_EQUITY_QUOTES,
|
||||||
|
equityQuoteConsumer
|
||||||
|
);
|
||||||
|
|
||||||
|
void (async () => {
|
||||||
|
for await (const msg of equityQuoteSubscription.messages) {
|
||||||
|
if (state.shuttingDown) {
|
||||||
|
msg.ack();
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
const quote = EquityQuoteSchema.parse(equityQuoteSubscription.decode(msg));
|
||||||
|
rememberContext(equityQuoteHistoryByUnderlying, quote.underlying_id.toUpperCase(), quote);
|
||||||
|
msg.ack();
|
||||||
|
} catch (error) {
|
||||||
|
logger.error("failed to process equity quote context", {
|
||||||
|
error: getErrorMessage(error)
|
||||||
|
});
|
||||||
|
msg.ack();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})();
|
||||||
|
|
||||||
const shutdown = async (signal: string) => {
|
const shutdown = async (signal: string) => {
|
||||||
if (state.shutdownPromise) {
|
if (state.shutdownPromise) {
|
||||||
return state.shutdownPromise;
|
return state.shutdownPromise;
|
||||||
|
|
|
||||||
88
services/ingest-options/tests/enrichment.test.ts
Normal file
88
services/ingest-options/tests/enrichment.test.ts
Normal file
|
|
@ -0,0 +1,88 @@
|
||||||
|
import { describe, expect, it } from "bun:test";
|
||||||
|
import type { EquityQuote, OptionNBBO, OptionPrint, OptionsSignalConfig } from "@islandflow/types";
|
||||||
|
import { enrichOptionPrint, selectAtOrBefore } from "../src/enrichment";
|
||||||
|
|
||||||
|
const config: OptionsSignalConfig = {
|
||||||
|
mode: "all",
|
||||||
|
minNotional: 0,
|
||||||
|
etfMinNotional: 0,
|
||||||
|
bidSideMinNotional: 0,
|
||||||
|
midMinNotional: 0,
|
||||||
|
missingNbboMinNotional: 0,
|
||||||
|
largePrintMinSize: 1,
|
||||||
|
largePrintMinNotional: 0,
|
||||||
|
sweepMinNotional: 0,
|
||||||
|
autoKeepMinNotional: 100_000,
|
||||||
|
nbboMaxAgeMs: 1_500,
|
||||||
|
etfUnderlyings: new Set(["SPY"])
|
||||||
|
};
|
||||||
|
|
||||||
|
const print: OptionPrint = {
|
||||||
|
source_ts: 1_000,
|
||||||
|
ingest_ts: 1_000,
|
||||||
|
seq: 1,
|
||||||
|
trace_id: "print-1",
|
||||||
|
ts: 1_000,
|
||||||
|
option_contract_id: "SPY-2025-01-17-450-C",
|
||||||
|
price: 1.3,
|
||||||
|
size: 10,
|
||||||
|
exchange: "TEST"
|
||||||
|
};
|
||||||
|
|
||||||
|
const nbbo = (overrides: Partial<OptionNBBO> = {}): OptionNBBO => ({
|
||||||
|
source_ts: 990,
|
||||||
|
ingest_ts: 990,
|
||||||
|
seq: 1,
|
||||||
|
trace_id: "nbbo-1",
|
||||||
|
ts: 990,
|
||||||
|
option_contract_id: "SPY-2025-01-17-450-C",
|
||||||
|
bid: 1.2,
|
||||||
|
ask: 1.3,
|
||||||
|
bidSize: 20,
|
||||||
|
askSize: 30,
|
||||||
|
...overrides
|
||||||
|
});
|
||||||
|
|
||||||
|
const equityQuote = (overrides: Partial<EquityQuote> = {}): EquityQuote => ({
|
||||||
|
source_ts: 980,
|
||||||
|
ingest_ts: 980,
|
||||||
|
seq: 1,
|
||||||
|
trace_id: "eq-1",
|
||||||
|
ts: 980,
|
||||||
|
underlying_id: "SPY",
|
||||||
|
bid: 450,
|
||||||
|
ask: 450.1,
|
||||||
|
...overrides
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("option print enrichment", () => {
|
||||||
|
it("attaches preserved NBBO context and mirrors nbbo_side", () => {
|
||||||
|
const enriched = enrichOptionPrint(print, nbbo(), null, config);
|
||||||
|
|
||||||
|
expect(enriched.execution_nbbo_bid).toBe(1.2);
|
||||||
|
expect(enriched.execution_nbbo_ask).toBe(1.3);
|
||||||
|
expect(enriched.execution_nbbo_mid).toBe(1.25);
|
||||||
|
expect(enriched.execution_nbbo_age_ms).toBe(10);
|
||||||
|
expect(enriched.execution_nbbo_side).toBe("A");
|
||||||
|
expect(enriched.nbbo_side).toBe(enriched.execution_nbbo_side);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("attaches preserved underlying quote mid as spot", () => {
|
||||||
|
const enriched = enrichOptionPrint(print, null, equityQuote(), config);
|
||||||
|
|
||||||
|
expect(enriched.execution_underlying_spot).toBe(450.05);
|
||||||
|
expect(enriched.execution_underlying_mid).toBe(450.05);
|
||||||
|
expect(enriched.execution_underlying_source).toBe("equity_quote_mid");
|
||||||
|
expect(enriched.execution_underlying_age_ms).toBe(20);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("selects context at or before the print timestamp only", () => {
|
||||||
|
const selected = selectAtOrBefore(
|
||||||
|
[nbbo({ ts: 900, seq: 1, bid: 1 }), nbbo({ ts: 1_001, seq: 2, bid: 2 })],
|
||||||
|
print.ts
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(selected?.ts).toBe(900);
|
||||||
|
expect(selected?.bid).toBe(1);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
@ -1,5 +1,5 @@
|
||||||
import { describe, expect, it } from "bun:test";
|
import { describe, expect, it } from "bun:test";
|
||||||
import { buildSyntheticBurstForTest } from "../src/adapters/synthetic";
|
import { buildSyntheticBurstForTest, updateSyntheticIvForTest } from "../src/adapters/synthetic";
|
||||||
|
|
||||||
const totalBurstNotional = (burst: {
|
const totalBurstNotional = (burst: {
|
||||||
basePrice: number;
|
basePrice: number;
|
||||||
|
|
@ -24,3 +24,66 @@ describe("synthetic options burst sizing", () => {
|
||||||
expect(totalBurstNotional(burst)).toBeLessThanOrEqual(240_000);
|
expect(totalBurstNotional(burst)).toBeLessThanOrEqual(240_000);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
describe("synthetic options IV model", () => {
|
||||||
|
it("increases under repeated same-contract ask buying", () => {
|
||||||
|
let state = updateSyntheticIvForTest(undefined, {
|
||||||
|
ts: 1_000,
|
||||||
|
placement: "A",
|
||||||
|
size: 100,
|
||||||
|
notional: 20_000,
|
||||||
|
dteDays: 1,
|
||||||
|
moneyness: 1.02
|
||||||
|
});
|
||||||
|
const firstIv = state.iv;
|
||||||
|
|
||||||
|
state = updateSyntheticIvForTest(state, {
|
||||||
|
ts: 1_100,
|
||||||
|
placement: "AA",
|
||||||
|
size: 300,
|
||||||
|
notional: 80_000,
|
||||||
|
dteDays: 1,
|
||||||
|
moneyness: 1.02
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(state.iv).toBeGreaterThan(firstIv);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("decays after inactivity", () => {
|
||||||
|
const active = updateSyntheticIvForTest(undefined, {
|
||||||
|
ts: 1_000,
|
||||||
|
placement: "AA",
|
||||||
|
size: 500,
|
||||||
|
notional: 120_000,
|
||||||
|
dteDays: 7,
|
||||||
|
moneyness: 1.1
|
||||||
|
});
|
||||||
|
const decayed = updateSyntheticIvForTest(active, {
|
||||||
|
ts: 181_000,
|
||||||
|
placement: "MID",
|
||||||
|
size: 10,
|
||||||
|
notional: 1_000,
|
||||||
|
dteDays: 7,
|
||||||
|
moneyness: 1.1
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(decayed.iv).toBeLessThan(active.iv);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("keeps IV within clamps", () => {
|
||||||
|
let state = undefined;
|
||||||
|
for (let i = 0; i < 80; i += 1) {
|
||||||
|
state = updateSyntheticIvForTest(state, {
|
||||||
|
ts: 1_000 + i * 10,
|
||||||
|
placement: "AA",
|
||||||
|
size: 10_000,
|
||||||
|
notional: 5_000_000,
|
||||||
|
dteDays: 0,
|
||||||
|
moneyness: 1.8
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
expect(state.iv).toBeGreaterThanOrEqual(0.05);
|
||||||
|
expect(state.iv).toBeLessThanOrEqual(2.5);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
|
||||||
170
tape-overhaul-phase1-1.md
Normal file
170
tape-overhaul-phase1-1.md
Normal file
|
|
@ -0,0 +1,170 @@
|
||||||
|
# Server-Backed Persistent History
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Make live mode server-authoritative across refreshes, sessions, and devices. The browser will not own data persistence. On load, the app will hydrate from ClickHouse-backed server history, then layer live WebSocket updates on top. Users will immediately see a substantial recent persisted window, with older records available through history pagination.
|
||||||
|
|
||||||
|
## Chosen Defaults
|
||||||
|
|
||||||
|
- Source of truth: ClickHouse on the server.
|
||||||
|
- Browser persistence: UI preferences only, no market-data cache.
|
||||||
|
- Initial load: recent persisted window per active channel.
|
||||||
|
- Older data: fetched on demand using cursor pagination.
|
||||||
|
- Scope: every channel the server handles, including options, NBBO, equities, equity quotes, equity joins, flow packets, classifier hits, alerts, inferred dark events, candles, and chart overlays.
|
||||||
|
- Freshness: freshness affects status labels only; it must not hide persisted history from a refreshed browser.
|
||||||
|
|
||||||
|
## Current State To Change
|
||||||
|
|
||||||
|
- `LiveStateManager` hydrates from Redis or ClickHouse, but freshness gates currently suppress stale options, NBBO, equities, and flow snapshots.
|
||||||
|
- The unified `/ws/live` protocol supports snapshots and `next_before`, but the frontend does not retain/use per-channel history cursors for live-mode pagination.
|
||||||
|
- Some channels have REST history endpoints, but `equity-quotes` is not fully represented in the unified live protocol/history API.
|
||||||
|
- Charts already query ClickHouse for candle and overlay ranges, but should be treated as part of the same server-history model.
|
||||||
|
|
||||||
|
## Public Interfaces And Types
|
||||||
|
|
||||||
|
Update `packages/types/src/live.ts`:
|
||||||
|
|
||||||
|
- Add `"equity-quotes"` to:
|
||||||
|
- `LiveGenericChannelSchema`
|
||||||
|
- `LiveChannelSchema`
|
||||||
|
- `LiveSubscriptionSchema`
|
||||||
|
- `livePayloadSchemas`
|
||||||
|
- Preserve existing `FeedSnapshot` shape:
|
||||||
|
- `items`
|
||||||
|
- `watermark`
|
||||||
|
- `next_before`
|
||||||
|
|
||||||
|
Update API routes in `services/api/src/index.ts`:
|
||||||
|
|
||||||
|
- Add `GET /history/equity-quotes?before_ts=&before_seq=&limit=`.
|
||||||
|
- Include `equity-quotes` in `/ws/live` subscriptions and fanout.
|
||||||
|
- Keep existing recent/replay endpoints compatible.
|
||||||
|
|
||||||
|
Update storage in `packages/storage/src/clickhouse.ts`:
|
||||||
|
|
||||||
|
- Add `fetchEquityQuotesBefore`.
|
||||||
|
- Reuse existing `(ts, seq)` cursor ordering.
|
||||||
|
- Keep limits clamped consistently with other history endpoints.
|
||||||
|
|
||||||
|
## Server Implementation
|
||||||
|
|
||||||
|
In `services/api/src/live.ts`:
|
||||||
|
|
||||||
|
1. Add generic config for `equity-quotes`:
|
||||||
|
- Redis key: `live:equity-quotes`
|
||||||
|
- cursor field: `equity-quotes`
|
||||||
|
- parser: `EquityQuoteSchema`
|
||||||
|
- cursor: `{ ts, seq }`
|
||||||
|
- fetchRecent: `fetchRecentEquityQuotes`
|
||||||
|
2. Stop filtering historical snapshots by freshness:
|
||||||
|
- Remove `filterFreshGenericItems` from snapshot construction.
|
||||||
|
- Keep `isLiveItemFresh` available for UI status/fanout behavior if needed.
|
||||||
|
- Do not reject persisted ClickHouse rows just because market timestamps are older than 15s/30s.
|
||||||
|
3. Stop rejecting stale ingests inside `LiveStateManager.ingest`.
|
||||||
|
- The manager should store valid events it receives.
|
||||||
|
- Event fanout can still choose how to label status, but should not silently lose durable cache state.
|
||||||
|
4. Preserve Redis as a hot cache:
|
||||||
|
- Redis remains an optimization.
|
||||||
|
- ClickHouse remains the fallback and source of truth.
|
||||||
|
- API startup should hydrate from Redis if present, otherwise from ClickHouse.
|
||||||
|
|
||||||
|
In `services/api/src/index.ts`:
|
||||||
|
|
||||||
|
1. Include `equity-quotes` in `consumerBindings`.
|
||||||
|
2. Pump `EquityQuoteSchema` payloads into:
|
||||||
|
- legacy `/ws/equity-quotes`
|
||||||
|
- unified `/ws/live`
|
||||||
|
- `LiveStateManager`
|
||||||
|
3. Add `/history/equity-quotes`.
|
||||||
|
4. Keep durable consumer defaults unchanged unless a test proves old events are skipped in a live-running API scenario. ClickHouse hydration handles restart and refresh persistence.
|
||||||
|
|
||||||
|
## Frontend Implementation
|
||||||
|
|
||||||
|
In `apps/web/app/terminal.tsx`:
|
||||||
|
|
||||||
|
1. Extend `LiveSessionState` with:
|
||||||
|
- per-subscription `next_before` cursors
|
||||||
|
- per-subscription loading/error state for older history
|
||||||
|
- equity quotes if exposed in UI state
|
||||||
|
2. When handling `snapshot` messages:
|
||||||
|
- Replace the channel's current items with snapshot items when non-empty.
|
||||||
|
- Store `snapshot.next_before`.
|
||||||
|
- Do not discard stale-but-persisted rows.
|
||||||
|
- Continue deduping by `trace_id/seq` or `id`.
|
||||||
|
3. Add a generic live-history loader:
|
||||||
|
- Map subscription channel to history endpoint:
|
||||||
|
- `options` -> `/history/options`
|
||||||
|
- `nbbo` -> `/history/nbbo`
|
||||||
|
- `equities` -> `/history/equities`
|
||||||
|
- `equity-quotes` -> `/history/equity-quotes`
|
||||||
|
- `equity-joins` -> `/history/equity-joins`
|
||||||
|
- `flow` -> `/history/flow`
|
||||||
|
- `classifier-hits` -> `/history/classifier-hits`
|
||||||
|
- `alerts` -> `/history/alerts`
|
||||||
|
- `inferred-dark` -> `/history/inferred-dark`
|
||||||
|
- Carry option/flow filters into options history queries.
|
||||||
|
- Merge older results into existing channel state.
|
||||||
|
- Advance `next_before` from the response.
|
||||||
|
- Stop when `next_before` is null or the response is empty.
|
||||||
|
4. UI behavior:
|
||||||
|
- Add a compact "Load older" control at the bottom of each applicable tape/list.
|
||||||
|
- Disable it while loading.
|
||||||
|
- Hide it when no more history exists.
|
||||||
|
- Keep existing pause/jump controls unchanged.
|
||||||
|
- Do not add browser market-data storage.
|
||||||
|
5. Chart behavior:
|
||||||
|
- Keep candles loading from `/candles/equities`.
|
||||||
|
- Keep overlay loading from `/prints/equities/range`.
|
||||||
|
- Ensure refresh and device changes show the same server data for the same ticker/window.
|
||||||
|
|
||||||
|
## Config And Deployment
|
||||||
|
|
||||||
|
Update `.env.example`:
|
||||||
|
|
||||||
|
- Add `LIVE_LIMIT_EQUITY_QUOTES=10000`.
|
||||||
|
- Document that `LIVE_LIMIT_*` controls initial server snapshot/hot-cache depth, not total persisted history.
|
||||||
|
|
||||||
|
Update README if needed:
|
||||||
|
|
||||||
|
- Clarify persistence model:
|
||||||
|
- ClickHouse is durable history.
|
||||||
|
- Redis is hot cache.
|
||||||
|
- Browser is not a market-data database.
|
||||||
|
- All devices connected to the same API see the same server-seen data.
|
||||||
|
|
||||||
|
Docker volumes already persist ClickHouse/Redis/NATS data locally and in deployment compose, so no migration is needed for volume-backed persistence.
|
||||||
|
|
||||||
|
## Tests
|
||||||
|
|
||||||
|
API tests in `services/api/tests/live.test.ts`:
|
||||||
|
|
||||||
|
- Snapshot hydration returns stale historical options/NBBO/equities/flow instead of filtering them out.
|
||||||
|
- `LiveStateManager.ingest` stores older valid events.
|
||||||
|
- `equity-quotes` hydrates from Redis.
|
||||||
|
- `equity-quotes` hydrates from ClickHouse when Redis is empty.
|
||||||
|
- `next_before` is set from the oldest item in returned snapshot.
|
||||||
|
- Redis hot cache persists hydrated ClickHouse data.
|
||||||
|
|
||||||
|
Storage tests:
|
||||||
|
|
||||||
|
- Add `fetchEquityQuotesBefore` coverage using the existing storage test style.
|
||||||
|
|
||||||
|
Frontend tests in `apps/web/app/terminal.test.ts`:
|
||||||
|
|
||||||
|
- Live snapshot with older persisted rows populates visible rows.
|
||||||
|
- Empty snapshot does not wipe existing visible rows only when preserving an already visible channel during reconnect.
|
||||||
|
- Older-history merge dedupes existing items.
|
||||||
|
- History cursor advances after loading older rows.
|
||||||
|
- "No more history" state is reached when `next_before` is null.
|
||||||
|
- Live status can be stale while items remain visible.
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
- Refreshing the app shows persisted data immediately, even when no new live events arrive after page load.
|
||||||
|
- Opening the app on another device connected to the same API shows the same server-backed recent history.
|
||||||
|
- Stale market timestamps do not cause persisted history to disappear.
|
||||||
|
- Users can load older data beyond the initial recent window.
|
||||||
|
- Live WebSocket updates still appear without requiring refresh.
|
||||||
|
- Redis loss does not lose history; API falls back to ClickHouse.
|
||||||
|
- Browser cache deletion does not lose market data.
|
||||||
|
- `bun test services/api/tests/live.test.ts apps/web/app/terminal.test.ts packages/storage/tests/*.test.ts` passes, or any unavailable test target is documented.
|
||||||
320
tape-overhaul-phase1.md
Normal file
320
tape-overhaul-phase1.md
Normal file
|
|
@ -0,0 +1,320 @@
|
||||||
|
# Options Overhaul Phase 1: Snapshot Tape Table
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Convert the Options tape into a dense table where every row is an individual option print with preserved execution context. The print itself becomes the authoritative record for what was known around that trade at the moment it printed: option NBBO, underlying spot, IV, notional, side/classification metadata, and classifier-derived row coloring.
|
||||||
|
|
||||||
|
This phase includes backend enrichment, storage/type changes, synthetic IV behavior, and the frontend table redesign together.
|
||||||
|
|
||||||
|
## Core Principle
|
||||||
|
|
||||||
|
Do not treat NBBO, spot, or IV as live lookups in the table once the print has been recorded.
|
||||||
|
|
||||||
|
Each option print should carry a snapshot of its execution context. The UI should prefer those preserved fields and only fall back to current side maps for legacy rows that predate the migration.
|
||||||
|
|
||||||
|
## Public Type Changes
|
||||||
|
|
||||||
|
Extend `OptionPrintSchema` / `OptionPrint` in `packages/types/src/events.ts`.
|
||||||
|
|
||||||
|
Add optional flat fields:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
execution_nbbo_bid?: number;
|
||||||
|
execution_nbbo_ask?: number;
|
||||||
|
execution_nbbo_mid?: number;
|
||||||
|
execution_nbbo_spread?: number;
|
||||||
|
execution_nbbo_bid_size?: number;
|
||||||
|
execution_nbbo_ask_size?: number;
|
||||||
|
execution_nbbo_ts?: number;
|
||||||
|
execution_nbbo_age_ms?: number;
|
||||||
|
execution_nbbo_side?: OptionNbboSide;
|
||||||
|
|
||||||
|
execution_underlying_spot?: number;
|
||||||
|
execution_underlying_bid?: number;
|
||||||
|
execution_underlying_ask?: number;
|
||||||
|
execution_underlying_mid?: number;
|
||||||
|
execution_underlying_spread?: number;
|
||||||
|
execution_underlying_ts?: number;
|
||||||
|
execution_underlying_age_ms?: number;
|
||||||
|
execution_underlying_source?: "equity_quote_mid";
|
||||||
|
|
||||||
|
execution_iv?: number;
|
||||||
|
execution_iv_source?: "provider" | "synthetic_pressure_model";
|
||||||
|
```
|
||||||
|
|
||||||
|
Keep existing fields for compatibility:
|
||||||
|
|
||||||
|
- `nbbo_side`
|
||||||
|
- `notional`
|
||||||
|
- `underlying_id`
|
||||||
|
- `option_type`
|
||||||
|
- `signal_*`
|
||||||
|
|
||||||
|
Set `nbbo_side` to match `execution_nbbo_side` for new prints so existing filters continue working.
|
||||||
|
|
||||||
|
## Storage Changes
|
||||||
|
|
||||||
|
Update `packages/storage/src/option-prints.ts`.
|
||||||
|
|
||||||
|
Add ClickHouse columns:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
execution_nbbo_bid Nullable(Float64),
|
||||||
|
execution_nbbo_ask Nullable(Float64),
|
||||||
|
execution_nbbo_mid Nullable(Float64),
|
||||||
|
execution_nbbo_spread Nullable(Float64),
|
||||||
|
execution_nbbo_bid_size Nullable(UInt32),
|
||||||
|
execution_nbbo_ask_size Nullable(UInt32),
|
||||||
|
execution_nbbo_ts Nullable(UInt64),
|
||||||
|
execution_nbbo_age_ms Nullable(Float64),
|
||||||
|
execution_nbbo_side Nullable(String),
|
||||||
|
|
||||||
|
execution_underlying_spot Nullable(Float64),
|
||||||
|
execution_underlying_bid Nullable(Float64),
|
||||||
|
execution_underlying_ask Nullable(Float64),
|
||||||
|
execution_underlying_mid Nullable(Float64),
|
||||||
|
execution_underlying_spread Nullable(Float64),
|
||||||
|
execution_underlying_ts Nullable(UInt64),
|
||||||
|
execution_underlying_age_ms Nullable(Float64),
|
||||||
|
execution_underlying_source Nullable(String),
|
||||||
|
|
||||||
|
execution_iv Nullable(Float64),
|
||||||
|
execution_iv_source Nullable(String)
|
||||||
|
```
|
||||||
|
|
||||||
|
Add `ALTER TABLE ... ADD COLUMN IF NOT EXISTS` migrations for all fields.
|
||||||
|
|
||||||
|
Update row normalization so missing legacy values parse as `undefined`.
|
||||||
|
|
||||||
|
## Ingest Enrichment
|
||||||
|
|
||||||
|
Update `services/ingest-options/src/index.ts`.
|
||||||
|
|
||||||
|
Maintain caches:
|
||||||
|
|
||||||
|
- latest option NBBO by contract
|
||||||
|
- latest equity quote by underlying
|
||||||
|
- synthetic/adapter-provided IV by contract when available
|
||||||
|
|
||||||
|
When an option trade arrives:
|
||||||
|
|
||||||
|
1. Parse raw print.
|
||||||
|
2. Derive underlying, option type, notional, ETF flag as today.
|
||||||
|
3. Select latest option NBBO for the contract at or before `print.ts`.
|
||||||
|
4. Attach preserved NBBO fields:
|
||||||
|
- bid, ask, mid, spread
|
||||||
|
- bid/ask sizes
|
||||||
|
- quote timestamp
|
||||||
|
- quote age
|
||||||
|
- execution NBBO side
|
||||||
|
5. Select latest equity quote for the underlying at or before `print.ts`.
|
||||||
|
6. Attach preserved underlying fields:
|
||||||
|
- bid, ask, mid
|
||||||
|
- spread
|
||||||
|
- quote timestamp
|
||||||
|
- quote age
|
||||||
|
- `execution_underlying_spot = mid`
|
||||||
|
- `execution_underlying_source = "equity_quote_mid"`
|
||||||
|
7. Attach IV if available.
|
||||||
|
8. Evaluate signal filters using preserved execution fields.
|
||||||
|
9. Persist and publish the enriched print.
|
||||||
|
|
||||||
|
Important behavior:
|
||||||
|
|
||||||
|
- Do not mark these preserved fields stale in the UI.
|
||||||
|
- Age fields are still stored for auditability.
|
||||||
|
- If no at-or-before quote exists, leave that context unset.
|
||||||
|
- Never use a quote after the option print timestamp for preserved execution context.
|
||||||
|
|
||||||
|
## Synthetic IV Model
|
||||||
|
|
||||||
|
Update `services/ingest-options/src/adapters/synthetic.ts`.
|
||||||
|
|
||||||
|
Add persistent contract-level IV state:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
type SyntheticContractIvState = {
|
||||||
|
iv: number;
|
||||||
|
pressure: number;
|
||||||
|
lastTs: number;
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
Behavior:
|
||||||
|
|
||||||
|
- Initialize IV from a plausible baseline based on DTE and moneyness.
|
||||||
|
- Maintain IV per contract across bursts.
|
||||||
|
- Repeated aggressive buying of the same contract raises pressure and IV.
|
||||||
|
- Aggressive buying means synthetic placement `A` or `AA`.
|
||||||
|
- `MID` has small/no pressure.
|
||||||
|
- `B` or `BB` reduces pressure slightly.
|
||||||
|
- Pressure decays over time after inactivity.
|
||||||
|
- IV is clamped to a plausible range.
|
||||||
|
|
||||||
|
Recommended defaults:
|
||||||
|
|
||||||
|
- Baseline IV: `0.18` to `0.65`
|
||||||
|
- 0DTE contracts start higher than far-dated contracts.
|
||||||
|
- Out-of-the-money contracts start slightly higher than near-the-money contracts.
|
||||||
|
- Ask/above-ask print pressure increment: proportional to size and notional.
|
||||||
|
- Decay half-life: roughly 30-90 seconds in synthetic time.
|
||||||
|
- Clamp IV to `0.05..2.5`.
|
||||||
|
|
||||||
|
Each synthetic `OptionPrint` should include:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
execution_iv
|
||||||
|
execution_iv_source: "synthetic_pressure_model"
|
||||||
|
```
|
||||||
|
|
||||||
|
Synthetic NBBO and trade price generation should remain coherent:
|
||||||
|
|
||||||
|
- As IV rises, option mid/ask should drift higher for that contract.
|
||||||
|
- Rapid same-contract buying should visibly increase both print price and IV over subsequent prints.
|
||||||
|
- Bid/ask spread may widen mildly with higher IV.
|
||||||
|
|
||||||
|
## Real Adapter IV Behavior
|
||||||
|
|
||||||
|
For Alpaca, Databento, and IBKR in Phase 1:
|
||||||
|
|
||||||
|
- Preserve NBBO and underlying spot context through ingest enrichment.
|
||||||
|
- Leave `execution_iv` unset unless the adapter already provides a reliable IV value.
|
||||||
|
- Do not invent IV for real feeds in Phase 1.
|
||||||
|
|
||||||
|
Synthetic is the only source that must generate IV in this phase.
|
||||||
|
|
||||||
|
## Frontend Table Redesign
|
||||||
|
|
||||||
|
Update `apps/web/app/terminal.tsx` and `apps/web/app/globals.css`.
|
||||||
|
|
||||||
|
Each Options row remains an `OptionPrint`.
|
||||||
|
|
||||||
|
Default columns:
|
||||||
|
|
||||||
|
- `TIME`
|
||||||
|
- `SYM`
|
||||||
|
- `EXP`
|
||||||
|
- `STRIKE`
|
||||||
|
- `C/P`
|
||||||
|
- `SPOT`
|
||||||
|
- `DETAILS`
|
||||||
|
- `TYPE`
|
||||||
|
- `VALUE`
|
||||||
|
- `SIDE`
|
||||||
|
- `IV`
|
||||||
|
- `CLASSIFIER`
|
||||||
|
|
||||||
|
Column sources:
|
||||||
|
|
||||||
|
- `SPOT`: `execution_underlying_spot`, fallback `--`
|
||||||
|
- `SIDE`: `execution_nbbo_side ?? nbbo_side`
|
||||||
|
- `IV`: `execution_iv`, formatted as percent, fallback `--`
|
||||||
|
- `DETAILS`: `{size}@{price}_{side}`
|
||||||
|
- `VALUE`: `notional ?? price * size * 100`
|
||||||
|
|
||||||
|
For legacy rows only:
|
||||||
|
|
||||||
|
- If preserved NBBO is missing, fallback to existing frontend NBBO map.
|
||||||
|
- If preserved spot/IV is missing, render `--`.
|
||||||
|
|
||||||
|
## Classifier Row Coloring
|
||||||
|
|
||||||
|
Add derived indexes in `TerminalProvider`:
|
||||||
|
|
||||||
|
- `classifierHitsByPacketId`
|
||||||
|
- `packetIdByOptionTraceId`
|
||||||
|
- `classifierDecorByOptionTraceId`
|
||||||
|
|
||||||
|
A print inherits classifier color if its trace ID belongs to a flow packet that produced classifier hits.
|
||||||
|
|
||||||
|
Primary hit selection:
|
||||||
|
|
||||||
|
1. Highest confidence
|
||||||
|
2. Newest `source_ts`
|
||||||
|
3. Highest `seq`
|
||||||
|
|
||||||
|
Classifier families:
|
||||||
|
|
||||||
|
- `large_bullish_call_sweep`: green
|
||||||
|
- `large_bearish_put_sweep`: red
|
||||||
|
- `unusual_contract_spike`: amber
|
||||||
|
- `large_call_sell_overwrite`: copper
|
||||||
|
- `large_put_sell_write`: copper
|
||||||
|
- `straddle` / `strangle`: blue
|
||||||
|
- `vertical_spread`: teal
|
||||||
|
- `ladder_accumulation`: yellow-green
|
||||||
|
- `roll_up_down_out`: violet
|
||||||
|
- `far_dated_conviction`: cyan
|
||||||
|
- `zero_dte_gamma_punch`: magenta
|
||||||
|
- unknown: neutral
|
||||||
|
|
||||||
|
Confidence controls row intensity.
|
||||||
|
|
||||||
|
## Interaction
|
||||||
|
|
||||||
|
Classified rows:
|
||||||
|
|
||||||
|
- Click opens existing classifier/alert drawer behavior through `state.openFromClassifierHit(primaryHit)`.
|
||||||
|
- Keyboard Enter/Space does the same.
|
||||||
|
- Row remains compact and table-like.
|
||||||
|
|
||||||
|
Unclassified rows:
|
||||||
|
|
||||||
|
- Hover only.
|
||||||
|
- No drawer action.
|
||||||
|
|
||||||
|
## Live Manifest
|
||||||
|
|
||||||
|
Update `/tape` live subscriptions to include classifier hits:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
[
|
||||||
|
{ channel: "options", filters: flowFilters },
|
||||||
|
{ channel: "nbbo" },
|
||||||
|
{ channel: "equities" },
|
||||||
|
{ channel: "flow", filters: flowFilters },
|
||||||
|
{ channel: "classifier-hits" }
|
||||||
|
]
|
||||||
|
```
|
||||||
|
|
||||||
|
The table uses preserved execution context from options first, not these side feeds.
|
||||||
|
|
||||||
|
## Tests
|
||||||
|
|
||||||
|
Add/update tests for:
|
||||||
|
|
||||||
|
- `OptionPrintSchema` accepts preserved execution context fields.
|
||||||
|
- ClickHouse option print normalization handles missing legacy context fields.
|
||||||
|
- Ingest enrichment attaches preserved NBBO context.
|
||||||
|
- Ingest enrichment attaches preserved underlying quote mid as spot.
|
||||||
|
- Enrichment never uses quotes after the option print timestamp.
|
||||||
|
- `nbbo_side` mirrors `execution_nbbo_side` for new enriched prints.
|
||||||
|
- Synthetic IV increases under repeated same-contract ask/above-ask buying.
|
||||||
|
- Synthetic IV decays after inactivity.
|
||||||
|
- Synthetic IV remains within clamps.
|
||||||
|
- Options table renders SPOT from `execution_underlying_spot`.
|
||||||
|
- Options table renders IV from `execution_iv`.
|
||||||
|
- Legacy rows render `--` for missing SPOT/IV.
|
||||||
|
- Classifier family mapping and primary hit selection work.
|
||||||
|
- Classified row opens existing classifier/alert drawer path.
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
- The Options tape is a dense table, not card rows.
|
||||||
|
- Every new option print stores preserved execution NBBO context.
|
||||||
|
- Every new option print stores preserved execution underlying spot when an at-or-before equity quote exists.
|
||||||
|
- Synthetic option prints store dynamic IV.
|
||||||
|
- Synthetic repeated buying of the same contract visibly increases IV.
|
||||||
|
- The table reads NBBO, SPOT, and IV from preserved print fields first.
|
||||||
|
- Classifier-hit rows are color-coded by classifier family.
|
||||||
|
- Existing live/replay filters and tape controls still work.
|
||||||
|
- No context field is visually treated as stale after being attached to the print.
|
||||||
|
- Legacy data remains readable with graceful fallbacks.
|
||||||
|
|
||||||
|
## Assumptions
|
||||||
|
|
||||||
|
- Phase 1 uses flat fields for queryability and simple table rendering.
|
||||||
|
- Underlying spot means equity quote mid at or before the option print timestamp.
|
||||||
|
- NBBO context means option quote at or before the option print timestamp.
|
||||||
|
- Preserved age fields are audit metadata, not UI freshness warnings.
|
||||||
|
- Real-feed IV can remain absent until a reliable provider value is available.
|
||||||
Loading…
Add table
Add a link
Reference in a new issue