islandflow/PLAN.md

PLAN.md — Real-Time Options Flow & Off-Exchange Analysis Platform

Purpose

Build a real-time, non-delayed market-flow analysis system for personal use that ingests options trades/quotes and equity prints, clusters raw activity into higher-level flow events, applies explainable rule-first classifiers, infers dark-pool-like behavior, and visualizes everything in a TradingView-smooth interface with full replay and backtesting.

---

Non-Negotiables

• Runtime & tooling: Bun everywhere (services, scripts, dev, CI)
• Language: TypeScript
• Frontend: Next.js + React (App Router)
• Realtime: WebSockets (server → client)
• Eventing: NATS JetStream (default) or Redpanda (Kafka-compatible)
• Storage: ClickHouse (authoritative event log + analytics), Redis (hot state)
• Charting: TradingView Lightweight Charts + custom Canvas/WebGL overlays
• Scope: Personal, non-delayed use only (no redistribution)

---

Guiding Principles

• Explainability first: every alert links to evidence and explicit logic.
• Event-sourced: raw and derived events are persisted and replayable.
• Microstructure correctness: conservative inference, explicit confidence.
• Low latency UX: smooth pan/zoom, minimal main-thread work.
• Determinism: live behavior equals replay behavior.

---

High-Level Architecture

Sources → Ingest → Event Bus → Compute → Storage → API/WS → UI

• Sources: options trades/quotes (OPRA-derived via licensed source), equity trades/quotes (incl. off-exchange flags)
• Ingest services normalize and publish immutable events
• Compute clusters prints, computes rolling stats, runs classifiers, emits alerts and inferred events
• ClickHouse stores everything; Redis serves hot joins/baselines
• API/WS streams curated live data and serves historical queries
• Next.js UI renders live terminals and charts

---

Monorepo Layout (Bun workspaces)

apps/ web/ # Next.js UI (flow, charts, alerts) services/ ingest-options/ # Options feed adapters (trades + NBBO) ingest-equities/ # Equity trades/quotes ingestion compute/ # Clustering, stats, classifiers, inference candles/ # Server-side candle aggregation refdata/ # Symbols, chains, corp actions eod-enricher/ # OI + metadata snapshots api/ # REST + WebSocket gateway packages/ types/ # Shared TS types + zod schemas ui/ # Design system + motion primitives chart/ # Chart wrappers + overlay renderers

---

Core Event Schemas (canonical)

• OptionPrint { ts, option_contract_id, price, size, exchange, conditions }
• OptionNBBO { ts, option_contract_id, bid, ask, bidSize, askSize }
• EquityPrint { ts, underlying_id, price, size, exchange, offExchangeFlag }
• EquityQuote { ts, underlying_id, bid, ask }
• FlowPacket { id, members[], features{}, join_quality{} }
• ClassifierHit { classifier_id, confidence, direction, explanations[] }
• AlertEvent { score, severity, hits[], evidence_refs[] }
• InferredDarkEvent { type, confidence, evidence_refs[] }

All events include { source_ts, ingest_ts, seq, trace_id }.

---

Epic 1 — Repo Scaffold & Infra (Day 1)

Build

• Initialize Bun monorepo; Docker compose for ClickHouse, Redis, NATS.
• Shared config, logging (JSON), metrics hooks.
• Define zod schemas + TS types.

Acceptance

• bun run dev boots infra + empty services + web shell.

---

Epic 2 — Realtime Ingestion (Days 2–4)

Options Ingestor

• Adapter interface: connect/subscribe/onTrade/onNBBO.
• Normalize to OptionPrint/OptionNBBO; publish.

Equity Ingestor

• Stream EquityPrint/EquityQuote; tag off-exchange when provided.

Acceptance

• Live events visible via CLI subscriber.
• Raw events persisted to ClickHouse.

---

Epic 3 — Rolling Stats & Clustering (Days 4–7)

Rolling Stats (Redis)

• Premium/size baselines (median/MAD or mean/std).
• Intraday curves; liquidity/spread penalties.

Clustering

• Contract sweeps (250–2000ms windows).
• Adjacent-strike ladders.
• Multi-leg detection (spreads/straddles/rolls).

Acceptance

• Deterministic FlowPacket emission; replayable.

---

Epic 4 — Classifiers & Alert Scoring (Days 7–10)

Implement rule-first classifiers (each returns confidence + explanation):

1. Large Bullish Call Sweep
2. Large Bearish Put Sweep
3. Large Call Sell (overwrite)
4. Large Put Sell (put write)
5. Unusual Contract Spike (z-score)
6. New Position Likely (vol/OI)
7. Closing/Unwind Likely
8. 0DTE Gamma Punch (ATM)
9. Far-Dated Conviction (60DTE+)
10. Straddle/Strangle (long/short vol)
11. Vertical Spread (debit/credit)
12. Roll Up/Down/Out
13. Multi-Strike Ladder Accumulation
14. No Follow-Through / Absorbed

Alert Scoring

• Weighted score from premium, aggressor, z-scores, structure bonus minus noise.
• Throttles, dedupe, cooldowns.

Acceptance

• Alerts fire with human-readable “why”; unit tests per classifier.

---

Epic 5 — Dark Pool Inference (Days 10–12)

Inference (derived, separate from raw)

• Absorbed blocks
• Stealth accumulation
• Distribution
• Hidden liquidity zones

Acceptance

• InferredDarkEvent links to evidence windows and chart markers.

---

Epic 6 — API & WebSockets (Days 12–14)

• REST: queries for prints, packets, inferred events; candle ranges.
• WS: channels for live flow, alerts, equity prints, inferred events.
• Backpressure aware fan-out.

Acceptance

• Stable live streaming under load.

---

Epic 7 — UI: Live Terminals & Workspaces (Days 14–18)

• Live options flow terminal (virtualized, deep filters).
• Dark pool/off-exchange tape.
• Alerts center.
• Ticker workspace (flow + chart + tape).
• Tangible UX: motion, depth, blueprint grid.

Acceptance

• 60fps interaction while streaming.

---

Epic 8 — Charting (Days 18–24)

Base

• TradingView Lightweight Charts (candles, volume, crosshair).

Overlays

• Off-exchange prints as circles (radius ~ sqrt(size)).
• Inferred events & classifier markers.
• Viewport-driven rendering; OffscreenCanvas when available.

Acceptance

• TV-smooth pan/zoom; overlays stay aligned.

---

Epic 9 — Replay & Backtesting (Days 24–28)

• Replay mode re-streams from ClickHouse.
• Metrics: forward returns, hit rates, calibration.

Acceptance

• Live and replay share the same pipeline.

---

ADDENDUM — Missing-but-Crucial Epics

Epic 10 — Reference Data, Symbology & Corporate Actions

• Underlyings, option chains, OCC adjustments, corp actions.
• Canonical IDs; symbol normalizer.

Acceptance

• Splits/adjustments don’t break replay or joins.

Epic 11 — EOD Enrichment (OI & Metadata)

• Nightly OI snapshots; provenance tagging.

Acceptance

• OI-based features reference correct snapshot date.

Epic 12 — Time Sync, Ordering & Join Quality

• NTP/chrony; bounded join windows; join quality scores.

Acceptance

• Stable aggressor inference under replay.

Epic 13 — Candle Aggregation Service

• Server-built 1s/5s/1m OHLCV; Redis hot cache.

Acceptance

• Candle queries <100ms for hot ranges.

Epic 14 — Backpressure & Load Shedding

• Bounded queues; UI sampling; DLQ; replay namespace.

Acceptance

• System degrades gracefully during spikes.

Epic 15 — Observability

• Metrics, structured logs, tracing IDs.

Acceptance

• End-to-end lag visible in one dashboard.

Epic 16 — Secure Personal Deployment

• Auth (single-user), TLS, rate limits; no public endpoints.

Acceptance

• Anonymous access blocked; VPS-safe.

Epic 17 — UX State: Saved Filters, Workspaces, Hotkeys

• Presets, layouts, evidence panel.

Acceptance

• One-click reproducibility of setups.

---

Milestones

• MVP-1: realtime options flow, clustering, 10+ classifiers, live terminal.
• MVP-2: off-exchange prints, inferred DP events, chart overlays, alerts.
• MVP-2.5: candles, observability, backpressure, auth.
• MVP-3: replay/backtesting, metrics dashboards.

---

Non-Goals

• No black-box predictions
• No profit guarantees
• No real-time redistribution

Notes

• Market data usage must comply with provider terms.
• All inference is probabilistic and labeled as such.