diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index 01e0621..12572eb 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -21,6 +21,7 @@ {"_type":"issue","id":"islandflow-ayo","title":"Drop stale backlog events from live fanout","description":"Follow-up to live freshness rollout: /ws/live was still fanning out stale backlog events for freshness-gated channels, which kept tape panes in Live feed behind despite active synthetic ingest. Gate fanout and cache ingest by freshness for options/nbbo/equities/flow.","status":"closed","priority":1,"issue_type":"bug","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T21:26:39Z","created_by":"dirtydishes","updated_at":"2026-04-28T21:26:44Z","started_at":"2026-04-28T21:26:44Z","closed_at":"2026-04-28T21:26:44Z","close_reason":"Completed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-0v6","title":"Fix tape freshness, NBBO coverage, pause controls, and filter popup","description":"Implement the tape fixes requested for synthetic options notional sizing, strict live freshness, live-mode pause/resume behavior, stronger NBBO snapshot coverage, and moving flow filters behind a popup. Includes server-side live cache changes, web terminal state/UI changes, and tests for synthetic pricing, live snapshot freshness/NBBO retention, and live pause/filter interactions.","status":"closed","priority":1,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T21:02:52Z","created_by":"dirtydishes","updated_at":"2026-04-28T21:13:38Z","started_at":"2026-04-28T21:02:57Z","closed_at":"2026-04-28T21:13:38Z","close_reason":"Completed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-e4r","title":"Implement smart-money flow filtering and synthetic firehose modes","description":"Implement the approved multi-surface plan for named synthetic market profiles, options raw-vs-signal filtering, live/API filter contracts, Tape page client-side flow filters, firehose-readiness improvements, tests, and README updates.","status":"closed","priority":1,"issue_type":"feature","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T20:10:49Z","created_by":"dirtydishes","updated_at":"2026-04-28T20:29:29Z","started_at":"2026-04-28T20:10:53Z","closed_at":"2026-04-28T20:29:29Z","close_reason":"Implemented synthetic market profiles, options signal-path filtering, signal-aware API/replay contracts, Tape page filters, tests, and README updates. Follow-up tracked in islandflow-biq.","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"islandflow-kgu","title":"Reconcile PR #8 branch with current main","description":"Why this issue exists and what needs to be done: user requested reconciliation for PR #8. Identify the PR #8 branch, merge/rebase with current main, resolve conflicts, validate, and push the updated branch so the PR can merge cleanly.","status":"closed","priority":2,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-23T20:14:36Z","created_by":"dirtydishes","updated_at":"2026-05-23T20:24:29Z","started_at":"2026-05-23T20:14:39Z","closed_at":"2026-05-23T20:24:29Z","close_reason":"Closed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-l9h","title":"stop persisting non-signal option prints in clickhouse","description":"Why: non-signal option prints are storage noise and should not be persisted by default.\\n\\nWhat: add OPTIONS_PERSIST_SIGNAL_ONLY env flag (default true), gate option_print inserts in ingest-options, add tests for persistence behavior, update env examples, and document one-off cleanup SQL for existing non-signal rows.","status":"closed","priority":2,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-23T03:02:32Z","created_by":"dirtydishes","updated_at":"2026-05-23T03:06:34Z","started_at":"2026-05-23T03:02:35Z","closed_at":"2026-05-23T03:06:34Z","close_reason":"Closed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-2cj","title":"Add Forgejo-first agent workflow guidance to AGENTS.md","description":"Why this issue exists and what needs to be done:\\n- The repository’s canonical home is Forgejo at git.deltaisland.io, but AGENTS.md does not currently direct agents to prefer Forgejo-specific workflows.\\n- Update AGENTS.md so agents treat Forgejo as primary and use the fj CLI for pull request workflows.\\n- Keep existing Beads and completion instructions intact while clarifying remote preference and command usage.","status":"closed","priority":2,"issue_type":"task","owner":"dishes@dpdrm.com","created_at":"2026-05-23T02:51:31Z","created_by":"dirtydishes","updated_at":"2026-05-23T02:55:42Z","closed_at":"2026-05-23T02:55:42Z","close_reason":"Closed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-xc5","title":"One-time bidirectional git remote backfill between github and forgejo","description":"Perform a one-time sync so github and forgejo contain the same branch/tag refs and historical commits, including pre-transition github history and newer forgejo commits. Document exact commands and validation results.","status":"closed","priority":2,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-21T01:25:05Z","created_by":"dirtydishes","updated_at":"2026-05-21T01:26:19Z","started_at":"2026-05-21T01:25:16Z","closed_at":"2026-05-21T01:26:19Z","close_reason":"Closed","dependency_count":0,"dependent_count":0,"comment_count":0} diff --git a/.env.example b/.env.example index be20b62..2d36859 100644 --- a/.env.example +++ b/.env.example @@ -106,7 +106,7 @@ REPLAY_LOG_EVERY=1000 # API live retention (generic channels) LIVE_LIMIT_DEFAULT=1000 -LIVE_LIMIT_OPTIONS=1000 +LIVE_LIMIT_OPTIONS=100 LIVE_LIMIT_NBBO=1000 LIVE_LIMIT_EQUITIES=1000 LIVE_LIMIT_EQUITY_QUOTES=500 @@ -116,6 +116,7 @@ LIVE_LIMIT_SMART_MONEY=300 LIVE_LIMIT_CLASSIFIER_HITS=300 LIVE_LIMIT_ALERTS=300 LIVE_LIMIT_INFERRED_DARK=300 +LIVE_LIMIT_NEWS=100 LIVE_SCOPED_CACHE_MAX_KEYS=32 LIVE_REDIS_FLUSH_INTERVAL_MS=250 LIVE_REDIS_FLUSH_MAX_ITEMS=100 diff --git a/apps/web/app/terminal.tsx b/apps/web/app/terminal.tsx index 3057f58..5bf1641 100644 --- a/apps/web/app/terminal.tsx +++ b/apps/web/app/terminal.tsx @@ -72,12 +72,12 @@ const parseBoundedInt = ( return Math.max(min, Math.min(max, Math.floor(parsed))); }; -const LIVE_HOT_WINDOW = parseBoundedInt(process.env.NEXT_PUBLIC_LIVE_HOT_WINDOW, 600, 1, 100000); +const LIVE_HOT_WINDOW = parseBoundedInt(process.env.NEXT_PUBLIC_LIVE_HOT_WINDOW, 600, 1, 2000); const LIVE_HOT_WINDOW_OPTIONS = parseBoundedInt( process.env.NEXT_PUBLIC_LIVE_HOT_WINDOW_OPTIONS, 1200, 1, - 100000 + 2000 ); const LIVE_OPTIONS_HEAD_LIMIT = 100; const LIVE_HISTORY_SOFT_CAP = parseBoundedInt( diff --git a/deployment/docker/.env.example b/deployment/docker/.env.example index 4972ada..ff4d7f3 100644 --- a/deployment/docker/.env.example +++ b/deployment/docker/.env.example @@ -132,7 +132,7 @@ REPLAY_LOG_EVERY=1000 # API live retention LIVE_LIMIT_DEFAULT=1000 -LIVE_LIMIT_OPTIONS=1000 +LIVE_LIMIT_OPTIONS=100 LIVE_LIMIT_NBBO=1000 LIVE_LIMIT_EQUITIES=1000 LIVE_LIMIT_EQUITY_QUOTES=500 @@ -142,6 +142,7 @@ LIVE_LIMIT_SMART_MONEY=300 LIVE_LIMIT_CLASSIFIER_HITS=300 LIVE_LIMIT_ALERTS=300 LIVE_LIMIT_INFERRED_DARK=300 +LIVE_LIMIT_NEWS=100 LIVE_SCOPED_CACHE_MAX_KEYS=32 LIVE_REDIS_FLUSH_INTERVAL_MS=250 LIVE_REDIS_FLUSH_MAX_ITEMS=100 diff --git a/docs/anatomy.html b/docs/anatomy.html new file mode 100644 index 0000000..bdd3da5 --- /dev/null +++ b/docs/anatomy.html @@ -0,0 +1,954 @@ + + + + + + The Anatomy of an Options Print and Smart Money + + + +
+
+
+ Islandflow Reference · Options Flow Pipeline + +
+ +
+
+

The Anatomy of an Options Print and Smart Money

+

+ This page explains how a single options print moves through Islandflow under normal market conditions, + how the signal gate decides whether compute should care, how a parent flow packet is assembled, and how + smart-money, classifier-hit, and alert events emerge from that packet. It is designed as one artifact + with three reading depths: executive, mixed technical, and operator-level. +

+
+ +
+
+ +
+
+
+

Legend

+

Color coding is semantic, not decorative, so you can scan the diagram without relearning the vocabulary.

+
+
+ Raw market or synthetic input + Derived compute stage + Stored or persisted state + API, websocket, or user-facing surface +
+
+ +
+
+

Main Flow Chart

+

+ The first row shows the common path every print touches. The second row shows the branch between prints + that remain tape-only and prints that become packet candidates for smart-money evaluation. +

+
+ +
+
+
+
Input
+
+ Stage 1 + Option print candidate arrives +

+ The source can be a native market adapter or the synthetic adapter. Synthetic mode can also emit a + matching NBBO update. +

+
+
+ Stage 2 + ingest-options enriches the print +

+ The service joins recent option NBBO and underlying equity quote context, derives metadata, and + computes signal_pass. +

+
+
+ Stage 3 + Raw print is written and published +
    +
  • ClickHouse: option_prints
  • +
  • NATS: options.prints
  • +
+
+
+ Stage 4 + Signal gate decides if compute should care +

+ Only signal_pass=true prints are published to options.prints.signal and + consumed by compute. +

+
+
+ Stage 5 + compute builds or updates a parent cluster +

+ Nearby signal prints for the same contract are grouped inside the cluster window while NBBO and + equity-quote caches supply context. +

+
+
+ Stage 6 + API and UI consume the resulting streams +

+ The API hydrates hot snapshots, history endpoints read ClickHouse, and the terminal surfaces tape, + flow, smart-money, classifier, and alert views. +

+
+
+ +
+
Tape-only branch
+
+
+
+ Branch A + Raw print remains visible +

+ Even if the print does not pass the signal gate, it still exists in ClickHouse and can appear in + raw tape or history views. +

+
+
+ Branch A outcome + No compute packet path +

+ No FlowPacket, no smart-money evaluation, no classifier hits, and no alert emission. +

+
+
+
+
+ +
+
Smart-money branch
+
+
+
+
+ Branch B + Signal print enters compute +

+ compute subscribes to options.prints.signal, not raw options.prints. +

+
+
+ Branch B outcome + FlowPacket is emitted +
    +
  • ClickHouse: flow_packets
  • +
  • NATS: flow.packets
  • +
+
+
+ Branch B continuation + Smart-money, classifier hits, alerts +

+ The packet is scored into a SmartMoneyEvent, which may abstain, produce classifier + hits, and finally emit an alert. +

+
+
+
+
+
+ +
+
+

Executive Read

+

+ The shortest truthful version of the system: not every options print is considered meaningful, and smart + money is not detected directly from a single print. +

+
+
+
+ 1. Tape +

Every print is stored

+

+ All enriched prints are written to ClickHouse and published to the raw options subject. This preserves + evidence even when the print is uninteresting for higher-order inference. +

+
+
+ 2. Compute +

Only signal prints reach the parent-event engine

+

+ A print must pass the signal gate before compute clusters it with neighboring prints and builds a + packet that represents a possible parent order. +

+
+
+ 3. Smart money +

Smart money is a scored interpretation

+

+ The model evaluates the packet using quote quality, aggressor mix, size, structure, DTE, IV, and event + context. It can still abstain if the evidence is weak or suppressed. +

+
+
+
+ +
+
+

Mixed Technical Walkthrough

+

+ This layer is for teammates who know the product and want the exact branching logic without reading + through service code first. +

+
+
+
+

+ Step 1: a candidate print enters ingest-options. In synthetic mode this + print was manufactured by the synthetic adapter, which may also emit a synthetic NBBO update for the + same contract. +

+

+ Step 2: the print is enriched with the most recent option NBBO and underlying equity + quote at or before the print timestamp. The service derives metadata, execution-side context, and the + signal_pass decision. +

+

+ Step 3: the enriched print is persisted to ClickHouse and published to + options.prints. If signal_pass=true, the same print is also published to + options.prints.signal. +

+

+ Step 4: compute subscribes to the signal subject plus NBBO and equity-quote subjects. + It does not build packet candidates from every raw print. It only clusters signal prints. +

+

+ Step 5: compute aggregates nearby signal prints for the same option contract into a + cluster, then flushes that cluster into a FlowPacket with features such as total premium, + print count, aggressor ratios, NBBO coverage, stale-quote counts, IV context, and structure clues. +

+

+ Step 6: the packet is transformed into a SmartMoneyEvent. If suppression + rules trip or the top profile probability is too weak, the event abstains. Otherwise, it can emit + classifier hits and finally an alert with evidence references back to the packet and member prints. +

+
+ +
+
+ +
+
+

Operator and Code-Level Detail

+

+ This section is for someone tracing the live pipeline, debugging a regression, or trying to understand + exactly why a given print surfaced on tape but did or did not become a smart-money event. +

+
+
+

+ The first fork is the signal gate in ingest-options. The enriched print is always stored and + published raw. The only thing signal_pass controls is whether compute receives that print on + options.prints.signal. +

+

+ The compute service maintains separate caches for option NBBO and underlying equity quotes. When signal + prints arrive, it flushes aged clusters, extends the active cluster for that contract if the print lands + within the configured window, or emits the old cluster and starts a new one. +

+

+ The cluster becomes a FlowPacket only after compute summarizes parent-level features. That + packet then passes through smart-money scoring. The scoring layer derives a profile set such as + institutional directional, retail whale, event driven, vol seller, arbitrage, or hedge reactive. +

+

+ A packet can still fail to produce actionable downstream artifacts. Suppression rules down-rank special + print context, stale or missing quote context, and cross-like execution patterns. The top profile must + also clear the probability threshold. If it does not, the smart-money event is emitted in abstained form + and classifier hits stop there. +

+

+ If the packet does clear those checks, compute writes and publishes the smart-money event, derives up to + a few classifier hits from the top profile set, scores a final alert, and publishes all three derived + streams. The API subscribes to those subjects and fans them out into live websocket channels while + ClickHouse remains the history source behind /history/*. +

+
+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Subject or tableProduced byCarriesWhy it exists
options.printsingest-optionsAll enriched option printsPreserves the full tape, even when a print is not interesting enough for compute.
options.prints.signalingest-optionsSignal-passing option printsActs as the compute admission gate so packet building starts from a filtered tape.
flow.packetscomputeParent-event candidatesTurns several child prints into one summarized event with market-structure features.
flow.smart_moneycomputeSmart-money evaluationsPublishes the scored interpretation of a packet, including abstained outcomes.
flow.classifier_hitscomputeTop classifier consequencesExposes the strongest profile-level labels that downstream UX and alerting can decorate.
flow.alertscomputeAlert events with evidence refsPackages the final severity and supporting evidence into a user-facing alert stream.
+
+
+ +
+
+

Normal Path Versus Smart-Money Path

+

+ These two sequences are easy to confuse, especially because both begin with the same enriched tape + record. +

+
+
+
+ Normal market path +

+ Print arrives, gets enriched, gets stored, appears on the raw tape, and stops there unless it passes + the signal gate. This is the dominant path for ordinary or low-signal activity. +

+
+
+ Smart-money path +

+ Print arrives, passes the signal gate, joins a cluster, becomes a packet, receives a smart-money score, + then may emit classifier hits and an alert if the packet is not suppressed or abstained. +

+
+
+
+ +
+
+

Annotated Event Sequence

+

+ The example below is the shortest operator-friendly way to think about the branch that leads to a + smart-money result. +

+
+
1. Synthetic or market adapter emits OptionPrint candidate
+2. ingest-options enriches it with latest NBBO and underlying quote context
+3. Enriched print is written to ClickHouse option_prints
+4. Enriched print is published to options.prints
+5. If signal_pass=true, the same print is also published to options.prints.signal
+6. compute consumes options.prints.signal and updates the active contract cluster
+7. Cluster flush builds a FlowPacket with parent-level features
+8. FlowPacket is written to ClickHouse flow_packets and published to flow.packets
+9. compute scores the packet into a SmartMoneyEvent
+10. If suppressed or low-confidence, the SmartMoneyEvent abstains and stops there
+11. Otherwise classifier hits are emitted
+12. Alert scoring emits a final alert with evidence refs to smart-money event, flow packet, and member prints
+13. API subscribes to these streams and exposes them through live websocket channels and ClickHouse-backed history
+
+ +
+
+

What Synthetic Mode Changes

+

+ Synthetic mode can make the upstream generator artificial, but the downstream branch logic stays + identical. +

+
+
+

+ The synthetic adapter constructs an OptionPrint with fields such as + execution_iv_source="synthetic_pressure_model", and it may emit a synthetic NBBO for the + same contract. From that point forward, the pipeline is the same one used for normal ingest. +

+

+ That means synthetic smart-money is not a special smart-money subsystem. It is the standard + signal-to-packet-to-smart-money pipeline running on synthetic upstream events. +

+
+
+ +
+
+

Code Anchors

+

+ If you want to confirm this page against the code, these are the most useful entry points. +

+
+
+
    +
  • services/ingest-options/src/enrichment.ts: enriches the print and decides signal_pass.
  • +
  • services/ingest-options/src/index.ts: writes prints and publishes raw versus signal subjects.
  • +
  • services/compute/src/index.ts: subscribes to signal prints, maintains clusters, emits packets, smart money, hits, and alerts.
  • +
  • services/compute/src/parent-events.ts: builds SmartMoneyEvent, suppression rules, primary profile, abstention, and classifier derivation.
  • +
  • packages/bus/src/subjects.ts: canonical subject names for the pipeline.
  • +
+
+ +
+
+
+ + diff --git a/docs/index.html b/docs/index.html index 211c5ac..140ee55 100644 --- a/docs/index.html +++ b/docs/index.html @@ -207,36 +207,76 @@
-
35 of 35 files shown
+
47 of 47 files shown
-
-

turns 28

+

turns 37