diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index 2b12057..fb97a9f 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -1,4 +1,4 @@ -{"_type":"issue","id":"islandflow-thp","title":"stabilize live api memory and reduce internal cache churn","description":"The native VPS deployment is repeatedly OOM-killing islandflow-api.service during live operation. The API live cache is retaining oversized channel histories and rewriting large Redis lists on every flush, which drives multi-GB Bun RSS and heavy loopback traffic between the API, Redis, NATS, and ClickHouse. Implement an emergency VPS mitigation plus repo hardening so unsafe env values, reconnect snapshots, and Redis persistence patterns cannot push the live API back into OOM.","acceptance_criteria":"1. VPS live cache env values are reduced to safe defaults and live redis state is cleared before restart. 2. services/api/src/live.ts enforces server-side live cache caps and clamps snapshot_limit accordingly. 3. Hot generic feed Redis persistence no longer rewrites entire lists on every flush. 4. Metrics/logging expose subscription counts, snapshot sizes, redis flush volume, and API memory trend. 5. Relevant tests pass and the deployment is restarted successfully.","notes":"Implemented and deployed the live-state hardening to the VPS. Final validation after restart showed the API around 120 MB RSS with capped live cache depths and clean systemd restarts.","status":"in_progress","priority":1,"issue_type":"bug","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-23T01:30:43Z","created_by":"dirtydishes","updated_at":"2026-05-23T01:50:29Z","started_at":"2026-05-23T01:30:52Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"islandflow-thp","title":"stabilize live api memory and reduce internal cache churn","description":"The native VPS deployment is repeatedly OOM-killing islandflow-api.service during live operation. The API live cache is retaining oversized channel histories and rewriting large Redis lists on every flush, which drives multi-GB Bun RSS and heavy loopback traffic between the API, Redis, NATS, and ClickHouse. Implement an emergency VPS mitigation plus repo hardening so unsafe env values, reconnect snapshots, and Redis persistence patterns cannot push the live API back into OOM.","acceptance_criteria":"1. VPS live cache env values are reduced to safe defaults and live redis state is cleared before restart. 2. services/api/src/live.ts enforces server-side live cache caps and clamps snapshot_limit accordingly. 3. Hot generic feed Redis persistence no longer rewrites entire lists on every flush. 4. Metrics/logging expose subscription counts, snapshot sizes, redis flush volume, and API memory trend. 5. Relevant tests pass and the deployment is restarted successfully.","notes":"Implemented and deployed the live-state hardening to the VPS. Final validation after restart showed the API around 120 MB RSS with capped live cache depths and clean systemd restarts.","status":"closed","priority":1,"issue_type":"bug","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-23T01:30:43Z","created_by":"dirtydishes","updated_at":"2026-05-23T01:50:41Z","started_at":"2026-05-23T01:30:52Z","closed_at":"2026-05-23T01:50:41Z","close_reason":"Closed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-sc6","title":"fix electron codex bridge preload loading","description":"Electron settings showed the browser-only Desktop Required fallback because the renderer did not see the native islandflowDesktop preload bridge or an Electron user-agent marker. Fix the desktop launch path so ChatGPT/Codex subscription controls are available inside Islandflow Desktop again.","notes":"Reopened after live Electron still showed the browser-only fallback. Follow-up fix adds an explicit preload runtime marker and web runtime detection for that marker so Electron is recognized even when the bridge is not ready and the user agent lacks an Electron token.","status":"closed","priority":1,"issue_type":"bug","owner":"dishes@dpdrm.com","created_at":"2026-05-20T23:42:58Z","created_by":"dirtydishes","updated_at":"2026-05-20T23:51:43Z","closed_at":"2026-05-20T23:51:43Z","close_reason":"Follow-up fix added an explicit islandflowDesktopRuntime preload marker and taught the web runtime to recognize that marker plus IslandflowDesktop user-agent tokens, so Electron no longer falls into the browser-only fallback when the AI bridge is delayed or unavailable. Desktop build and focused desktop/web tests pass; full web build still blocked by islandflow-c8f.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-hj3","title":"Fix Electron preload for desktop AI bridge","description":"## Why\\nThe desktop settings page reports the native AI bridge as unavailable because Electron fails to load the preload script in local dev.\\n\\n## What\\nUpdate the desktop preload implementation/build so Electron can execute it, restore window.islandflowDesktop, and verify the Copilot settings panel detects the bridge again.\\n\\n## Acceptance Criteria\\n- Electron no longer logs a preload syntax error\\n- window.islandflowDesktop is available in the desktop renderer\\n- The settings page no longer shows bridge unavailable solely because preload failed\\n- Relevant desktop/web tests pass","status":"closed","priority":1,"issue_type":"bug","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-20T23:16:39Z","created_by":"dirtydishes","updated_at":"2026-05-20T23:20:20Z","started_at":"2026-05-20T23:16:48Z","closed_at":"2026-05-20T23:20:20Z","close_reason":"Closed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-199","title":"fix desktop copilot fallback inside electron","description":"## Why\\nThe settings page can render the browser-only fallback even when Islandflow is running inside the Electron desktop shell.\\n\\n## What\\nSeparate desktop-shell detection from desktop AI transport state, make the provider recover if the bridge appears late or initial state loading fails, and cover the regression with tests.\\n\\n## Acceptance Criteria\\n- The desktop shell no longer shows the browser-only fallback solely because initial bridge state failed or arrived late\\n- Desktop-only actions can distinguish between missing Electron bridge and transport/auth problems\\n- Automated tests cover the recovery behavior","status":"closed","priority":1,"issue_type":"bug","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-20T22:30:16Z","created_by":"dirtydishes","updated_at":"2026-05-20T22:37:21Z","started_at":"2026-05-20T22:30:23Z","closed_at":"2026-05-20T22:37:21Z","close_reason":"Fixed desktop-shell Copilot fallback handling, added bridge recovery logic, updated desktop-vs-bridge UI messaging, and added regression tests. Follow-up tracked in islandflow-c8f for unrelated web build blocker.","dependency_count":0,"dependent_count":0,"comment_count":0} @@ -72,6 +72,7 @@ {"_type":"issue","id":"islandflow-zs0","title":"Migrate terminal UI to smart-money profiles","description":"Migrate apps/web terminal rendering to consume SmartMoneyEvent directly: primary profile, probability ladder, reason codes, and suppression/abstention state, while preserving legacy alert/classifier displays during the bridge.","status":"closed","priority":2,"issue_type":"task","owner":"dishes@dpdrm.com","created_at":"2026-05-04T21:35:23Z","created_by":"dirtydishes","updated_at":"2026-05-05T05:39:58Z","closed_at":"2026-05-05T05:39:58Z","close_reason":"Completed terminal smart-money profile migration","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-igk","title":"Add plan mode","description":"Implement a user-facing plan mode in the application so users can switch into planning before taking action. Scope to be clarified from existing app patterns.","status":"closed","priority":2,"issue_type":"feature","owner":"dishes@dpdrm.com","created_at":"2026-05-04T04:22:37Z","created_by":"dirtydishes","updated_at":"2026-05-04T04:26:18Z","started_at":"2026-05-04T04:22:40Z","closed_at":"2026-05-04T04:26:18Z","close_reason":"Implemented as a global pi extension toggled with Shift+P","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-biq","title":"Finish raw live options delivery and filter/backpressure observability","description":"The smart-money signal path and Tape filters are in place, but the next firehose pass should finish server-side selective raw live delivery for options subscriptions and add explicit filtered-out/backpressure observability for API/web counters. This was discovered while landing islandflow-e4r.\n","status":"in_progress","priority":2,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T20:28:58Z","created_by":"dirtydishes","updated_at":"2026-04-29T03:54:12Z","started_at":"2026-04-29T03:54:12Z","dependencies":[{"issue_id":"islandflow-biq","depends_on_id":"islandflow-e4r","type":"discovered-from","created_at":"2026-04-28T16:28:58Z","created_by":"auto-import","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"islandflow-hpf","title":"add anatomy explainer for options print and smart money flow","description":"Create a standalone docs/anatomy.html reference page that explains the end-to-end lifecycle of an options print through enrichment, signal filtering, compute clustering, flow packet creation, smart-money evaluation, classifier hits, alerts, and API/live consumption. The page should be polished, user-readable, and visually strong enough to serve as a reusable reference artifact for both technical and non-technical readers.","notes":"Added docs/anatomy.html as a standalone reference page for the options-print to smart-money pipeline, styled in the repo product register and layered for executive, mixed technical, and operator-level readers. Regenerated docs/index.html so the page is discoverable from the docs surface.","status":"closed","priority":3,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-23T02:18:48Z","created_by":"dirtydishes","updated_at":"2026-05-23T02:24:58Z","started_at":"2026-05-23T02:18:53Z","closed_at":"2026-05-23T02:24:58Z","close_reason":"Closed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-4ca","title":"Publish May 21 standup git summary","description":"Create the daily standup-ready git activity summary for 2026-05-21, save the HTML artifact under docs/general, add the required turn document, and push the result so the automation leaves a durable record.","status":"closed","priority":3,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-22T13:03:00Z","created_by":"dirtydishes","updated_at":"2026-05-22T13:05:05Z","started_at":"2026-05-22T13:03:03Z","closed_at":"2026-05-22T13:05:05Z","close_reason":"Created the 2026-05-21 standup summary in docs/general, added the required turn document, and prepared the repo for commit/push.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-hgm","title":"Publish May 20 standup git summary","description":"Create the daily standup-ready git activity summary for 2026-05-20, save the HTML artifact under docs/general, and push the result so the automation leaves a durable record.","status":"closed","priority":3,"issue_type":"task","owner":"dishes@dpdrm.com","created_at":"2026-05-21T13:02:38Z","created_by":"dirtydishes","updated_at":"2026-05-21T13:05:16Z","closed_at":"2026-05-21T13:05:16Z","close_reason":"Closed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-4q0","title":"refresh readme app description with current classification approach","description":"Update README intro content to better describe the app's current architecture and include a concise explanation of how Islandflow classifies prints, aligned with smartmoney.md and current services.","status":"closed","priority":3,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-21T01:53:30Z","created_by":"dirtydishes","updated_at":"2026-05-21T01:55:01Z","started_at":"2026-05-21T01:53:33Z","closed_at":"2026-05-21T01:55:01Z","close_reason":"Closed","dependency_count":0,"dependent_count":0,"comment_count":0} diff --git a/docs/anatomy.html b/docs/anatomy.html new file mode 100644 index 0000000..bdd3da5 --- /dev/null +++ b/docs/anatomy.html @@ -0,0 +1,954 @@ + + + + + + The Anatomy of an Options Print and Smart Money + + + +
+
+
+ Islandflow Reference · Options Flow Pipeline + +
+ +
+
+

The Anatomy of an Options Print and Smart Money

+

+ This page explains how a single options print moves through Islandflow under normal market conditions, + how the signal gate decides whether compute should care, how a parent flow packet is assembled, and how + smart-money, classifier-hit, and alert events emerge from that packet. It is designed as one artifact + with three reading depths: executive, mixed technical, and operator-level. +

+
+ +
+
+ +
+
+
+

Legend

+

Color coding is semantic, not decorative, so you can scan the diagram without relearning the vocabulary.

+
+
+ Raw market or synthetic input + Derived compute stage + Stored or persisted state + API, websocket, or user-facing surface +
+
+ +
+
+

Main Flow Chart

+

+ The first row shows the common path every print touches. The second row shows the branch between prints + that remain tape-only and prints that become packet candidates for smart-money evaluation. +

+
+ +
+
+
+
Input
+
+ Stage 1 + Option print candidate arrives +

+ The source can be a native market adapter or the synthetic adapter. Synthetic mode can also emit a + matching NBBO update. +

+
+
+ Stage 2 + ingest-options enriches the print +

+ The service joins recent option NBBO and underlying equity quote context, derives metadata, and + computes signal_pass. +

+
+
+ Stage 3 + Raw print is written and published +
    +
  • ClickHouse: option_prints
  • +
  • NATS: options.prints
  • +
+
+
+ Stage 4 + Signal gate decides if compute should care +

+ Only signal_pass=true prints are published to options.prints.signal and + consumed by compute. +

+
+
+ Stage 5 + compute builds or updates a parent cluster +

+ Nearby signal prints for the same contract are grouped inside the cluster window while NBBO and + equity-quote caches supply context. +

+
+
+ Stage 6 + API and UI consume the resulting streams +

+ The API hydrates hot snapshots, history endpoints read ClickHouse, and the terminal surfaces tape, + flow, smart-money, classifier, and alert views. +

+
+
+ +
+
Tape-only branch
+
+
+
+ Branch A + Raw print remains visible +

+ Even if the print does not pass the signal gate, it still exists in ClickHouse and can appear in + raw tape or history views. +

+
+
+ Branch A outcome + No compute packet path +

+ No FlowPacket, no smart-money evaluation, no classifier hits, and no alert emission. +

+
+
+
+
+ +
+
Smart-money branch
+
+
+
+
+ Branch B + Signal print enters compute +

+ compute subscribes to options.prints.signal, not raw options.prints. +

+
+
+ Branch B outcome + FlowPacket is emitted +
    +
  • ClickHouse: flow_packets
  • +
  • NATS: flow.packets
  • +
+
+
+ Branch B continuation + Smart-money, classifier hits, alerts +

+ The packet is scored into a SmartMoneyEvent, which may abstain, produce classifier + hits, and finally emit an alert. +

+
+
+
+
+
+ +
+
+

Executive Read

+

+ The shortest truthful version of the system: not every options print is considered meaningful, and smart + money is not detected directly from a single print. +

+
+
+
+ 1. Tape +

Every print is stored

+

+ All enriched prints are written to ClickHouse and published to the raw options subject. This preserves + evidence even when the print is uninteresting for higher-order inference. +

+
+
+ 2. Compute +

Only signal prints reach the parent-event engine

+

+ A print must pass the signal gate before compute clusters it with neighboring prints and builds a + packet that represents a possible parent order. +

+
+
+ 3. Smart money +

Smart money is a scored interpretation

+

+ The model evaluates the packet using quote quality, aggressor mix, size, structure, DTE, IV, and event + context. It can still abstain if the evidence is weak or suppressed. +

+
+
+
+ +
+
+

Mixed Technical Walkthrough

+

+ This layer is for teammates who know the product and want the exact branching logic without reading + through service code first. +

+
+
+
+

+ Step 1: a candidate print enters ingest-options. In synthetic mode this + print was manufactured by the synthetic adapter, which may also emit a synthetic NBBO update for the + same contract. +

+

+ Step 2: the print is enriched with the most recent option NBBO and underlying equity + quote at or before the print timestamp. The service derives metadata, execution-side context, and the + signal_pass decision. +

+

+ Step 3: the enriched print is persisted to ClickHouse and published to + options.prints. If signal_pass=true, the same print is also published to + options.prints.signal. +

+

+ Step 4: compute subscribes to the signal subject plus NBBO and equity-quote subjects. + It does not build packet candidates from every raw print. It only clusters signal prints. +

+

+ Step 5: compute aggregates nearby signal prints for the same option contract into a + cluster, then flushes that cluster into a FlowPacket with features such as total premium, + print count, aggressor ratios, NBBO coverage, stale-quote counts, IV context, and structure clues. +

+

+ Step 6: the packet is transformed into a SmartMoneyEvent. If suppression + rules trip or the top profile probability is too weak, the event abstains. Otherwise, it can emit + classifier hits and finally an alert with evidence references back to the packet and member prints. +

+
+ +
+
+ +
+
+

Operator and Code-Level Detail

+

+ This section is for someone tracing the live pipeline, debugging a regression, or trying to understand + exactly why a given print surfaced on tape but did or did not become a smart-money event. +

+
+
+

+ The first fork is the signal gate in ingest-options. The enriched print is always stored and + published raw. The only thing signal_pass controls is whether compute receives that print on + options.prints.signal. +

+

+ The compute service maintains separate caches for option NBBO and underlying equity quotes. When signal + prints arrive, it flushes aged clusters, extends the active cluster for that contract if the print lands + within the configured window, or emits the old cluster and starts a new one. +

+

+ The cluster becomes a FlowPacket only after compute summarizes parent-level features. That + packet then passes through smart-money scoring. The scoring layer derives a profile set such as + institutional directional, retail whale, event driven, vol seller, arbitrage, or hedge reactive. +

+

+ A packet can still fail to produce actionable downstream artifacts. Suppression rules down-rank special + print context, stale or missing quote context, and cross-like execution patterns. The top profile must + also clear the probability threshold. If it does not, the smart-money event is emitted in abstained form + and classifier hits stop there. +

+

+ If the packet does clear those checks, compute writes and publishes the smart-money event, derives up to + a few classifier hits from the top profile set, scores a final alert, and publishes all three derived + streams. The API subscribes to those subjects and fans them out into live websocket channels while + ClickHouse remains the history source behind /history/*. +

+
+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Subject or tableProduced byCarriesWhy it exists
options.printsingest-optionsAll enriched option printsPreserves the full tape, even when a print is not interesting enough for compute.
options.prints.signalingest-optionsSignal-passing option printsActs as the compute admission gate so packet building starts from a filtered tape.
flow.packetscomputeParent-event candidatesTurns several child prints into one summarized event with market-structure features.
flow.smart_moneycomputeSmart-money evaluationsPublishes the scored interpretation of a packet, including abstained outcomes.
flow.classifier_hitscomputeTop classifier consequencesExposes the strongest profile-level labels that downstream UX and alerting can decorate.
flow.alertscomputeAlert events with evidence refsPackages the final severity and supporting evidence into a user-facing alert stream.
+
+
+ +
+
+

Normal Path Versus Smart-Money Path

+

+ These two sequences are easy to confuse, especially because both begin with the same enriched tape + record. +

+
+
+
+ Normal market path +

+ Print arrives, gets enriched, gets stored, appears on the raw tape, and stops there unless it passes + the signal gate. This is the dominant path for ordinary or low-signal activity. +

+
+
+ Smart-money path +

+ Print arrives, passes the signal gate, joins a cluster, becomes a packet, receives a smart-money score, + then may emit classifier hits and an alert if the packet is not suppressed or abstained. +

+
+
+
+ +
+
+

Annotated Event Sequence

+

+ The example below is the shortest operator-friendly way to think about the branch that leads to a + smart-money result. +

+
+
1. Synthetic or market adapter emits OptionPrint candidate
+2. ingest-options enriches it with latest NBBO and underlying quote context
+3. Enriched print is written to ClickHouse option_prints
+4. Enriched print is published to options.prints
+5. If signal_pass=true, the same print is also published to options.prints.signal
+6. compute consumes options.prints.signal and updates the active contract cluster
+7. Cluster flush builds a FlowPacket with parent-level features
+8. FlowPacket is written to ClickHouse flow_packets and published to flow.packets
+9. compute scores the packet into a SmartMoneyEvent
+10. If suppressed or low-confidence, the SmartMoneyEvent abstains and stops there
+11. Otherwise classifier hits are emitted
+12. Alert scoring emits a final alert with evidence refs to smart-money event, flow packet, and member prints
+13. API subscribes to these streams and exposes them through live websocket channels and ClickHouse-backed history
+
+ +
+
+

What Synthetic Mode Changes

+

+ Synthetic mode can make the upstream generator artificial, but the downstream branch logic stays + identical. +

+
+
+

+ The synthetic adapter constructs an OptionPrint with fields such as + execution_iv_source="synthetic_pressure_model", and it may emit a synthetic NBBO for the + same contract. From that point forward, the pipeline is the same one used for normal ingest. +

+

+ That means synthetic smart-money is not a special smart-money subsystem. It is the standard + signal-to-packet-to-smart-money pipeline running on synthetic upstream events. +

+
+
+ +
+
+

Code Anchors

+

+ If you want to confirm this page against the code, these are the most useful entry points. +

+
+
+
    +
  • services/ingest-options/src/enrichment.ts: enriches the print and decides signal_pass.
  • +
  • services/ingest-options/src/index.ts: writes prints and publishes raw versus signal subjects.
  • +
  • services/compute/src/index.ts: subscribes to signal prints, maintains clusters, emits packets, smart money, hits, and alerts.
  • +
  • services/compute/src/parent-events.ts: builds SmartMoneyEvent, suppression rules, primary profile, abstention, and classifier derivation.
  • +
  • packages/bus/src/subjects.ts: canonical subject names for the pipeline.
  • +
+
+ +
+
+
+ + diff --git a/docs/index.html b/docs/index.html index 211c5ac..140ee55 100644 --- a/docs/index.html +++ b/docs/index.html @@ -207,36 +207,76 @@
-
35 of 35 files shown
+
47 of 47 files shown
-
-

turns 28

+

turns 37