From f91856ca5ef1e525858b96636c2507044d779e6c Mon Sep 17 00:00:00 2001 From: dirtydishes Date: Wed, 13 May 2026 22:53:33 -0400 Subject: [PATCH] Add HTML synthetic tape redesign plans --- .beads/issues.jsonl | 1 + plans/synthetic-tape-redesign-impeccable.html | 816 ++++++++++++++++++ plans/synthetic-tape-redesign.html | 620 +++++++++++++ 3 files changed, 1437 insertions(+) create mode 100644 plans/synthetic-tape-redesign-impeccable.html create mode 100644 plans/synthetic-tape-redesign.html diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index b6f4b0b..6439063 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -8,6 +8,7 @@ {"_type":"issue","id":"islandflow-ayo","title":"Drop stale backlog events from live fanout","description":"Follow-up to live freshness rollout: /ws/live was still fanning out stale backlog events for freshness-gated channels, which kept tape panes in Live feed behind despite active synthetic ingest. Gate fanout and cache ingest by freshness for options/nbbo/equities/flow.","status":"closed","priority":1,"issue_type":"bug","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T21:26:39Z","created_by":"dirtydishes","updated_at":"2026-04-28T21:26:44Z","started_at":"2026-04-28T21:26:44Z","closed_at":"2026-04-28T21:26:44Z","close_reason":"Completed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-0v6","title":"Fix tape freshness, NBBO coverage, pause controls, and filter popup","description":"Implement the tape fixes requested for synthetic options notional sizing, strict live freshness, live-mode pause/resume behavior, stronger NBBO snapshot coverage, and moving flow filters behind a popup. Includes server-side live cache changes, web terminal state/UI changes, and tests for synthetic pricing, live snapshot freshness/NBBO retention, and live pause/filter interactions.","status":"closed","priority":1,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T21:02:52Z","created_by":"dirtydishes","updated_at":"2026-04-28T21:13:38Z","started_at":"2026-04-28T21:02:57Z","closed_at":"2026-04-28T21:13:38Z","close_reason":"Completed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-e4r","title":"Implement smart-money flow filtering and synthetic firehose modes","description":"Implement the approved multi-surface plan for named synthetic market profiles, options raw-vs-signal filtering, live/API filter contracts, Tape page client-side flow filters, firehose-readiness improvements, tests, and README updates.","status":"closed","priority":1,"issue_type":"feature","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T20:10:49Z","created_by":"dirtydishes","updated_at":"2026-04-28T20:29:29Z","started_at":"2026-04-28T20:10:53Z","closed_at":"2026-04-28T20:29:29Z","close_reason":"Implemented synthetic market profiles, options signal-path filtering, signal-aware API/replay contracts, Tape page filters, tests, and README updates. Follow-up tracked in islandflow-biq.","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"islandflow-a50","title":"Add HTML plan docs for synthetic tape redesign","description":"Create two HTML planning docs under plans/: one straightforward end-user readable version and one more polished impeccable-style version, both covering the hosted synthetic tape redesign with summary, scope, affected services, UI notes, rollout, tests, and the full detailed implementation plan.\n","status":"closed","priority":2,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-14T02:47:44Z","created_by":"dirtydishes","updated_at":"2026-05-14T02:53:11Z","started_at":"2026-05-14T02:47:48Z","closed_at":"2026-05-14T02:53:11Z","close_reason":"Completed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-932","title":"Desktop follow-up native features","description":"Track deferred native desktop features after the thin hosted-wrapper v1 lands: notifications, keyboard shortcuts, local preferences storage, remembered window state, signed/notarized macOS distribution, auto-update evaluation, and optional local frontend bundling.\n","status":"open","priority":2,"issue_type":"task","owner":"dishes@dpdrm.com","created_at":"2026-05-13T13:20:12Z","created_by":"dirtydishes","updated_at":"2026-05-13T13:20:12Z","dependencies":[{"issue_id":"islandflow-932","depends_on_id":"islandflow-9ug","type":"discovered-from","created_at":"2026-05-13T09:20:12Z","created_by":"dirtydishes","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-vbk","title":"Remove deprecated Alpaca key-pair auth","description":"Remove legacy Alpaca key-pair authentication support and keep ALPACA_API_KEY as the only supported auth method across options/equities ingest and docs.\n","status":"closed","priority":2,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-05T07:19:51Z","created_by":"dirtydishes","updated_at":"2026-05-05T07:21:10Z","started_at":"2026-05-05T07:19:54Z","closed_at":"2026-05-05T07:21:10Z","close_reason":"Removed key-pair auth and kept ALPACA_API_KEY only","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-h47","title":"Support single-token Alpaca auth","description":"Support single-token Alpaca authentication across ingest adapters using ALPACA_API_KEY with fallback to ALPACA_KEY_ID/ALPACA_SECRET_KEY, and document env usage.\n","status":"closed","priority":2,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-05T07:12:22Z","created_by":"dirtydishes","updated_at":"2026-05-05T07:13:54Z","started_at":"2026-05-05T07:12:25Z","closed_at":"2026-05-05T07:13:54Z","close_reason":"Added ALPACA_API_KEY support with key-pair fallback","dependency_count":0,"dependent_count":0,"comment_count":0} diff --git a/plans/synthetic-tape-redesign-impeccable.html b/plans/synthetic-tape-redesign-impeccable.html new file mode 100644 index 0000000..ddf621c --- /dev/null +++ b/plans/synthetic-tape-redesign-impeccable.html @@ -0,0 +1,816 @@ + + + + + + Hosted Synthetic Tape Redesign · Impeccable Version + + + +
+
+
+ Plan · Impeccable HTML Version + Internal Control Surface + Hosted Synthetic Backend +
+
+
+

Make the tape feel alive, not scheduled.

+

+ The hosted synthetic market already reaches the smart-money categories, but it often reaches them too cleanly. This redesign makes the demo feel more like a market session: one shared regime drives options, equities, quotes, event context, and coverage pressure, while operators keep a compact internal handle on the simulator through a bottom-right gear. +

+
+ Public smart-money taxonomy stays stable + Cross-asset coupling is the first priority + Internal-only controls, not a public settings page +
+
+
+
+ What changes for a viewer + The tape looks less templated, more coherent, and more educational because the surrounding market conditions finally support the category hits. +
+
+ What changes for an operator + A small bottom-right gear opens a non-modal synthetic-control drawer for the hosted backend. +
+
+ What does not change + Existing public smart-money event types, endpoints, and surface labels remain intact. +
+
+
+
+ +
+ + +
+
+

Simplified Overview

+

+ The current synthetic system is strong at coverage and weaker at realism. It can produce the categories, but the tape often reveals the machinery: option bursts appear on a rhythm, quotes are too consistently clean, equities and options are only loosely related, and the market context around a labeled event is not convincing enough. +

+

+ The redesign introduces a shared market regime engine. Instead of “emit a category-shaped burst now,” the system will model a believable session state first, then let both options and equities express that state. That keeps the smart-money demo behavior while making the experience feel more grounded. +

+
+
+ Audience + Internal operators and demo owners + This is about making hosted synthetic sessions more convincing during demos and product evaluation. +
+
+ Primary outcome + Higher realism with preserved category coverage + The demo should still surface every smart-money category, but not in a visibly scripted way. +
+
+ UI entry point + Bottom-right gear, not a settings page + The operator control surface should stay compact and contextual. +
+
+ Compatibility + Public contracts remain stable + No public API break for smart-money consumers. +
+
+
+ +
+

Scope

+
+
+

In scope

+
    +
  • Hosted synthetic regime engine
    • +
  • Options and equities generator redesign
    • +
  • Hidden subtype scenario families
    • +
  • Soft coverage logic
    • +
  • Internal control API
    • +
  • Internal control drawer in the terminal shell
    • +
  • Regression and realism tests
    • +
+
+
+

Out of scope

+
    +
  • Changing public smart-money categories
    • +
  • General settings-page work
    • +
  • User profile or token-spend UI
    • +
  • Public simulator controls
    • +
  • Live-feed product changes
    • +
+
+
+
+ +
+

Services Affected

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
AreaFilesWhy they change
Shared types and regime modelpackages/types/src/synthetic-market.ts, packages/types/src/events.tsIntroduce the control-state model and the deterministic shared market regime.
Hosted APIservices/api/src/index.tsAdd internal synthetic-control status and mutation endpoints.
Options ingestservices/ingest-options/src/index.ts, services/ingest-options/src/adapters/synthetic.tsSwap burst scheduling for regime-driven scenario selection and coverage debt.
Equities ingestservices/ingest-equities/src/index.ts, services/ingest-equities/src/adapters/synthetic.tsMake equity prints and quotes react to the same latent regime as the options side.
Web and Electron shellapps/web/app/terminal.tsx, apps/web/app/api/admin/synthetic/*Add the internal-only gear trigger, drawer, and secure proxy layer.
TestsOptions tests, API tests, web testsProtect determinism, realism, UI visibility rules, and classification alignment.
+
+ +
+

Locked Decisions

+
+ Keep the six public smart-money categories + Add hidden subtype families + Use soft coverage guarantees + Prioritize cross-asset coupling first + Target the hosted synthetic backend + Internal-only control surface + No general settings page in this effort + Bottom-right gear opens a drawer +
+
+ +
+

Full Architecture

+ +

1. Replace the burst pulse with a shared regime engine

+

Expand packages/types/src/synthetic-market.ts into the shared deterministic engine used by both ingest services.

+

Shared functions:

+
    +
  • getSyntheticSessionState(ts, control)
    • +
  • getSyntheticUnderlyingState(symbol, ts, control, sessionState)
    • +
  • getSyntheticScenarioWeights(symbol, ts, control, sessionState)
    • +
  • getSyntheticCoverageBoost(profileId, coverageState, control)
    • +
+

sessionState includes:

+
    +
  • session_phase: open | midday | power_hour | after_event
    • +
  • regime: trend_up | trend_down | mean_revert | retail_chase | event_ramp | dealer_gamma | arb_calm
    • +
  • volatility_level
    • +
  • liquidity_level
    • +
  • quote_cleanliness
    • +
  • focus_symbols
    • +
  • event_symbols
    • +
  • seed_bucket
    • +
+ +

2. Add hosted synthetic control state

+

Add internal control schemas in packages/types:

+
    +
  • SyntheticControlPresetId
    • +
  • SyntheticControlState
    • +
  • SyntheticProfileWeightMap
    • +
  • SyntheticCoverageConfig
    • +
  • SyntheticDerivedStatus
    • +
+ 
type SyntheticControlState = {
+  preset_id: "balanced_demo" | "event_day" | "dealer_day" | "retail_chase" | "quiet_range";
+  coverage_assist: boolean;
+  coverage_window_minutes: 10 | 20 | 30;
+  shared_seed: number;
+  profile_weights: {
+    institutional_directional: 0.6 | 1.0 | 1.6;
+    retail_whale: 0.6 | 1.0 | 1.6;
+    event_driven: 0.6 | 1.0 | 1.6;
+    vol_seller: 0.6 | 1.0 | 1.6;
+    arbitrage: 0.6 | 1.0 | 1.6;
+    hedge_reactive: 0.6 | 1.0 | 1.6;
+  };
+  updated_at: number;
+  updated_by: string;
+};
+

Defaults: preset_id: balanced_demo, coverage_assist: true, coverage_window_minutes: 20, all profile weights 1.0.

+ +

3. Persist and distribute control state through NATS

+
    +
  • Use JetStream KV bucket synthetic_control
    • +
  • Use key global
    • +
  • services/api reads and writes the KV entry
    • +
  • services/ingest-options loads on boot and watches for updates
    • +
  • services/ingest-equities does the same
    • +
+ +

4. Rebuild options scenarios as hidden subtype families

+
    +
  • institutional_directional: call_sweep, put_sweep, ask_lift_accumulation, far_dated_conviction
    • +
  • retail_whale: 0dte_call_chase, short_dated_put_panic, attention_contract_spike
    • +
  • event_driven: earnings_vol_probe, pre_event_directional_ramp, post_gap_followthrough
    • +
  • vol_seller: covered_call_overwrite, cash_secured_put_write, short_straddle_harvest
    • +
  • arbitrage: parity_vertical, conversion_reversal, box_spread
    • +
  • hedge_reactive: gamma_pinch_call_hedge, reactive_put_wall, dealer_unwind
    • +
  • neutral_noise: single_print_mid, two_sided_scalp, stale_quote_noise
    • +
+

Hidden subtype labels remain internal and test-only. They should never appear on emitted option prints or public smart-money events.

+ +

5. Make equities and options react to the same latent state

+
+
+

Equities changes

+
    +
  • Remove the fixed dark-sequence loop
    • +
  • Make lit versus dark balance regime-dependent
    • +
  • Make spread, quote cleanliness, off-exchange frequency, and clustering regime-dependent
    • +
  • Use shared focus symbols
    • +
  • Make event_ramp and retail_chase show modest trend and wider quotes
    • +
  • Make dealer_gamma show choppier reversals and denser quote changes
    • +
  • Make arb_calm quieter and more neutral
    • +
+
+
+

Options changes

+
    +
  • Replace hardcoded coverage forcing with weighted family selection plus coverage debt
    • +
  • Make venue count, placement, stale or missing quote probability, and structure prevalence regime-sensitive
    • +
  • Derive execution_iv_shock, underlying_move_bps, and nbbo_spread_z from shared state
    • +
  • Generate event-driven timestamps and symbols from shared regime state
    • +
+
+
+ +

6. Add soft coverage accounting

+
    +
  • Track rolling coverage debt per public profile inside each ingest service
    • +
  • Maintain a rolling counter across the selected coverage_window_minutes
    • +
  • Only public profiles count toward coverage
    • +
  • Missing profiles get a bounded weight boost
    • +
  • Noise and low-key scenarios continue to appear between labeled bursts
    • +
+ +

7. Add internal hosted control endpoints

+

Add routes in services/api/src/index.ts:

+
    +
  • GET /admin/synthetic/status
    • +
  • GET /admin/synthetic/control
    • +
  • PUT /admin/synthetic/control
    • +
+ 
{
+  enabled: boolean;
+  backend_mode: "synthetic" | "mixed" | "live";
+  adapters: {
+    options: string;
+    equities: string;
+  };
+  control: SyntheticControlState | null;
+  derived: {
+    session_phase: string;
+    regime: string;
+    focus_symbols: string[];
+    profile_hit_counts: Record<SmartMoneyProfileId, number>;
+    coverage_window_minutes: number;
+  } | null;
+  disabled_reason?: string;
+}
+

Behavior: return 404 when admin mode is disabled, return 409 when hosted adapters are not synthetic, validate full payloads on PUT, and keep public smart-money interfaces unchanged.

+ +

8. Keep secrets out of the browser with Next.js proxy routes

+
    +
  • apps/web/app/api/admin/synthetic/status/route.ts
    • +
  • apps/web/app/api/admin/synthetic/control/route.ts
    • +
+

The proxy reads server-only SYNTHETIC_ADMIN_TOKEN, forwards to NEXT_PUBLIC_API_URL, returns 404 when the internal UI flag is off, and never exposes the token client-side.

+ +

9. Add an internal control surface

+

UI rules for the first pass:

+
    +
  • Small floating gear in the bottom-right corner
    • +
  • Opens a right-edge non-modal drawer
    • +
  • Internal-only visibility
    • +
  • Preset dropdown: Balanced Demo, Event Day, Dealer Day, Retail Chase, Quiet Range
    • +
  • Coverage assist toggle
    • +
  • Coverage window selector: 10m, 20m, 30m
    • +
  • Six profile-weight controls: Low, Normal, High
    • +
  • Read-only live status: regime, session phase, focus symbols, rolling hit counts, backend state
    • +
  • Optimistic updates with rollback on error
    • +
  • Debounced writes at 250ms
    • +
  • Status polling every 5s, no admin websocket in v1
    • +
+
+ +
+

Interfaces and Environment

+
+
+

Public contracts unchanged

+
    +
  • SmartMoneyProfileId
    • +
  • SmartMoneyEvent
    • +
  • /flow/smart-money
    • +
  • /history/smart-money
    • +
  • /replay/smart-money
    • +
  • /ws/smart-money
    • +
+
+
+

New internal contracts

+
    +
  • SyntheticControlState
    • +
  • SyntheticControlPresetId
    • +
  • SyntheticDerivedStatus
    • +
+
+
+
+
+

New internal endpoints

+
    +
  • GET /admin/synthetic/status
    • +
  • GET /admin/synthetic/control
    • +
  • PUT /admin/synthetic/control
    • +
+
+
+

New env vars

+

Backend

+
    +
  • SYNTHETIC_CONTROL_ENABLED=0|1
    • +
  • SYNTHETIC_ADMIN_TOKEN=...
    • +
+

Web

+
    +
  • NEXT_PUBLIC_SYNTHETIC_ADMIN=0|1
    • +
  • SYNTHETIC_ADMIN_TOKEN=... for the Next server proxy only
    • +
+
+
+
+ +
+

Implementation Phases

+
    +
  1. + Phase 1. Shared types and regime engine +

    Touch packages/types/src/synthetic-market.ts and related exports and tests. Deliver control schemas, preset definitions, deterministic session and regime functions, and coverage boost helpers.

    +
    2. +
  2. + Phase 2. Hosted control plane +

    Touch services/api/src/index.ts and NATS or KV helpers as needed. Deliver admin endpoints, KV persistence, status payloads, and disabled or error behavior.

    +
    3. +
  3. + Phase 3. Ingest service coupling +

    Touch both ingest services and their synthetic adapters. Deliver boot-time control loading, KV watch updates, shared regime-driven generation, and removal of visibly scripted fixed sequences.

    +
    4. +
  4. + Phase 4. Internal control UI +

    Touch apps/web/app/terminal.tsx and the internal admin proxy routes. Deliver the floating gear, non-modal drawer, polling, optimistic updates, and disabled state.

    +
    5. +
  5. + Phase 5. Regression and realism tests +

    Deliver determinism tests, control API tests, scenario coverage tests, UI visibility tests, and classifier-alignment tests for hidden subtype families.

    +
    6. +
+
+ +
+

Tests and Acceptance

+
+
+

Shared engine

+
    +
  • Same timestamp + control snapshot + seed yields the same regime and focus symbols in both ingest services.
    • +
  • Presets materially change regime weights without breaking determinism.
    • +
  • balanced_demo yields mixed regimes over a session.
    • +
  • quiet_range yields lower volatility, tighter spreads, and fewer labeled events than retail_chase.
    • +
+
+
+

Cross-asset coupling

+
    +
  • event_ramp produces event-aligned option scenarios and synchronized underlying drift and spread behavior.
    • +
  • dealer_gamma produces short-dated ATM-heavy options plus choppier underlying reversals.
    • +
  • arb_calm increases neutral multi-leg structures without strong directional underlying moves.
    • +
  • retail_chase increases short-dated OTM call behavior, IV shock, and louder underlying momentum.
    • +
+
+
+
+
+

Coverage and classification

+
    +
  • With default controls, every public smart-money profile appears at least once in a 20-minute synthetic session sample.
    • +
  • With coverage_assist=false, there is no forced coverage logic.
    • +
  • Raising one profile to High increases its frequency without starving other categories.
    • +
  • Neutral noise remains below the smart-money emission threshold.
    • +
  • Each hidden subtype family still classifies into the intended public profile.
    • +
+
+
+

Admin API and UI

+
    +
  • Disabled admin mode returns 404.
    • +
  • Non-synthetic hosted mode returns 409 with a useful reason.
    • +
  • Valid PUT persists to KV and becomes visible to both ingest services.
    • +
  • The floating gear is hidden when NEXT_PUBLIC_SYNTHETIC_ADMIN is off.
    • +
  • The browser client never receives the backend admin token.
    • +
+
+
+
+ +
+

Assumptions and Defaults

+
    +
  • Hosted synthetic control applies only when both options and equities ingest adapters are synthetic.
    • +
  • No general settings page, user-info work, or token-spend work is in scope here.
    • +
  • Hidden subtype labels remain internal and test-only and never attach to emitted prints.
    • +
  • The first pass uses polling for admin status rather than a new admin websocket.
    • +
  • The default operator experience is Balanced Demo with soft coverage on and a 20-minute window.
    • +
  • The repo currently lacks local PRODUCT.md, DESIGN.md, and the local impeccable loader path. This version therefore follows the spirit of the terminal shell and impeccable product-UI principles rather than project-specific design-context files.
    • +
+
+
+
+
+ + diff --git a/plans/synthetic-tape-redesign.html b/plans/synthetic-tape-redesign.html new file mode 100644 index 0000000..b8548d0 --- /dev/null +++ b/plans/synthetic-tape-redesign.html @@ -0,0 +1,620 @@ + + + + + + Hosted Synthetic Tape Redesign Plan + + + +
+
+
Plan · Standard HTML Version
+

Hosted Synthetic Tape Redesign

+

+ This plan redesigns the hosted synthetic market so the tape feels more like a real market session while still surfacing all six smart-money categories during a demo window. It keeps the public labels stable, adds richer hidden scenario families underneath them, and introduces an internal control surface for shaping the hosted simulator. +

+
+
+ Main outcome + More believable synthetic options, equities, quotes, and smart-money events, with softer coverage guarantees and stronger cross-asset coupling. +
+
+ User-facing change + An internal-only bottom-right gear opens a compact synthetic-control drawer for operators. +
+
+ Public API impact + No change to existing smart-money event types or public smart-money endpoints. +
+
+ Why it matters + The current tape reaches the categories, but it looks too templated and too clean in ways that weaken the demo. +
+
+
+ +
+

Simplified Summary

+

+ Today the simulator does the important part mechanically: it hits the categories. The problem is that the surrounding market behavior does not always look convincing. Options bursts, equity prints, quote quality, and event timing can feel loosely stitched together instead of driven by one believable market state. +

+

+ The redesign fixes that by introducing a shared regime engine. Both synthetic options and synthetic equities will respond to the same session conditions, such as event ramps, dealer-gamma chop, retail chase, quiet range trading, and neutral arbitrage-heavy periods. The result should be a tape that still teaches the product, but no longer feels obviously scripted. +

+
+ The public smart-money taxonomy stays the same: institutional_directional, retail_whale, event_driven, vol_seller, arbitrage, and hedge_reactive. +
+
+ +
+

Scope

+
+
+ In scope +
    +
  • Hosted synthetic market regime engine
    • +
  • Options and equities synthetic generator redesign
    • +
  • Hidden subtype scenario families
    • +
  • Soft coverage logic
    • +
  • Internal control API and UI
    • +
  • Documentation and tests for the new behavior
    • +
+
+
+ Out of scope +
    +
  • Changing public smart-money profile IDs
    • +
  • General settings page work
    • +
  • User profile or token-spend features
    • +
  • Live market feed changes
    • +
  • Public-facing simulator controls
    • +
+
+
+
+ +
+

Services Affected

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
AreaPrimary filesRole in the redesign
Shared typespackages/types/src/synthetic-market.ts, packages/types/src/events.tsAdd the shared regime model and internal synthetic-control schemas.
Hosted APIservices/api/src/index.tsAdd internal control endpoints and expose hosted simulator status.
Options ingestservices/ingest-options/src/index.ts, services/ingest-options/src/adapters/synthetic.tsAdopt the new regime engine, scenario families, and soft coverage logic.
Equities ingestservices/ingest-equities/src/index.ts, services/ingest-equities/src/adapters/synthetic.tsSynchronize synthetic quotes and prints with the same latent market regime.
Web and Electron shellapps/web/app/terminal.tsx, apps/web/app/api/admin/synthetic/*Add the internal gear-triggered control drawer and server-side proxy routes.
Testsservices/ingest-options/tests/synthetic.test.ts, web tests, API testsProtect classification alignment, determinism, coverage behavior, and control-plane behavior.
+
+ +
+

Full Plan Contents

+
    +
  1. Product decisions locked
    2. +
  2. Architecture
    3. +
  3. Public and internal interfaces
    4. +
  4. Implementation sequence
    5. +
  5. Test cases and scenarios
    6. +
  6. Assumptions and defaults
    7. +
+
+ +
+

Product Decisions Locked

+
    +
  • Keep the current six public smart-money categories.
    • +
  • Add richer hidden sub-scenarios beneath them.
    • +
  • Use soft coverage guarantees, not hard forced sequencing.
    • +
  • Prioritize cross-asset coupling first.
    • +
  • Controls affect the hosted synthetic backend.
    • +
  • Controls are internal-only.
    • +
  • Do not build a general settings page, user-info work, or token-spend work in this effort.
    • +
  • Use a bottom-right gear that opens a synthetic-control drawer.
    • +
+
+ +
+

Architecture

+ +

1. Replace the simple burst pulse with a shared regime engine

+

Expand packages/types/src/synthetic-market.ts into the shared deterministic market-state engine used by both ingest services.

+

Shared functions:

+
    +
  • getSyntheticSessionState(ts, control)
    • +
  • getSyntheticUnderlyingState(symbol, ts, control, sessionState)
    • +
  • getSyntheticScenarioWeights(symbol, ts, control, sessionState)
    • +
  • getSyntheticCoverageBoost(profileId, coverageState, control)
    • +
+

sessionState includes:

+
    +
  • session_phase: open | midday | power_hour | after_event
    • +
  • regime: trend_up | trend_down | mean_revert | retail_chase | event_ramp | dealer_gamma | arb_calm
    • +
  • volatility_level
    • +
  • liquidity_level
    • +
  • quote_cleanliness
    • +
  • focus_symbols
    • +
  • event_symbols
    • +
  • seed_bucket
    • +
+ +

2. Add hosted synthetic control state

+

Add internal control schemas in packages/types:

+
    +
  • SyntheticControlPresetId
    • +
  • SyntheticControlState
    • +
  • SyntheticProfileWeightMap
    • +
  • SyntheticCoverageConfig
    • +
  • SyntheticDerivedStatus
    • +
+ 
type SyntheticControlState = {
+  preset_id: "balanced_demo" | "event_day" | "dealer_day" | "retail_chase" | "quiet_range";
+  coverage_assist: boolean;
+  coverage_window_minutes: 10 | 20 | 30;
+  shared_seed: number;
+  profile_weights: {
+    institutional_directional: 0.6 | 1.0 | 1.6;
+    retail_whale: 0.6 | 1.0 | 1.6;
+    event_driven: 0.6 | 1.0 | 1.6;
+    vol_seller: 0.6 | 1.0 | 1.6;
+    arbitrage: 0.6 | 1.0 | 1.6;
+    hedge_reactive: 0.6 | 1.0 | 1.6;
+  };
+  updated_at: number;
+  updated_by: string;
+};
+

Defaults:

+
    +
  • preset_id: balanced_demo
    • +
  • coverage_assist: true
    • +
  • coverage_window_minutes: 20
    • +
  • All profile weights 1.0
    • +
+ +

3. Persist and distribute control state through NATS

+
    +
  • Use JetStream KV bucket synthetic_control
    • +
  • Use key global
    • +
  • services/api reads and writes the KV entry
    • +
  • services/ingest-options loads on boot and watches for updates
    • +
  • services/ingest-equities does the same
    • +
+ +

4. Rebuild options scenarios as hidden subtype families

+

Keep public profiles the same, but generate them through richer hidden subtype families.

+
    +
  • institutional_directional: call_sweep, put_sweep, ask_lift_accumulation, far_dated_conviction
    • +
  • retail_whale: 0dte_call_chase, short_dated_put_panic, attention_contract_spike
    • +
  • event_driven: earnings_vol_probe, pre_event_directional_ramp, post_gap_followthrough
    • +
  • vol_seller: covered_call_overwrite, cash_secured_put_write, short_straddle_harvest
    • +
  • arbitrage: parity_vertical, conversion_reversal, box_spread
    • +
  • hedge_reactive: gamma_pinch_call_hedge, reactive_put_wall, dealer_unwind
    • +
  • neutral_noise: single_print_mid, two_sided_scalp, stale_quote_noise
    • +
+ +

5. Make equities and options react to the same latent state

+

Equities changes:

+
    +
  • Remove the fixed dark-sequence loop
    • +
  • Make lit versus dark balance regime-dependent
    • +
  • Make spread, quote cleanliness, off-exchange frequency, and clustering regime-dependent
    • +
  • Use shared focus symbols
    • +
  • During event_ramp and retail_chase, create modest trend and wider quotes
    • +
  • During dealer_gamma, create choppier reversals and denser quote changes
    • +
  • During arb_calm, create quieter underlying motion and more neutral execution context
    • +
+

Options changes:

+
    +
  • Replace hardcoded coverage forcing with weighted family selection plus coverage debt
    • +
  • Make venue count, placement, stale or missing quote probability, and structure prevalence regime-sensitive
    • +
  • Derive execution_iv_shock, underlying_move_bps, and nbbo_spread_z from shared state
    • +
  • Generate event-driven timestamps and symbols from shared regime state
    • +
+ +

6. Add soft coverage accounting

+
    +
  • Track rolling coverage debt per public profile inside each ingest service
    • +
  • Maintain a rolling counter across the selected coverage_window_minutes
    • +
  • Only public profiles count toward coverage
    • +
  • Missing profiles get a bounded weight boost
    • +
  • Noise and low-key scenarios continue to appear between labeled bursts
    • +
+ +

7. Add internal hosted control endpoints

+

Add routes in services/api/src/index.ts:

+
    +
  • GET /admin/synthetic/status
    • +
  • GET /admin/synthetic/control
    • +
  • PUT /admin/synthetic/control
    • +
+ 
{
+  enabled: boolean;
+  backend_mode: "synthetic" | "mixed" | "live";
+  adapters: {
+    options: string;
+    equities: string;
+  };
+  control: SyntheticControlState | null;
+  derived: {
+    session_phase: string;
+    regime: string;
+    focus_symbols: string[];
+    profile_hit_counts: Record<SmartMoneyProfileId, number>;
+    coverage_window_minutes: number;
+  } | null;
+  disabled_reason?: string;
+}
+

Behavior:

+
    +
  • Return 404 when admin mode is disabled
    • +
  • Return 409 when hosted adapters are not synthetic
    • +
  • Validate full payloads on PUT
    • +
  • Keep all existing public smart-money, history, replay, and websocket endpoints unchanged
    • +
+ +

8. Keep secrets out of the browser with Next.js proxy routes

+

Add server-side proxy routes:

+
    +
  • apps/web/app/api/admin/synthetic/status/route.ts
    • +
  • apps/web/app/api/admin/synthetic/control/route.ts
    • +
+

Proxy behavior:

+
    +
  • Read server-only SYNTHETIC_ADMIN_TOKEN
    • +
  • Forward to backend admin endpoints at NEXT_PUBLIC_API_URL
    • +
  • Return 404 when the internal UI flag is off
    • +
  • Never send the token to the browser
    • +
+ +

9. Add an internal control surface

+

UI surface:

+
    +
  • A small floating gear button in the bottom-right corner
    • +
  • Opens a right-edge non-modal drawer
    • +
  • Internal-only visibility
    • +
+

Drawer sections:

+
    +
  • Preset
    • +
  • Coverage
    • +
  • Profile Bias
    • +
  • Live Status
    • +
+

Controls:

+
    +
  • Preset dropdown: Balanced Demo, Event Day, Dealer Day, Retail Chase, Quiet Range
    • +
  • Coverage assist toggle
    • +
  • Coverage window selector: 10m, 20m, 30m
    • +
  • Six profile weight controls with Low, Normal, High
    • +
+
+ +
+

Public and Internal Interfaces

+

Public contracts unchanged

+
    +
  • SmartMoneyProfileId
    • +
  • SmartMoneyEvent
    • +
  • /flow/smart-money
    • +
  • /history/smart-money
    • +
  • /replay/smart-money
    • +
  • /ws/smart-money
    • +
+

New internal contracts

+
    +
  • SyntheticControlState
    • +
  • SyntheticControlPresetId
    • +
  • SyntheticDerivedStatus
    • +
+

New internal endpoints

+
    +
  • GET /admin/synthetic/status
    • +
  • GET /admin/synthetic/control
    • +
  • PUT /admin/synthetic/control
    • +
+

New environment variables

+

Backend:

+
    +
  • SYNTHETIC_CONTROL_ENABLED=0|1
    • +
  • SYNTHETIC_ADMIN_TOKEN=...
    • +
+

Web:

+
    +
  • NEXT_PUBLIC_SYNTHETIC_ADMIN=0|1
    • +
  • SYNTHETIC_ADMIN_TOKEN=... for the Next server proxy only
    • +
+
+ +
+

Implementation Sequence

+
    +
  1. + Phase 1. Shared types and regime engine +

    Touch packages/types/src/synthetic-market.ts and related exports and tests. Deliver control schemas, preset definitions, deterministic session and regime functions, and coverage boost helpers.

    +
    2. +
  2. + Phase 2. Hosted control plane +

    Touch services/api/src/index.ts and NATS or KV helpers as needed. Deliver admin endpoints, KV persistence, status payloads, and disabled or error behavior.

    +
    3. +
  3. + Phase 3. Ingest service coupling +

    Touch both ingest services and their synthetic adapters. Deliver boot-time control loading, KV watch updates, shared regime-driven generation, and removal of visibly scripted fixed sequences.

    +
    4. +
  4. + Phase 4. Internal control UI +

    Touch apps/web/app/terminal.tsx and the internal admin proxy routes. Deliver the floating gear, non-modal drawer, polling, optimistic updates, and disabled state.

    +
    5. +
  5. + Phase 5. Regression and realism tests +

    Deliver determinism tests, control API tests, scenario coverage tests, UI visibility tests, and classifier-alignment tests for hidden subtype families.

    +
    6. +
+
+ +
+

Test Cases and Scenarios

+

Shared engine

+
    +
  • Same timestamp + control snapshot + seed yields the same regime and focus symbols in both ingest services.
    • +
  • Presets materially change regime weights without breaking determinism.
    • +
  • balanced_demo yields mixed regimes over a session.
    • +
  • quiet_range yields lower volatility, tighter spreads, and fewer labeled events than retail_chase.
    • +
+

Cross-asset coupling

+
    +
  • event_ramp produces event-aligned option scenarios and synchronized underlying drift and spread behavior.
    • +
  • dealer_gamma produces short-dated ATM-heavy options plus choppier underlying reversals.
    • +
  • arb_calm increases neutral multi-leg structures without strong directional underlying moves.
    • +
  • retail_chase increases short-dated OTM call behavior, IV shock, and louder underlying momentum.
    • +
+

Coverage behavior

+
    +
  • With default controls, every public smart-money profile appears at least once in a 20-minute synthetic session sample.
    • +
  • With coverage_assist=false, there is no forced coverage logic.
    • +
  • Raising one profile to High increases its frequency without starving all other categories.
    • +
  • The quiet preset still emits noise and occasional signals rather than a dead tape.
    • +
+

Classification alignment

+
    +
  • Each hidden subtype family still classifies primarily into its intended public profile.
    • +
  • Neutral noise remains below the smart-money emission threshold.
    • +
  • Nearby wrong profiles stay below threshold in subtype template tests.
    • +
+

Admin API and UI

+
    +
  • Disabled admin mode returns 404.
    • +
  • Non-synthetic hosted mode returns 409 with a useful reason.
    • +
  • Valid PUT persists to KV and becomes visible to both ingest services.
    • +
  • The floating gear is hidden when NEXT_PUBLIC_SYNTHETIC_ADMIN is off.
    • +
  • The browser client never receives the backend admin token.
    • +
+
+ +
+

Assumptions and Defaults

+
    +
  • Hosted synthetic control applies only when both options and equities ingest adapters are synthetic.
    • +
  • No general settings page, user-info work, or token-spend work is in scope here.
    • +
  • Hidden subtype labels remain internal and test-only and never attach to emitted prints.
    • +
  • The first pass uses polling for admin status rather than a new admin websocket.
    • +
  • The default operator experience is Balanced Demo with soft coverage on and a 20-minute window.
    • +
  • The repo currently lacks local PRODUCT.md, DESIGN.md, and the local impeccable loader path, so UI implementation should use the existing terminal shell as the visual source of truth unless those design-context files are added later.
    • +
+
+
+ +