islandflow/plans/synthetic-tape-redesign.html

620 lines
26 KiB
HTML

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Hosted Synthetic Tape Redesign Plan</title>
<style>
:root {
color-scheme: dark;
--bg: #0f1216;
--panel: #151a20;
--panel-soft: #1b2129;
--border: #2a3440;
--text: #e4e9ef;
--muted: #a4afbc;
--accent: #69c3a5;
--accent-soft: rgba(105, 195, 165, 0.14);
--warn: #e0b36c;
--code: #10151a;
--shadow: 0 18px 44px rgba(0, 0, 0, 0.28);
}
* {
box-sizing: border-box;
}
body {
margin: 0;
font-family: "SF Pro Text", "Inter", "Segoe UI", sans-serif;
background:
radial-gradient(circle at top, rgba(105, 195, 165, 0.08), transparent 28rem),
linear-gradient(180deg, #0c0f13, var(--bg));
color: var(--text);
line-height: 1.55;
}
main {
width: min(1100px, calc(100vw - 2rem));
margin: 0 auto;
padding: 2rem 0 4rem;
}
header,
section {
background: rgba(21, 26, 32, 0.92);
border: 1px solid var(--border);
border-radius: 18px;
box-shadow: var(--shadow);
padding: 1.5rem;
margin-bottom: 1rem;
}
header {
padding: 1.8rem;
}
h1,
h2,
h3 {
margin: 0 0 0.75rem;
line-height: 1.15;
}
h1 {
font-size: clamp(2rem, 5vw, 3rem);
letter-spacing: -0.04em;
}
h2 {
font-size: 1.3rem;
color: var(--accent);
}
h3 {
font-size: 1rem;
color: #f1f5f9;
}
p,
li {
max-width: 76ch;
color: var(--text);
}
.lede {
font-size: 1.05rem;
color: #d7dde5;
}
.eyebrow {
display: inline-flex;
align-items: center;
gap: 0.45rem;
margin-bottom: 0.75rem;
padding: 0.35rem 0.75rem;
border-radius: 999px;
background: var(--accent-soft);
color: var(--accent);
font-size: 0.8rem;
font-weight: 700;
text-transform: uppercase;
letter-spacing: 0.08em;
}
.summary-grid,
.scope-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(220px, 1fr));
gap: 0.85rem;
margin-top: 1rem;
}
.summary-item,
.scope-item {
background: var(--panel-soft);
border: 1px solid var(--border);
border-radius: 14px;
padding: 1rem;
}
.summary-item strong,
.scope-item strong {
display: block;
margin-bottom: 0.35rem;
}
ul,
ol {
padding-left: 1.2rem;
}
code,
pre {
font-family: "SF Mono", "JetBrains Mono", "Cascadia Code", monospace;
}
pre {
overflow-x: auto;
background: var(--code);
border: 1px solid var(--border);
border-radius: 14px;
padding: 1rem;
}
table {
width: 100%;
border-collapse: collapse;
margin-top: 0.75rem;
}
th,
td {
border-bottom: 1px solid var(--border);
text-align: left;
vertical-align: top;
padding: 0.8rem 0.65rem;
}
th {
color: var(--accent);
font-size: 0.86rem;
text-transform: uppercase;
letter-spacing: 0.08em;
}
.muted {
color: var(--muted);
}
.callout {
padding: 0.95rem 1rem;
background: rgba(224, 179, 108, 0.12);
border: 1px solid rgba(224, 179, 108, 0.28);
border-radius: 14px;
color: #f6debb;
}
.toc a {
color: var(--accent);
text-decoration: none;
}
.toc li {
margin-bottom: 0.35rem;
}
.small {
font-size: 0.92rem;
}
</style>
</head>
<body>
<main>
<header>
<div class="eyebrow">Plan · Standard HTML Version</div>
<h1>Hosted Synthetic Tape Redesign</h1>
<p class="lede">
This plan redesigns the hosted synthetic market so the tape feels more like a real market session while still surfacing all six smart-money categories during a demo window. It keeps the public labels stable, adds richer hidden scenario families underneath them, and introduces an internal control surface for shaping the hosted simulator.
</p>
<div class="summary-grid">
<div class="summary-item">
<strong>Main outcome</strong>
<span>More believable synthetic options, equities, quotes, and smart-money events, with softer coverage guarantees and stronger cross-asset coupling.</span>
</div>
<div class="summary-item">
<strong>User-facing change</strong>
<span>An internal-only bottom-right gear opens a compact synthetic-control drawer for operators.</span>
</div>
<div class="summary-item">
<strong>Public API impact</strong>
<span>No change to existing smart-money event types or public smart-money endpoints.</span>
</div>
<div class="summary-item">
<strong>Why it matters</strong>
<span>The current tape reaches the categories, but it looks too templated and too clean in ways that weaken the demo.</span>
</div>
</div>
</header>
<section>
<h2>Simplified Summary</h2>
<p>
Today the simulator does the important part mechanically: it hits the categories. The problem is that the surrounding market behavior does not always look convincing. Options bursts, equity prints, quote quality, and event timing can feel loosely stitched together instead of driven by one believable market state.
</p>
<p>
The redesign fixes that by introducing a shared regime engine. Both synthetic options and synthetic equities will respond to the same session conditions, such as event ramps, dealer-gamma chop, retail chase, quiet range trading, and neutral arbitrage-heavy periods. The result should be a tape that still teaches the product, but no longer feels obviously scripted.
</p>
<div class="callout small">
The public smart-money taxonomy stays the same: <code>institutional_directional</code>, <code>retail_whale</code>, <code>event_driven</code>, <code>vol_seller</code>, <code>arbitrage</code>, and <code>hedge_reactive</code>.
</div>
</section>
<section>
<h2>Scope</h2>
<div class="scope-grid">
<div class="scope-item">
<strong>In scope</strong>
<ul>
<li>Hosted synthetic market regime engine</li>
<li>Options and equities synthetic generator redesign</li>
<li>Hidden subtype scenario families</li>
<li>Soft coverage logic</li>
<li>Internal control API and UI</li>
<li>Documentation and tests for the new behavior</li>
</ul>
</div>
<div class="scope-item">
<strong>Out of scope</strong>
<ul>
<li>Changing public smart-money profile IDs</li>
<li>General settings page work</li>
<li>User profile or token-spend features</li>
<li>Live market feed changes</li>
<li>Public-facing simulator controls</li>
</ul>
</div>
</div>
</section>
<section>
<h2>Services Affected</h2>
<table>
<thead>
<tr>
<th>Area</th>
<th>Primary files</th>
<th>Role in the redesign</th>
</tr>
</thead>
<tbody>
<tr>
<td>Shared types</td>
<td><code>packages/types/src/synthetic-market.ts</code>, <code>packages/types/src/events.ts</code></td>
<td>Add the shared regime model and internal synthetic-control schemas.</td>
</tr>
<tr>
<td>Hosted API</td>
<td><code>services/api/src/index.ts</code></td>
<td>Add internal control endpoints and expose hosted simulator status.</td>
</tr>
<tr>
<td>Options ingest</td>
<td><code>services/ingest-options/src/index.ts</code>, <code>services/ingest-options/src/adapters/synthetic.ts</code></td>
<td>Adopt the new regime engine, scenario families, and soft coverage logic.</td>
</tr>
<tr>
<td>Equities ingest</td>
<td><code>services/ingest-equities/src/index.ts</code>, <code>services/ingest-equities/src/adapters/synthetic.ts</code></td>
<td>Synchronize synthetic quotes and prints with the same latent market regime.</td>
</tr>
<tr>
<td>Web and Electron shell</td>
<td><code>apps/web/app/terminal.tsx</code>, <code>apps/web/app/api/admin/synthetic/*</code></td>
<td>Add the internal gear-triggered control drawer and server-side proxy routes.</td>
</tr>
<tr>
<td>Tests</td>
<td><code>services/ingest-options/tests/synthetic.test.ts</code>, web tests, API tests</td>
<td>Protect classification alignment, determinism, coverage behavior, and control-plane behavior.</td>
</tr>
</tbody>
</table>
</section>
<section class="toc">
<h2>Full Plan Contents</h2>
<ol>
<li><a href="#locked">Product decisions locked</a></li>
<li><a href="#architecture">Architecture</a></li>
<li><a href="#interfaces">Public and internal interfaces</a></li>
<li><a href="#sequence">Implementation sequence</a></li>
<li><a href="#tests">Test cases and scenarios</a></li>
<li><a href="#assumptions">Assumptions and defaults</a></li>
</ol>
</section>
<section id="locked">
<h2>Product Decisions Locked</h2>
<ul>
<li>Keep the current six public smart-money categories.</li>
<li>Add richer hidden sub-scenarios beneath them.</li>
<li>Use soft coverage guarantees, not hard forced sequencing.</li>
<li>Prioritize cross-asset coupling first.</li>
<li>Controls affect the hosted synthetic backend.</li>
<li>Controls are internal-only.</li>
<li>Do not build a general settings page, user-info work, or token-spend work in this effort.</li>
<li>Use a bottom-right gear that opens a synthetic-control drawer.</li>
</ul>
</section>
<section id="architecture">
<h2>Architecture</h2>
<h3>1. Replace the simple burst pulse with a shared regime engine</h3>
<p>Expand <code>packages/types/src/synthetic-market.ts</code> into the shared deterministic market-state engine used by both ingest services.</p>
<p>Shared functions:</p>
<ul>
<li><code>getSyntheticSessionState(ts, control)</code></li>
<li><code>getSyntheticUnderlyingState(symbol, ts, control, sessionState)</code></li>
<li><code>getSyntheticScenarioWeights(symbol, ts, control, sessionState)</code></li>
<li><code>getSyntheticCoverageBoost(profileId, coverageState, control)</code></li>
</ul>
<p><code>sessionState</code> includes:</p>
<ul>
<li><code>session_phase</code>: <code>open | midday | power_hour | after_event</code></li>
<li><code>regime</code>: <code>trend_up | trend_down | mean_revert | retail_chase | event_ramp | dealer_gamma | arb_calm</code></li>
<li><code>volatility_level</code></li>
<li><code>liquidity_level</code></li>
<li><code>quote_cleanliness</code></li>
<li><code>focus_symbols</code></li>
<li><code>event_symbols</code></li>
<li><code>seed_bucket</code></li>
</ul>
<h3>2. Add hosted synthetic control state</h3>
<p>Add internal control schemas in <code>packages/types</code>:</p>
<ul>
<li><code>SyntheticControlPresetId</code></li>
<li><code>SyntheticControlState</code></li>
<li><code>SyntheticProfileWeightMap</code></li>
<li><code>SyntheticCoverageConfig</code></li>
<li><code>SyntheticDerivedStatus</code></li>
</ul>
<pre><code>type SyntheticControlState = {
preset_id: "balanced_demo" | "event_day" | "dealer_day" | "retail_chase" | "quiet_range";
coverage_assist: boolean;
coverage_window_minutes: 10 | 20 | 30;
shared_seed: number;
profile_weights: {
institutional_directional: 0.6 | 1.0 | 1.6;
retail_whale: 0.6 | 1.0 | 1.6;
event_driven: 0.6 | 1.0 | 1.6;
vol_seller: 0.6 | 1.0 | 1.6;
arbitrage: 0.6 | 1.0 | 1.6;
hedge_reactive: 0.6 | 1.0 | 1.6;
};
updated_at: number;
updated_by: string;
};</code></pre>
<p>Defaults:</p>
<ul>
<li><code>preset_id: balanced_demo</code></li>
<li><code>coverage_assist: true</code></li>
<li><code>coverage_window_minutes: 20</code></li>
<li>All profile weights <code>1.0</code></li>
</ul>
<h3>3. Persist and distribute control state through NATS</h3>
<ul>
<li>Use JetStream KV bucket <code>synthetic_control</code></li>
<li>Use key <code>global</code></li>
<li><code>services/api</code> reads and writes the KV entry</li>
<li><code>services/ingest-options</code> loads on boot and watches for updates</li>
<li><code>services/ingest-equities</code> does the same</li>
</ul>
<h3>4. Rebuild options scenarios as hidden subtype families</h3>
<p>Keep public profiles the same, but generate them through richer hidden subtype families.</p>
<ul>
<li><strong><code>institutional_directional</code></strong>: <code>call_sweep</code>, <code>put_sweep</code>, <code>ask_lift_accumulation</code>, <code>far_dated_conviction</code></li>
<li><strong><code>retail_whale</code></strong>: <code>0dte_call_chase</code>, <code>short_dated_put_panic</code>, <code>attention_contract_spike</code></li>
<li><strong><code>event_driven</code></strong>: <code>earnings_vol_probe</code>, <code>pre_event_directional_ramp</code>, <code>post_gap_followthrough</code></li>
<li><strong><code>vol_seller</code></strong>: <code>covered_call_overwrite</code>, <code>cash_secured_put_write</code>, <code>short_straddle_harvest</code></li>
<li><strong><code>arbitrage</code></strong>: <code>parity_vertical</code>, <code>conversion_reversal</code>, <code>box_spread</code></li>
<li><strong><code>hedge_reactive</code></strong>: <code>gamma_pinch_call_hedge</code>, <code>reactive_put_wall</code>, <code>dealer_unwind</code></li>
<li><strong><code>neutral_noise</code></strong>: <code>single_print_mid</code>, <code>two_sided_scalp</code>, <code>stale_quote_noise</code></li>
</ul>
<h3>5. Make equities and options react to the same latent state</h3>
<p>Equities changes:</p>
<ul>
<li>Remove the fixed dark-sequence loop</li>
<li>Make lit versus dark balance regime-dependent</li>
<li>Make spread, quote cleanliness, off-exchange frequency, and clustering regime-dependent</li>
<li>Use shared focus symbols</li>
<li>During <code>event_ramp</code> and <code>retail_chase</code>, create modest trend and wider quotes</li>
<li>During <code>dealer_gamma</code>, create choppier reversals and denser quote changes</li>
<li>During <code>arb_calm</code>, create quieter underlying motion and more neutral execution context</li>
</ul>
<p>Options changes:</p>
<ul>
<li>Replace hardcoded coverage forcing with weighted family selection plus coverage debt</li>
<li>Make venue count, placement, stale or missing quote probability, and structure prevalence regime-sensitive</li>
<li>Derive <code>execution_iv_shock</code>, <code>underlying_move_bps</code>, and <code>nbbo_spread_z</code> from shared state</li>
<li>Generate event-driven timestamps and symbols from shared regime state</li>
</ul>
<h3>6. Add soft coverage accounting</h3>
<ul>
<li>Track rolling coverage debt per public profile inside each ingest service</li>
<li>Maintain a rolling counter across the selected <code>coverage_window_minutes</code></li>
<li>Only public profiles count toward coverage</li>
<li>Missing profiles get a bounded weight boost</li>
<li>Noise and low-key scenarios continue to appear between labeled bursts</li>
</ul>
<h3>7. Add internal hosted control endpoints</h3>
<p>Add routes in <code>services/api/src/index.ts</code>:</p>
<ul>
<li><code>GET /admin/synthetic/status</code></li>
<li><code>GET /admin/synthetic/control</code></li>
<li><code>PUT /admin/synthetic/control</code></li>
</ul>
<pre><code>{
enabled: boolean;
backend_mode: "synthetic" | "mixed" | "live";
adapters: {
options: string;
equities: string;
};
control: SyntheticControlState | null;
derived: {
session_phase: string;
regime: string;
focus_symbols: string[];
profile_hit_counts: Record&lt;SmartMoneyProfileId, number&gt;;
coverage_window_minutes: number;
} | null;
disabled_reason?: string;
}</code></pre>
<p>Behavior:</p>
<ul>
<li>Return <code>404</code> when admin mode is disabled</li>
<li>Return <code>409</code> when hosted adapters are not synthetic</li>
<li>Validate full payloads on <code>PUT</code></li>
<li>Keep all existing public smart-money, history, replay, and websocket endpoints unchanged</li>
</ul>
<h3>8. Keep secrets out of the browser with Next.js proxy routes</h3>
<p>Add server-side proxy routes:</p>
<ul>
<li><code>apps/web/app/api/admin/synthetic/status/route.ts</code></li>
<li><code>apps/web/app/api/admin/synthetic/control/route.ts</code></li>
</ul>
<p>Proxy behavior:</p>
<ul>
<li>Read server-only <code>SYNTHETIC_ADMIN_TOKEN</code></li>
<li>Forward to backend admin endpoints at <code>NEXT_PUBLIC_API_URL</code></li>
<li>Return <code>404</code> when the internal UI flag is off</li>
<li>Never send the token to the browser</li>
</ul>
<h3>9. Add an internal control surface</h3>
<p>UI surface:</p>
<ul>
<li>A small floating gear button in the bottom-right corner</li>
<li>Opens a right-edge non-modal drawer</li>
<li>Internal-only visibility</li>
</ul>
<p>Drawer sections:</p>
<ul>
<li><code>Preset</code></li>
<li><code>Coverage</code></li>
<li><code>Profile Bias</code></li>
<li><code>Live Status</code></li>
</ul>
<p>Controls:</p>
<ul>
<li>Preset dropdown: <code>Balanced Demo</code>, <code>Event Day</code>, <code>Dealer Day</code>, <code>Retail Chase</code>, <code>Quiet Range</code></li>
<li>Coverage assist toggle</li>
<li>Coverage window selector: <code>10m</code>, <code>20m</code>, <code>30m</code></li>
<li>Six profile weight controls with <code>Low</code>, <code>Normal</code>, <code>High</code></li>
</ul>
</section>
<section id="interfaces">
<h2>Public and Internal Interfaces</h2>
<h3>Public contracts unchanged</h3>
<ul>
<li><code>SmartMoneyProfileId</code></li>
<li><code>SmartMoneyEvent</code></li>
<li><code>/flow/smart-money</code></li>
<li><code>/history/smart-money</code></li>
<li><code>/replay/smart-money</code></li>
<li><code>/ws/smart-money</code></li>
</ul>
<h3>New internal contracts</h3>
<ul>
<li><code>SyntheticControlState</code></li>
<li><code>SyntheticControlPresetId</code></li>
<li><code>SyntheticDerivedStatus</code></li>
</ul>
<h3>New internal endpoints</h3>
<ul>
<li><code>GET /admin/synthetic/status</code></li>
<li><code>GET /admin/synthetic/control</code></li>
<li><code>PUT /admin/synthetic/control</code></li>
</ul>
<h3>New environment variables</h3>
<p>Backend:</p>
<ul>
<li><code>SYNTHETIC_CONTROL_ENABLED=0|1</code></li>
<li><code>SYNTHETIC_ADMIN_TOKEN=...</code></li>
</ul>
<p>Web:</p>
<ul>
<li><code>NEXT_PUBLIC_SYNTHETIC_ADMIN=0|1</code></li>
<li><code>SYNTHETIC_ADMIN_TOKEN=...</code> for the Next server proxy only</li>
</ul>
</section>
<section id="sequence">
<h2>Implementation Sequence</h2>
<ol>
<li>
<strong>Phase 1. Shared types and regime engine</strong>
<p class="muted small">Touch <code>packages/types/src/synthetic-market.ts</code> and related exports and tests. Deliver control schemas, preset definitions, deterministic session and regime functions, and coverage boost helpers.</p>
</li>
<li>
<strong>Phase 2. Hosted control plane</strong>
<p class="muted small">Touch <code>services/api/src/index.ts</code> and NATS or KV helpers as needed. Deliver admin endpoints, KV persistence, status payloads, and disabled or error behavior.</p>
</li>
<li>
<strong>Phase 3. Ingest service coupling</strong>
<p class="muted small">Touch both ingest services and their synthetic adapters. Deliver boot-time control loading, KV watch updates, shared regime-driven generation, and removal of visibly scripted fixed sequences.</p>
</li>
<li>
<strong>Phase 4. Internal control UI</strong>
<p class="muted small">Touch <code>apps/web/app/terminal.tsx</code> and the internal admin proxy routes. Deliver the floating gear, non-modal drawer, polling, optimistic updates, and disabled state.</p>
</li>
<li>
<strong>Phase 5. Regression and realism tests</strong>
<p class="muted small">Deliver determinism tests, control API tests, scenario coverage tests, UI visibility tests, and classifier-alignment tests for hidden subtype families.</p>
</li>
</ol>
</section>
<section id="tests">
<h2>Test Cases and Scenarios</h2>
<h3>Shared engine</h3>
<ul>
<li>Same <code>timestamp + control snapshot + seed</code> yields the same regime and focus symbols in both ingest services.</li>
<li>Presets materially change regime weights without breaking determinism.</li>
<li><code>balanced_demo</code> yields mixed regimes over a session.</li>
<li><code>quiet_range</code> yields lower volatility, tighter spreads, and fewer labeled events than <code>retail_chase</code>.</li>
</ul>
<h3>Cross-asset coupling</h3>
<ul>
<li><code>event_ramp</code> produces event-aligned option scenarios and synchronized underlying drift and spread behavior.</li>
<li><code>dealer_gamma</code> produces short-dated ATM-heavy options plus choppier underlying reversals.</li>
<li><code>arb_calm</code> increases neutral multi-leg structures without strong directional underlying moves.</li>
<li><code>retail_chase</code> increases short-dated OTM call behavior, IV shock, and louder underlying momentum.</li>
</ul>
<h3>Coverage behavior</h3>
<ul>
<li>With default controls, every public smart-money profile appears at least once in a 20-minute synthetic session sample.</li>
<li>With <code>coverage_assist=false</code>, there is no forced coverage logic.</li>
<li>Raising one profile to <code>High</code> increases its frequency without starving all other categories.</li>
<li>The quiet preset still emits noise and occasional signals rather than a dead tape.</li>
</ul>
<h3>Classification alignment</h3>
<ul>
<li>Each hidden subtype family still classifies primarily into its intended public profile.</li>
<li>Neutral noise remains below the smart-money emission threshold.</li>
<li>Nearby wrong profiles stay below threshold in subtype template tests.</li>
</ul>
<h3>Admin API and UI</h3>
<ul>
<li>Disabled admin mode returns <code>404</code>.</li>
<li>Non-synthetic hosted mode returns <code>409</code> with a useful reason.</li>
<li>Valid <code>PUT</code> persists to KV and becomes visible to both ingest services.</li>
<li>The floating gear is hidden when <code>NEXT_PUBLIC_SYNTHETIC_ADMIN</code> is off.</li>
<li>The browser client never receives the backend admin token.</li>
</ul>
</section>
<section id="assumptions">
<h2>Assumptions and Defaults</h2>
<ul>
<li>Hosted synthetic control applies only when both options and equities ingest adapters are synthetic.</li>
<li>No general settings page, user-info work, or token-spend work is in scope here.</li>
<li>Hidden subtype labels remain internal and test-only and never attach to emitted prints.</li>
<li>The first pass uses polling for admin status rather than a new admin websocket.</li>
<li>The default operator experience is <code>Balanced Demo</code> with soft coverage on and a 20-minute window.</li>
<li>The repo currently lacks local <code>PRODUCT.md</code>, <code>DESIGN.md</code>, and the local impeccable loader path, so UI implementation should use the existing terminal shell as the visual source of truth unless those design-context files are added later.</li>
</ul>
</section>
</main>
</body>
</html>