implement durable options tape history
This commit is contained in:
parent
e3940eb0a6
commit
bd60d0d5d5
9 changed files with 423 additions and 56 deletions
57
docs/clickhouse-reset-runbook.md
Normal file
57
docs/clickhouse-reset-runbook.md
Normal file
|
|
@ -0,0 +1,57 @@
|
|||
# ClickHouse Reset Runbook
|
||||
|
||||
This runbook is for deliberately wiping durable market-data history from ClickHouse in local development or on the VPS. It is destructive. Do not run these commands from application startup, deployment hooks, or unattended scripts.
|
||||
|
||||
## When To Use
|
||||
|
||||
Use this only when an operator has decided that existing option, equity, flow, and derived-event history should be discarded and rebuilt from fresh ingest.
|
||||
|
||||
Before running a reset:
|
||||
|
||||
- Confirm the target environment: local Docker or VPS Docker.
|
||||
- Confirm there is no active analysis depending on the existing history.
|
||||
- Take a backup if the data may be needed later.
|
||||
- Stop ingest and API services so new writes do not race the reset.
|
||||
|
||||
## Local Docker Reset
|
||||
|
||||
From the repository root:
|
||||
|
||||
```bash
|
||||
bun run dev:infra
|
||||
docker compose exec clickhouse clickhouse-client --query "SHOW TABLES"
|
||||
docker compose exec clickhouse clickhouse-client --query "TRUNCATE TABLE IF EXISTS option_prints"
|
||||
docker compose exec clickhouse clickhouse-client --query "TRUNCATE TABLE IF EXISTS option_nbbo"
|
||||
docker compose exec clickhouse clickhouse-client --query "TRUNCATE TABLE IF EXISTS equity_prints"
|
||||
docker compose exec clickhouse clickhouse-client --query "TRUNCATE TABLE IF EXISTS equity_quotes"
|
||||
docker compose exec clickhouse clickhouse-client --query "TRUNCATE TABLE IF EXISTS equity_print_joins"
|
||||
docker compose exec clickhouse clickhouse-client --query "TRUNCATE TABLE IF EXISTS flow_packets"
|
||||
docker compose exec clickhouse clickhouse-client --query "TRUNCATE TABLE IF EXISTS smart_money_events"
|
||||
docker compose exec clickhouse clickhouse-client --query "TRUNCATE TABLE IF EXISTS classifier_hits"
|
||||
docker compose exec clickhouse clickhouse-client --query "TRUNCATE TABLE IF EXISTS alerts"
|
||||
docker compose exec clickhouse clickhouse-client --query "TRUNCATE TABLE IF EXISTS inferred_dark_events"
|
||||
```
|
||||
|
||||
If the local compose project uses `deployment/docker/docker-compose.yml`, run the same commands with `docker compose -f deployment/docker/docker-compose.yml exec clickhouse ...`.
|
||||
|
||||
## VPS Docker Reset
|
||||
|
||||
On the VPS, first identify the active compose project and ClickHouse service:
|
||||
|
||||
```bash
|
||||
docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}"
|
||||
docker compose -f deployment/docker/docker-compose.yml ps
|
||||
```
|
||||
|
||||
Then stop writers and run the same `TRUNCATE TABLE IF EXISTS ...` commands against the active ClickHouse container. Prefer `docker compose exec clickhouse clickhouse-client --query "<query>"` when the compose project is healthy; otherwise use `docker exec <clickhouse-container> clickhouse-client --query "<query>"`.
|
||||
|
||||
## Verification
|
||||
|
||||
After the reset:
|
||||
|
||||
```bash
|
||||
docker compose exec clickhouse clickhouse-client --query "SELECT count() FROM option_prints"
|
||||
docker compose exec clickhouse clickhouse-client --query "SELECT count() FROM flow_packets"
|
||||
```
|
||||
|
||||
Restart ingest/API services through the normal dev or deployment path. The options tape should repopulate its 100-row hot head from new signal prints, and older rows should appear only after the scroll gate asks `/history/options` for ClickHouse-backed history.
|
||||
245
docs/turns/2026-05-16-1725-durable-options-tape-history.html
Normal file
245
docs/turns/2026-05-16-1725-durable-options-tape-history.html
Normal file
|
|
@ -0,0 +1,245 @@
|
|||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<title>Durable Options Tape History</title>
|
||||
<style>
|
||||
:root {
|
||||
color-scheme: dark;
|
||||
--bg: #06080b;
|
||||
--panel: #111820;
|
||||
--panel-2: #0d141b;
|
||||
--border: rgba(255, 255, 255, 0.1);
|
||||
--border-strong: rgba(245, 166, 35, 0.34);
|
||||
--text: #e6edf4;
|
||||
--muted: #90a0b2;
|
||||
--faint: #6e7b8c;
|
||||
--accent: #f5a623;
|
||||
--accent-soft: rgba(245, 166, 35, 0.14);
|
||||
--good: #25c17a;
|
||||
}
|
||||
|
||||
* {
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
body {
|
||||
margin: 0;
|
||||
background: linear-gradient(180deg, rgba(17, 24, 32, 0.92), rgba(6, 8, 11, 0.98) 320px), var(--bg);
|
||||
color: var(--text);
|
||||
font: 15px/1.55 "IBM Plex Sans", -apple-system, BlinkMacSystemFont, "Segoe UI", system-ui, sans-serif;
|
||||
}
|
||||
|
||||
main {
|
||||
max-width: 1040px;
|
||||
margin: 0 auto;
|
||||
padding: 40px 24px 56px;
|
||||
}
|
||||
|
||||
header {
|
||||
border-bottom: 1px solid var(--border);
|
||||
padding-bottom: 24px;
|
||||
}
|
||||
|
||||
.eyebrow,
|
||||
h2,
|
||||
.chip {
|
||||
font-family: "IBM Plex Mono", ui-monospace, SFMono-Regular, Menlo, monospace;
|
||||
letter-spacing: 0.1em;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
|
||||
.eyebrow {
|
||||
margin: 0 0 10px;
|
||||
color: var(--accent);
|
||||
font-size: 0.76rem;
|
||||
font-weight: 700;
|
||||
}
|
||||
|
||||
h1 {
|
||||
margin: 0;
|
||||
max-width: 760px;
|
||||
font-size: 2rem;
|
||||
line-height: 1.1;
|
||||
}
|
||||
|
||||
.summary {
|
||||
max-width: 820px;
|
||||
margin: 16px 0 0;
|
||||
color: var(--muted);
|
||||
font-size: 1rem;
|
||||
}
|
||||
|
||||
.meta {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 8px;
|
||||
margin-top: 18px;
|
||||
}
|
||||
|
||||
.chip {
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 999px;
|
||||
padding: 4px 9px;
|
||||
background: rgba(255, 255, 255, 0.04);
|
||||
color: var(--muted);
|
||||
font-size: 0.7rem;
|
||||
font-weight: 700;
|
||||
}
|
||||
|
||||
section {
|
||||
margin-top: 26px;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 8px;
|
||||
padding: 20px 22px;
|
||||
background: linear-gradient(180deg, rgba(17, 24, 32, 0.94), rgba(13, 20, 27, 0.94));
|
||||
}
|
||||
|
||||
section.summary-band {
|
||||
border-color: var(--border-strong);
|
||||
background: linear-gradient(180deg, rgba(245, 166, 35, 0.12), rgba(17, 24, 32, 0.92));
|
||||
}
|
||||
|
||||
h2 {
|
||||
margin: 0 0 12px;
|
||||
color: var(--text);
|
||||
font-size: 0.82rem;
|
||||
}
|
||||
|
||||
p,
|
||||
li {
|
||||
color: var(--muted);
|
||||
}
|
||||
|
||||
p {
|
||||
margin: 0;
|
||||
}
|
||||
|
||||
ul {
|
||||
margin: 0;
|
||||
padding-left: 1.1rem;
|
||||
}
|
||||
|
||||
li {
|
||||
margin: 8px 0;
|
||||
}
|
||||
|
||||
strong {
|
||||
color: var(--text);
|
||||
}
|
||||
|
||||
code {
|
||||
border: 1px solid rgba(255, 255, 255, 0.08);
|
||||
border-radius: 6px;
|
||||
padding: 0.12rem 0.35rem;
|
||||
background: rgba(255, 255, 255, 0.05);
|
||||
color: var(--text);
|
||||
font-family: "IBM Plex Mono", ui-monospace, SFMono-Regular, Menlo, monospace;
|
||||
font-size: 0.9em;
|
||||
}
|
||||
|
||||
.ok {
|
||||
color: var(--good);
|
||||
font-weight: 700;
|
||||
}
|
||||
|
||||
@media (max-width: 720px) {
|
||||
main {
|
||||
padding: 28px 16px 40px;
|
||||
}
|
||||
|
||||
h1 {
|
||||
font-size: 1.55rem;
|
||||
}
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<main>
|
||||
<header>
|
||||
<p class="eyebrow">Turn Document</p>
|
||||
<h1>Durable Options Tape History</h1>
|
||||
<p class="summary">
|
||||
Implemented the durable options tape plan: the live hot head is capped at 100 rows, older rows are preserved behind
|
||||
the scroll gate, ClickHouse history keeps execution context, and the Filter menu now exposes Signal versus All
|
||||
prints semantics.
|
||||
</p>
|
||||
<div class="meta" aria-label="Turn metadata">
|
||||
<span class="chip">2026-05-16 17:25</span>
|
||||
<span class="chip">Beads: islandflow-200</span>
|
||||
<span class="chip">Surface: Options Tape</span>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<section class="summary-band">
|
||||
<h2>Summary</h2>
|
||||
<p>
|
||||
The options tape now behaves as a continuous instrument: the live cache stays lean, historical rows arrive only
|
||||
when scrolling asks for them, and old valid rows are not treated as degraded just because they came from durable
|
||||
history.
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2>Changes Made</h2>
|
||||
<ul>
|
||||
<li>Changed the API default options live cache limit to <code>100</code>.</li>
|
||||
<li>Removed the unused scoped live auto-hydration path so history is loaded by the scroll gate.</li>
|
||||
<li>Fixed unbounded options/equities history retention so a cap of <code>0</code> means keep the loaded tail.</li>
|
||||
<li>Added a Filter menu <code>Options View</code> toggle for <code>Signal</code> and <code>All prints</code>.</li>
|
||||
<li>Ensured All prints clears signal-only side/type/min-notional/security constraints.</li>
|
||||
<li>Added a destructive ClickHouse reset runbook for local and VPS operators.</li>
|
||||
</ul>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2>Context</h2>
|
||||
<p>
|
||||
The prior plan called out useful partial work already in the repo: ClickHouse history endpoints, execution-context
|
||||
columns, scroll-hold behavior, and a shared row renderer. This implementation keeps those pieces and removes the
|
||||
ambiguous history/autohydration behavior around them.
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2>Important Implementation Details</h2>
|
||||
<ul>
|
||||
<li><code>/history/options</code> still uses the selected option filters and scope, including raw contract drilldowns.</li>
|
||||
<li>Storage tests now verify execution NBBO side, underlying spot, IV, and signal reasons survive normalization.</li>
|
||||
<li>The options row path already preferred <code>execution_nbbo_side</code>, <code>execution_underlying_spot</code>, and <code>execution_iv</code>; tests cover that behavior.</li>
|
||||
<li>The reset runbook is documented in <code>docs/clickhouse-reset-runbook.md</code> and is explicitly operator-confirmed.</li>
|
||||
</ul>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2>Expected Impact for End-Users</h2>
|
||||
<p>
|
||||
Traders can stay on a signal-first tape by default, switch to raw prints when investigating, and scroll into older
|
||||
ClickHouse-backed flow without seeing a separate stale-history treatment.
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2>Validation</h2>
|
||||
<ul>
|
||||
<li><span class="ok">Passed:</span> <code>bun test packages/storage/tests/option-prints.test.ts services/api/tests/live.test.ts apps/web/app/terminal.test.ts</code></li>
|
||||
<li><span class="ok">Passed:</span> <code>bun --cwd=apps/web run build</code></li>
|
||||
</ul>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2>Issues, Limitations, and Mitigations</h2>
|
||||
<ul>
|
||||
<li>The ClickHouse reset remains destructive. Mitigation: documented as a manual runbook only, never automatic startup behavior.</li>
|
||||
<li>No live browser smoke test was run in this turn. Mitigation: unit coverage and production build exercised the changed web paths.</li>
|
||||
</ul>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2>Follow-up Work</h2>
|
||||
<p>No new follow-up issue was needed. The implementation task is tracked and completed in <code>islandflow-200</code>.</p>
|
||||
</section>
|
||||
</main>
|
||||
</body>
|
||||
</html>
|
||||
Loading…
Add table
Add a link
Reference in a new issue