Stop Persisting Non-Signal Option Prints in ClickHouse

Date: 2026-05-22 23:06:02 EDT
Beads Issue: islandflow-l9h

Summary

Changes Made

• Added OPTIONS_PERSIST_SIGNAL_ONLY env support (default true) in services/ingest-options/src/index.ts.
• Refactored option-trade side effects into a new helper module: services/ingest-options/src/trade-pipeline.ts.
• Updated ingest trade handling to gate ClickHouse inserts by signal_pass when signal-only mode is enabled.
• Kept publish behavior as-is: always publish to options.prints, publish to options.signal_prints only when signal_pass=true.
• Added targeted tests in services/ingest-options/tests/trade-pipeline.test.ts.
• Added OPTIONS_PERSIST_SIGNAL_ONLY=true to both env example files.
• Documented one-time cleanup SQL and mutation verification steps in docs/clickhouse-reset-runbook.md.

Context

The options pipeline enriches and classifies prints before persistence and fanout. Previously, all enriched prints were inserted into ClickHouse regardless of signal eligibility, which retained low-value noise in durable history. The intended direction is to keep durable history aligned with the signal gate while preserving stream fanout compatibility.

Important Implementation Details

• OPTIONS_PERSIST_SIGNAL_ONLY parses standard boolean env strings (true/false, 1/0, yes/no, on/off) and defaults to true.
• Persistence decision logic is centralized in shouldPersistOptionPrint() for easy testing and future reuse.
• A startup log line now reports the active persistence mode for quick operator visibility.
• Cleanup SQL is intentionally documented as a manual one-off operational step, not automatic startup behavior.
• Expected semantic effect: new “raw” ClickHouse history for options will be signal-only when default mode is used; replay paths sourced from ClickHouse option prints will reflect the same dataset.

Relevant Diff Snippets

Unified diffs below are formatted to be compatible with diffs.com rendering conventions.

diff --git a/services/ingest-options/src/index.ts b/services/ingest-options/src/index.ts
@@
+  OPTIONS_PERSIST_SIGNAL_ONLY: z.preprocess(..., z.boolean()).default(true),
@@
+  logger.info("option print clickhouse persistence mode", { signal_only: env.OPTIONS_PERSIST_SIGNAL_ONLY });
@@
-        await insertOptionPrint(clickhouse, print);
-        await publishJson(js, SUBJECT_OPTION_PRINTS, print);
-        if (print.signal_pass) {
-          await publishJson(js, SUBJECT_OPTION_SIGNAL_PRINTS, print);
-        }
+        await processOptionTrade(print, {
+          persistSignalOnly: env.OPTIONS_PERSIST_SIGNAL_ONLY,
+          persist: async (value) => insertOptionPrint(clickhouse, value),
+          publishRaw: async (value) => publishJson(js, SUBJECT_OPTION_PRINTS, value),
+          publishSignal: async (value) => publishJson(js, SUBJECT_OPTION_SIGNAL_PRINTS, value)
+        });

diff --git a/services/ingest-options/src/trade-pipeline.ts b/services/ingest-options/src/trade-pipeline.ts
@@
+export const shouldPersistOptionPrint = (print, persistSignalOnly) => !persistSignalOnly || print.signal_pass === true;
+
+export const processOptionTrade = async (print, deps) => {
+  if (shouldPersistOptionPrint(print, deps.persistSignalOnly)) {
+    await deps.persist(print);
+  }
+  await deps.publishRaw(print);
+  if (print.signal_pass) {
+    await deps.publishSignal(print);
+  }
+};

diff --git a/docs/clickhouse-reset-runbook.md b/docs/clickhouse-reset-runbook.md
@@
+## One-Time Cleanup: Remove Non-Signal Option Prints
+docker compose exec clickhouse clickhouse-client --query "ALTER TABLE option_prints DELETE WHERE signal_pass = 0"
+...monitor with system.mutations and verify remaining_non_signal count...

Expected Impact for End-Users

Options history and replay streams backed by ClickHouse contain less noise and better reflect actionable signal flow. This improves signal-to-noise in historical tape usage without changing event schemas or API contract shapes.

Validation

• Ran focused tests: bun test services/ingest-options/tests/trade-pipeline.test.ts (pass).
• Attempted broader ingest-options test run: bun test services/ingest-options/tests (failed in this worktree due to missing module resolution for @islandflow/types in existing tests unrelated to this change).
• Manual review confirmed no API schema/type signature changes were introduced.

Issues, Limitations, and Mitigations

• Cleanup of historical non-signal rows is manual; it is not auto-executed by services to avoid accidental destructive behavior.
• ClickHouse delete mutations are asynchronous. Mitigation: documented mutation-status query and post-delete verification query.
• The full local test suite for services/ingest-options/tests was not fully runnable in this worktree due to module-resolution setup, so validation relied on targeted tests plus static review.

Follow-up Work

• Run the one-time cleanup mutation in each environment that should drop historical non-signal rows.
• After cleanup completion, verify zero remaining rows where signal_pass = 0.
• Optionally add an operational metric for dropped non-signal persistence decisions if observability of suppression volume is needed.