Synthetic Roadmap

Synthetic Market-Data Roadmap

Implementation-sized phases for extracting deterministic synthetic generation into a first-class reusable engine while keeping the useful NATS, ClickHouse, compute, API, replay, and web stack.

Beads islandflow-259 Recommended: Option B Infra-free early gates

Jump to

Linked Docs

Active Implementation Set

Use these while planning or implementing synthetic market-data phase work.

Implementation overview Full phase dossier Roadmap Markdown

Background Plans

These explain rationale, but do not add scope unless pulled into a phase doc and Beads issue.

Architecture HTML Architecture Markdown Generation research

Core Constraints

Emit canonical market event types: OptionPrint, OptionNBBO, EquityPrint, and EquityQuote.
Do not create synthetic-only market event types for the main pipeline.
Keep hidden ground-truth labels separate from emitted market events.
Keep early quality gates infra-free: bun test should not require Docker, ClickHouse, NATS, or Redis.
Build deterministic foundations before demos, UI controls, or live synthetic service behavior.
Treat historical calibration as future work, not as a dependency for the MVP synthetic generator.

Phase Sequence

Phase	Beads issue	Depends on	Purpose
01 - Deterministic spine	`islandflow-259.1`	None	Create the seeded generation foundation and canonical event output contract.
02 - Manifests, fixtures, CLI	`islandflow-259.2`	`islandflow-zxh.1`	Turn deterministic generation into durable fixtures and manifests.
03 - Scenarios, labels, expected outputs	`islandflow-259.3`	`islandflow-zxh.2`	Author named scenarios, separate labels, and expected derived outputs.
04 - Replay integration	`islandflow-259.4`	`islandflow-zxh.3`	Make replay consume synthetic runs with stable ordering and output comparison.
05 - Demo and load profiles	`islandflow-259.5`	`islandflow-zxh.4`	Expose named deterministic demo/load profiles after replay validation.
99 - Future historical calibration	`islandflow-259.6`	`islandflow-259.5`	Calibrate parameters from historical data later, after the MVP is stable.

PR Split Notes

Phase 03a

`islandflow-259.3.1` - Scenario catalog and labels

Keep scenario authoring and ground-truth label shape focused before expected-output comparison grows around it.

Phase 03b

`islandflow-259.3.2` - Expected-output manifests

Store expected derived outputs as reviewable artifacts for downstream smart-flow validation.

If any other phase starts touching unrelated service, API, UI, and storage behavior in one PR, split it before implementation continues.

Matching Beads Epic

islandflow-259 - Plan synthetic market-data implementation phases.