lavender/implementation-phase-plans #24
19 changed files with 332 additions and 4 deletions
|
|
@ -6,8 +6,8 @@
|
|||
{"_type":"issue","id":"islandflow-259.2","title":"Synthetic market-data phase 02: manifests, fixtures, and CLI","description":"Implement manifest, fixture, and CLI support for deterministic synthetic runs as described in docs/implementation/synthetic-market-data/02-manifests-fixtures-cli.md. The phase turns the deterministic engine into reusable test/demo artifacts.","acceptance_criteria":"CLI can generate fixtures and expected-output-ready manifests from seed bundles and profiles; manifests pin generator version, seed, parameter hash, event hashes, and replay ordering; fixture helpers support infra-free Bun tests; generated artifacts avoid embedding hidden labels in market events.","spec_id":"docs/implementation/synthetic-market-data/02-manifests-fixtures-cli.md","status":"open","priority":1,"issue_type":"feature","owner":"dishes@dpdrm.com","created_at":"2026-06-16T17:38:37Z","created_by":"dirtydishes","updated_at":"2026-06-16T17:38:37Z","labels":["phase","planning","synthetic-market-data"],"dependencies":[{"issue_id":"islandflow-259.2","depends_on_id":"islandflow-259","type":"parent-child","created_at":"2026-06-16T13:38:37Z","created_by":"dirtydishes","metadata":"{}"},{"issue_id":"islandflow-259.2","depends_on_id":"islandflow-zxh.1","type":"blocks","created_at":"2026-06-16T13:39:04Z","created_by":"dirtydishes","metadata":"{}"}],"dependency_count":1,"dependent_count":2,"comment_count":0}
|
||||
{"_type":"issue","id":"islandflow-zxh.1","title":"Smart-flow phase 01: contracts and vocabulary","description":"Implement the contract and naming foundation described in docs/implementation/smart-money/01-contracts-vocabulary.md. The phase separates facts, evidence, hypotheses, confidence, abstention, and user-facing insight projections before classifier behavior is expanded.","acceptance_criteria":"Canonical contracts distinguish observations, evidence clusters, hypotheses, confidence vectors, abstention reasons, and insight projections; legacy smart-money naming is compatibility-only where needed; version fields are present; migration risks and aliases are documented.","spec_id":"docs/implementation/smart-money/01-contracts-vocabulary.md","status":"open","priority":1,"issue_type":"feature","owner":"dishes@dpdrm.com","created_at":"2026-06-16T17:38:36Z","created_by":"dirtydishes","updated_at":"2026-06-16T17:38:36Z","labels":["phase","planning","smart-flow","smart-money"],"dependencies":[{"issue_id":"islandflow-zxh.1","depends_on_id":"islandflow-259.1","type":"blocks","created_at":"2026-06-16T13:39:03Z","created_by":"dirtydishes","metadata":"{}"},{"issue_id":"islandflow-zxh.1","depends_on_id":"islandflow-zxh","type":"parent-child","created_at":"2026-06-16T13:38:35Z","created_by":"dirtydishes","metadata":"{}"}],"dependency_count":1,"dependent_count":1,"comment_count":0}
|
||||
{"_type":"issue","id":"islandflow-259.1","title":"Synthetic market-data phase 01: deterministic spine","description":"Implement the deterministic synthetic market-data foundation described in docs/implementation/synthetic-market-data/01-deterministic-spine.md. The phase extracts generation into a seeded package/API that emits canonical market events with provenance while keeping labels separate.","acceptance_criteria":"Seeded generation is byte/hash stable for fixed inputs; emitted events use canonical OptionPrint, OptionNBBO, EquityPrint, and EquityQuote contracts; provenance metadata includes run/seed/parameter context; hidden labels are not embedded in market events; early tests run without Docker, ClickHouse, NATS, or Redis.","spec_id":"docs/implementation/synthetic-market-data/01-deterministic-spine.md","status":"open","priority":1,"issue_type":"feature","owner":"dishes@dpdrm.com","created_at":"2026-06-16T17:38:34Z","created_by":"dirtydishes","updated_at":"2026-06-16T17:38:34Z","labels":["phase","planning","synthetic-market-data"],"dependencies":[{"issue_id":"islandflow-259.1","depends_on_id":"islandflow-259","type":"parent-child","created_at":"2026-06-16T13:38:33Z","created_by":"dirtydishes","metadata":"{}"}],"dependency_count":0,"dependent_count":1,"comment_count":0}
|
||||
{"_type":"issue","id":"islandflow-zxh","title":"Plan smart-money to smart-flow implementation phases","description":"Track the phase-by-phase implementation plan split from docs/plans/smart-flow-architecture-review.md. Durable roadmap: docs/implementation/smart-money/00-roadmap.md. This epic covers evidence-backed smart-flow contracts, clustering, hypothesis scoring, replay evaluation, API/UI explainability, and future calibration.","acceptance_criteria":"Phase docs exist under docs/implementation/smart-money; child phase issues link to their docs; blocker dependencies reflect the planned implementation order; no application code is implemented as part of this planning epic.","spec_id":"docs/implementation/smart-money/00-roadmap.md","status":"open","priority":1,"issue_type":"epic","owner":"dishes@dpdrm.com","created_at":"2026-06-16T17:38:32Z","created_by":"dirtydishes","updated_at":"2026-06-16T17:38:32Z","labels":["planning","smart-flow","smart-money"],"dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||
{"_type":"issue","id":"islandflow-259","title":"Plan synthetic market-data implementation phases","description":"Track the phase-by-phase implementation plan split from docs/plans/synthetic-market-data-architecture-review.md. Durable roadmap: docs/implementation/synthetic-market-data/00-roadmap.md. This epic covers deterministic synthetic event generation with canonical market event types, separate labels/manifests, replay integration, demo/load profiles, and future historical calibration.","acceptance_criteria":"Phase docs exist under docs/implementation/synthetic-market-data; child phase issues link to their docs; blocker dependencies reflect the planned implementation order; no application code is implemented as part of this planning epic.","spec_id":"docs/implementation/synthetic-market-data/00-roadmap.md","status":"open","priority":1,"issue_type":"epic","owner":"dishes@dpdrm.com","created_at":"2026-06-16T17:38:30Z","created_by":"dirtydishes","updated_at":"2026-06-16T17:38:30Z","labels":["planning","synthetic-market-data"],"dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||
{"_type":"issue","id":"islandflow-zxh","title":"Plan smart-money to smart-flow implementation phases","description":"Track the phase-by-phase implementation plan split from docs/plans/smart-flow-architecture-review.md. Durable roadmap: docs/implementation/smart-money/00-roadmap.md. This epic covers evidence-backed smart-flow contracts, clustering, hypothesis scoring, replay evaluation, API/UI explainability, and future calibration.","acceptance_criteria":"Phase docs exist under docs/implementation/smart-money; child phase issues link to their docs; blocker dependencies reflect the planned implementation order; no application code is implemented as part of this planning epic.","notes":"2026-06-16 planning-doc pass: added source-document links from architecture/phase docs to research docs, added compact research basis/deferred research sections, and clarified research docs as background rather than implementation scope. No application code changed.","spec_id":"docs/implementation/smart-money/00-roadmap.md","status":"in_progress","priority":1,"issue_type":"epic","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-06-16T17:38:32Z","created_by":"dirtydishes","updated_at":"2026-06-16T17:53:41Z","started_at":"2026-06-16T17:51:54Z","labels":["planning","smart-flow","smart-money"],"dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||
{"_type":"issue","id":"islandflow-259","title":"Plan synthetic market-data implementation phases","description":"Track the phase-by-phase implementation plan split from docs/plans/synthetic-market-data-architecture-review.md. Durable roadmap: docs/implementation/synthetic-market-data/00-roadmap.md. This epic covers deterministic synthetic event generation with canonical market event types, separate labels/manifests, replay integration, demo/load profiles, and future historical calibration.","acceptance_criteria":"Phase docs exist under docs/implementation/synthetic-market-data; child phase issues link to their docs; blocker dependencies reflect the planned implementation order; no application code is implemented as part of this planning epic.","notes":"2026-06-16 planning-doc pass: added source-document links from architecture/phase docs to research docs, added compact research basis/deferred research sections, and clarified research docs as background rather than implementation scope. No application code changed.","spec_id":"docs/implementation/synthetic-market-data/00-roadmap.md","status":"in_progress","priority":1,"issue_type":"epic","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-06-16T17:38:30Z","created_by":"dirtydishes","updated_at":"2026-06-16T17:53:40Z","started_at":"2026-06-16T17:51:54Z","labels":["planning","synthetic-market-data"],"dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||
{"_type":"issue","id":"islandflow-0e3","title":"Fix PR 23 CI failures","description":"PR 23 is failing the Forgejo CI Validate workflow. Reproduce the failing gates locally, fix the underlying formatting/lint/typecheck/test/build issues, update the PR branch, and confirm the remote check passes.","status":"closed","priority":1,"issue_type":"bug","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-06-14T19:35:07Z","created_by":"dirtydishes","updated_at":"2026-06-14T19:37:01Z","started_at":"2026-06-14T19:35:12Z","closed_at":"2026-06-14T19:37:01Z","close_reason":"Local Validate workflow passes after applying formatter output and syncing the Docker workspace snapshot.","dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||
{"_type":"issue","id":"islandflow-9w7","title":"Allow local dev origins on hosted API","description":"Local bun run dev:web and desktop-local point at the hosted API, but browser requests from http://127.0.0.1:3000 are blocked because the API omits CORS headers and returns 404 for OPTIONS preflight. Add API-side CORS handling, validate local web/desktop browser access, and deploy the API fix.","acceptance_criteria":"API responses include Access-Control-Allow-Origin for allowed local/dev origins; OPTIONS preflight succeeds; bun run dev:web reaches hosted REST/WS endpoints from a browser; bun run dev:desktop local mode reaches the backend through the local web UI; tests/build pass; fix is deployed to api.flow.deltaisland.io.","status":"closed","priority":1,"issue_type":"bug","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-06-13T15:04:19Z","created_by":"dirtydishes","updated_at":"2026-06-13T15:29:42Z","started_at":"2026-06-13T15:04:26Z","closed_at":"2026-06-13T15:29:42Z","close_reason":"Hosted API now reflects allowed local dev origins and handles OPTIONS preflight; local web and desktop dev runners both reach https://api.flow.deltaisland.io; API tests, typecheck, and web build passed.","dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||
{"_type":"issue","id":"islandflow-xkq","title":"Rebuild production dashboard options news around mock9 aesthetic","description":"Reconstruct the production web UI for Dashboard, Options, and News around the mock9 through mock12 dense terminal aesthetic while preserving production data subscriptions, drawers, virtualization, route helpers, redirects, and validation.","status":"closed","priority":1,"issue_type":"feature","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-06-13T14:07:34Z","created_by":"dirtydishes","updated_at":"2026-06-13T14:26:46Z","started_at":"2026-06-13T14:07:53Z","closed_at":"2026-06-13T14:26:46Z","close_reason":"Rebuilt Dashboard, Options, and News around the dense mock9 to mock12 production aesthetic; tests and build passed, and Browser visual inspection was documented as blocked by the unavailable in-app browser backend.","dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||
|
|
|
|||
|
|
@ -2,7 +2,18 @@
|
|||
|
||||
This directory is the active planning layer for the synthetic market-data and smart-money/smart-flow architecture work.
|
||||
|
||||
The architecture reviews in `docs/plans/` are background guidance. Future implementation work should use the current phase document and matching Beads issue as the active scope. If a phase document and an older architecture review disagree, pause and update the phase document or Beads issue before writing code.
|
||||
The architecture reviews in `docs/plans/` and research reports in `docs/research-docs/` are background guidance. Future implementation work should use the current phase document and matching Beads issue as the active scope. If a phase document and an older architecture review or research report disagree, pause and update the phase document or Beads issue before writing code.
|
||||
|
||||
## Document Precedence
|
||||
|
||||
Use this precedence order when planning or implementing phase work:
|
||||
|
||||
1. Beads issue
|
||||
2. Phase document in `docs/implementation/`
|
||||
3. Architecture plan in `docs/plans/`
|
||||
4. Research report in `docs/research-docs/`
|
||||
|
||||
Research reports provide rationale and useful constraints. They do not add active implementation scope unless that scope is explicitly pulled into a phase document and Beads issue.
|
||||
|
||||
## Source Plans
|
||||
|
||||
|
|
|
|||
|
|
@ -2,6 +2,14 @@
|
|||
|
||||
This roadmap breaks `docs/plans/smart-flow-architecture-review.md` into implementation-sized phases. The recommended direction is Option B: keep the working stack, but rebuild the domain pipeline around observations, evidence clusters, cautious hypotheses, confidence, alternatives, abstention, replay evaluation, and user-facing insight projections.
|
||||
|
||||
## Source Documents
|
||||
|
||||
- Architecture plan: [`docs/plans/smart-flow-architecture-review.md`](../../plans/smart-flow-architecture-review.md)
|
||||
- Research report: [`docs/research-docs/smart-flow-market-mechanics.md`](../../research-docs/smart-flow-market-mechanics.md)
|
||||
- Research architecture review copy: [`docs/research-docs/smart-flow-architecture-review.md`](../../research-docs/smart-flow-architecture-review.md)
|
||||
|
||||
The research documents are background and rationale only. Scope comes from the Beads issue and the phase document.
|
||||
|
||||
## Core Constraints
|
||||
|
||||
- Do not treat "smart money" as a canonical fact emitted by the system.
|
||||
|
|
|
|||
|
|
@ -8,6 +8,23 @@ Introduce the domain vocabulary and contracts that distinguish observations, evi
|
|||
|
||||
The current system has useful infrastructure but overconfident domain names. Before changing classifier behavior, the codebase needs the language to express what is observed, what is inferred, what is uncertain, and when the system should abstain.
|
||||
|
||||
## Source documents
|
||||
|
||||
- Architecture plan: [`docs/plans/smart-flow-architecture-review.md`](../../plans/smart-flow-architecture-review.md)
|
||||
- Research report: [`docs/research-docs/smart-flow-market-mechanics.md`](../../research-docs/smart-flow-market-mechanics.md)
|
||||
|
||||
These documents are rationale, not added scope. This phase implements only vocabulary, contracts, versioning, and compatibility notes.
|
||||
|
||||
## Research basis
|
||||
|
||||
- The research direction is direct observation to inference to hypothesis, with preserved evidence and visible uncertainty.
|
||||
- "Smart money" should not be modeled as a canonical fact; user-facing insight should be a projection from evidence-backed hypotheses.
|
||||
- Confidence, conviction, alternatives, and abstention need separate language before behavior changes.
|
||||
|
||||
## Deferred research ideas
|
||||
|
||||
- Participant identity claims and research-grade calibration stay outside the vocabulary foundation.
|
||||
|
||||
## Dependencies on earlier phases
|
||||
|
||||
- `islandflow-259.1` - Synthetic deterministic spine, so contract work can align with canonical raw event and provenance assumptions.
|
||||
|
|
|
|||
|
|
@ -8,6 +8,23 @@ Make evidence extraction, eligibility, quote/context joins, clustering, and feat
|
|||
|
||||
Contracts alone do not change behavior. This phase gives the system a clean evidence layer so later scoring can reason from auditable facts instead of a generic feature bag or overconfident classifier labels.
|
||||
|
||||
## Source documents
|
||||
|
||||
- Architecture plan: [`docs/plans/smart-flow-architecture-review.md`](../../plans/smart-flow-architecture-review.md)
|
||||
- Research report: [`docs/research-docs/smart-flow-market-mechanics.md`](../../research-docs/smart-flow-market-mechanics.md)
|
||||
|
||||
These documents are rationale, not added scope. This phase implements only eligibility, evidence facts, clustering, and traceable features.
|
||||
|
||||
## Research basis
|
||||
|
||||
- Trade signing, quote context, sale conditions, stale quotes, wide markets, and event context all affect whether a print is usable evidence.
|
||||
- Evidence should preserve raw refs, eligibility decisions, quality signals, and negative context before any hypothesis is scored.
|
||||
- Ingest should normalize observations; signal policy belongs in explicit evidence/scoring stages.
|
||||
|
||||
## Deferred research ideas
|
||||
|
||||
- Full IV-surface modeling, broad news/FDA event feeds, and deep historical baselines can be added later when scoped.
|
||||
|
||||
## Dependencies on earlier phases
|
||||
|
||||
- `islandflow-zxh.1` - Smart-flow contracts and vocabulary
|
||||
|
|
|
|||
|
|
@ -8,6 +8,23 @@ Convert evidence clusters into cautious flow hypotheses with explicit score vect
|
|||
|
||||
Scoring should wait until the system can represent evidence clearly and synthetic scenarios can describe expected positive, negative, and abstention cases. This phase is where the product stops acting like every signal is a confident "smart money" claim.
|
||||
|
||||
## Source documents
|
||||
|
||||
- Architecture plan: [`docs/plans/smart-flow-architecture-review.md`](../../plans/smart-flow-architecture-review.md)
|
||||
- Research report: [`docs/research-docs/smart-flow-market-mechanics.md`](../../research-docs/smart-flow-market-mechanics.md)
|
||||
|
||||
These documents are rationale, not added scope. This phase implements only cautious hypothesis scoring, alternatives, penalties, and abstention.
|
||||
|
||||
## Research basis
|
||||
|
||||
- Premium concentration, sweep-like activity, IV movement, and equity confirmation support hypotheses only when evidence quality and context agree.
|
||||
- False positives from deep-ITM stock replacement, spreads/hedges, stale quotes, and event-driven flow need explicit penalties or abstention.
|
||||
- Confidence should reflect policy confidence in the evidence, not a claim of hidden participant identity.
|
||||
|
||||
## Deferred research ideas
|
||||
|
||||
- Empirical threshold tuning, historical calibration, and ML-based scoring stay future work until replay/golden validation exists.
|
||||
|
||||
## Dependencies on earlier phases
|
||||
|
||||
- `islandflow-zxh.1` - Smart-flow contracts and vocabulary
|
||||
|
|
|
|||
|
|
@ -8,6 +8,24 @@ Make deterministic replay and golden output comparison the acceptance gate for s
|
|||
|
||||
Replay evaluation should come after synthetic replay can select stable runs and after hypothesis scoring has outputs worth validating. This phase turns architecture discipline into a repeatable test path.
|
||||
|
||||
## Source documents
|
||||
|
||||
- Architecture plan: [`docs/plans/smart-flow-architecture-review.md`](../../plans/smart-flow-architecture-review.md)
|
||||
- Research report: [`docs/research-docs/smart-flow-market-mechanics.md`](../../research-docs/smart-flow-market-mechanics.md)
|
||||
- Synthetic research report: [`docs/research-docs/synthetic-market-data-generation.md`](../../research-docs/synthetic-market-data-generation.md)
|
||||
|
||||
These documents are rationale, not added scope. This phase implements only deterministic replay evaluation and compact golden tests.
|
||||
|
||||
## Research basis
|
||||
|
||||
- Replay is the acceptance gate for derived smart-flow outputs because evidence and hypotheses must be reproducible.
|
||||
- Validation must include positive cases, false positives, noisy contexts, and abstentions.
|
||||
- Tests should avoid lookahead bias and compare stable signatures instead of brittle full-payload dumps.
|
||||
|
||||
## Deferred research ideas
|
||||
|
||||
- Historical backtesting windows, empirical calibration datasets, and broad benchmark reports belong in later calibration work.
|
||||
|
||||
## Dependencies on earlier phases
|
||||
|
||||
- `islandflow-zxh.1` - Smart-flow contracts and vocabulary
|
||||
|
|
|
|||
|
|
@ -8,6 +8,23 @@ Expose evidence-backed smart-flow outputs through API, websocket, and UI surface
|
|||
|
||||
The presentation layer should wait until contracts, evidence, scoring, and replay evaluation are stable. Otherwise the UI will harden old overconfident language or teach users to trust unvalidated outputs.
|
||||
|
||||
## Source documents
|
||||
|
||||
- Architecture plan: [`docs/plans/smart-flow-architecture-review.md`](../../plans/smart-flow-architecture-review.md)
|
||||
- Research report: [`docs/research-docs/smart-flow-market-mechanics.md`](../../research-docs/smart-flow-market-mechanics.md)
|
||||
|
||||
These documents are rationale, not added scope. This phase implements only API, websocket, and UI explainability surfaces for validated outputs.
|
||||
|
||||
## Research basis
|
||||
|
||||
- Users need to see evidence quality, confidence versus conviction, alternatives, and abstention instead of a single certainty label.
|
||||
- The research supports cautious smart-flow insight projections, not canonical "smart money" facts.
|
||||
- Why-not and penalty context are part of the product surface because false positives are central to the domain.
|
||||
|
||||
## Deferred research ideas
|
||||
|
||||
- Advanced explanatory analytics, learned confidence calibration, and broad catalyst intelligence should wait for future scoped work.
|
||||
|
||||
## Dependencies on earlier phases
|
||||
|
||||
- `islandflow-zxh.1` - Smart-flow contracts and vocabulary
|
||||
|
|
|
|||
|
|
@ -8,6 +8,23 @@ Plan future calibration of smart-flow confidence, policy thresholds, penalties,
|
|||
|
||||
The architecture should leave room for calibration, but calibration should not block the MVP. The system first needs clean facts, evidence, hypotheses, and replayable evaluation before tuning can be meaningful.
|
||||
|
||||
## Source documents
|
||||
|
||||
- Architecture plan: [`docs/plans/smart-flow-architecture-review.md`](../../plans/smart-flow-architecture-review.md)
|
||||
- Research report: [`docs/research-docs/smart-flow-market-mechanics.md`](../../research-docs/smart-flow-market-mechanics.md)
|
||||
|
||||
These documents are rationale, not added scope. This future phase is the place to turn research ideas into scoped calibration work after MVP.
|
||||
|
||||
## Research basis
|
||||
|
||||
- Historical validation should be time-of-day aware and avoid lookahead bias.
|
||||
- Baselines for "unusual" should account for ticker, tenor bucket, regime, and event-day exclusions.
|
||||
- Confidence, penalties, abstention, and alternatives need versioned policy outputs so calibration stays auditable.
|
||||
|
||||
## Deferred research ideas
|
||||
|
||||
- ML scoring, learned calibration, richer catalyst feeds, and large historical benchmark suites require separate future Beads scope.
|
||||
|
||||
## Dependencies on earlier phases
|
||||
|
||||
- `islandflow-zxh.5` - Smart-flow API/UI explainability
|
||||
|
|
|
|||
|
|
@ -2,6 +2,14 @@
|
|||
|
||||
This roadmap breaks `docs/plans/synthetic-market-data-architecture-review.md` into implementation-sized phases. The recommended direction is still Option B: extract deterministic synthetic generation into a first-class reusable engine while keeping the useful NATS, ClickHouse, compute, API, replay, and web stack.
|
||||
|
||||
## Source Documents
|
||||
|
||||
- Architecture plan: [`docs/plans/synthetic-market-data-architecture-review.md`](../../plans/synthetic-market-data-architecture-review.md)
|
||||
- Research report: [`docs/research-docs/synthetic-market-data-generation.md`](../../research-docs/synthetic-market-data-generation.md)
|
||||
- Research architecture review copy: [`docs/research-docs/synthetic-data-architecture-review.md`](../../research-docs/synthetic-data-architecture-review.md)
|
||||
|
||||
The research documents are background and rationale only. Scope comes from the Beads issue and the phase document.
|
||||
|
||||
## Core Constraints
|
||||
|
||||
- Emit canonical market event types: `OptionPrint`, `OptionNBBO`, `EquityPrint`, and `EquityQuote`.
|
||||
|
|
|
|||
|
|
@ -8,6 +8,23 @@ Create the reusable deterministic foundation for synthetic market data. This pha
|
|||
|
||||
Everything else depends on reproducible raw events. Manifests, labels, replay, demos, and smart-flow tests are only trustworthy if the same seed/profile bundle produces the same canonical market event stream every time.
|
||||
|
||||
## Source documents
|
||||
|
||||
- Architecture plan: [`docs/plans/synthetic-market-data-architecture-review.md`](../../plans/synthetic-market-data-architecture-review.md)
|
||||
- Research report: [`docs/research-docs/synthetic-market-data-generation.md`](../../research-docs/synthetic-market-data-generation.md)
|
||||
|
||||
These documents are rationale, not added scope. This phase implements only the deterministic spine described below.
|
||||
|
||||
## Research basis
|
||||
|
||||
- The research recommends a no-history-first, transparent, deterministic generator rather than historical replay as an MVP prerequisite.
|
||||
- The generator needs core market realism handles from the start: discrete ticks, varying spreads, clustered arrivals, heterogeneous sizes, quote/trade separation, and options-chain sparsity.
|
||||
- Full agent-based, limit-order-book, and generative-ML simulation are too heavy for the first foundation.
|
||||
|
||||
## Deferred research ideas
|
||||
|
||||
- Full LOB simulation, agent-based simulation, generative ML, and empirical calibration stay out of this phase.
|
||||
|
||||
## Dependencies on earlier phases
|
||||
|
||||
None. This is the first synthetic phase.
|
||||
|
|
|
|||
|
|
@ -8,6 +8,23 @@ Turn the deterministic generator into reusable artifacts: fixture files, run man
|
|||
|
||||
The deterministic spine gives the repo stable raw events. The next step is to make those events durable and addressable so downstream phases can reference exact generated runs instead of recreating ad hoc local randomness.
|
||||
|
||||
## Source documents
|
||||
|
||||
- Architecture plan: [`docs/plans/synthetic-market-data-architecture-review.md`](../../plans/synthetic-market-data-architecture-review.md)
|
||||
- Research report: [`docs/research-docs/synthetic-market-data-generation.md`](../../research-docs/synthetic-market-data-generation.md)
|
||||
|
||||
These documents are rationale, not added scope. This phase implements only manifests, fixtures, and CLI support.
|
||||
|
||||
## Research basis
|
||||
|
||||
- Deterministic replay and reviewable artifacts are necessary for synthetic data to be useful as validation data, not just demo data.
|
||||
- Expected-output manifests should pin seed, profile, generator version, event hashes, and replay ordering.
|
||||
- Hidden labels must stay separate from market events so tests do not leak ground truth into production-like paths.
|
||||
|
||||
## Deferred research ideas
|
||||
|
||||
- Empirical residual resampling and historical-window bootstrapping are future artifact sources, not this CLI's first requirement.
|
||||
|
||||
## Dependencies on earlier phases
|
||||
|
||||
- `islandflow-259.1` - Synthetic deterministic spine
|
||||
|
|
|
|||
|
|
@ -8,6 +8,24 @@ Author named deterministic scenarios, separate ground-truth labels, and expected
|
|||
|
||||
The generator and manifest layers should exist before scenario authoring. Smart-flow evidence clustering should also define enough vocabulary for expected outputs to describe evidence requirements without leaking labels into emitted market events.
|
||||
|
||||
## Source documents
|
||||
|
||||
- Architecture plan: [`docs/plans/synthetic-market-data-architecture-review.md`](../../plans/synthetic-market-data-architecture-review.md)
|
||||
- Research report: [`docs/research-docs/synthetic-market-data-generation.md`](../../research-docs/synthetic-market-data-generation.md)
|
||||
- Smart-flow research report: [`docs/research-docs/smart-flow-market-mechanics.md`](../../research-docs/smart-flow-market-mechanics.md)
|
||||
|
||||
These documents are rationale, not added scope. This phase implements only named scenarios, separate labels, and expected-output contracts.
|
||||
|
||||
## Research basis
|
||||
|
||||
- Scenario injection into a realistic synthetic background is mandatory for labeled, replayable alert tests.
|
||||
- Negative, noisy, stale, wide-market, and event-context cases matter as much as positive "should detect" scenarios.
|
||||
- Labels and expected outputs need required evidence, forbidden evidence, confidence bands, and false-positive penalties.
|
||||
|
||||
## Deferred research ideas
|
||||
|
||||
- Empirical tuning of scenario frequencies, full historical replay-plus-mutation, and learned scenario generation belong after the MVP scenario catalog is stable.
|
||||
|
||||
## Dependencies on earlier phases
|
||||
|
||||
- `islandflow-259.1` - Synthetic deterministic spine
|
||||
|
|
|
|||
|
|
@ -8,6 +8,23 @@ Make replay consume synthetic runs deterministically, either directly from gener
|
|||
|
||||
Replay should not be wired to synthetic data until the generator, manifests, labels, and smart-flow hypothesis pipeline have stable semantics. At this point, replay can become a serious acceptance gate instead of a demo convenience.
|
||||
|
||||
## Source documents
|
||||
|
||||
- Architecture plan: [`docs/plans/synthetic-market-data-architecture-review.md`](../../plans/synthetic-market-data-architecture-review.md)
|
||||
- Research report: [`docs/research-docs/synthetic-market-data-generation.md`](../../research-docs/synthetic-market-data-generation.md)
|
||||
|
||||
These documents are rationale, not added scope. This phase implements only deterministic synthetic replay integration.
|
||||
|
||||
## Research basis
|
||||
|
||||
- Replay must preserve event-time ordering and deterministic run identity to prove derived behavior.
|
||||
- Synthetic runs should be selectable by source and run metadata rather than ambient randomness.
|
||||
- Optional ClickHouse/NATS materialization can exist later, but fast validation should remain infra-free.
|
||||
|
||||
## Deferred research ideas
|
||||
|
||||
- Historical replay-plus-mutation and calibrated replay benchmarks are future layers after synthetic replay semantics are stable.
|
||||
|
||||
## Dependencies on earlier phases
|
||||
|
||||
- `islandflow-259.1` - Synthetic deterministic spine
|
||||
|
|
|
|||
|
|
@ -8,6 +8,23 @@ Expose deterministic synthetic runs as named demo and load profiles after the ge
|
|||
|
||||
Demos are useful only after the underlying data can be trusted. This phase deliberately waits until replay and golden evaluation prove the event semantics, so hosted controls do not become a front door to ambient randomness.
|
||||
|
||||
## Source documents
|
||||
|
||||
- Architecture plan: [`docs/plans/synthetic-market-data-architecture-review.md`](../../plans/synthetic-market-data-architecture-review.md)
|
||||
- Research report: [`docs/research-docs/synthetic-market-data-generation.md`](../../research-docs/synthetic-market-data-generation.md)
|
||||
|
||||
These documents are rationale, not added scope. This phase implements only named deterministic demo and load profiles.
|
||||
|
||||
## Research basis
|
||||
|
||||
- Demo streams should use named, seeded profiles so product behavior is reproducible.
|
||||
- Load profiles should scale rate or volume without changing event semantics.
|
||||
- Realism should come from the generator and scenarios, not hidden UI knobs or wall-clock randomness.
|
||||
|
||||
## Deferred research ideas
|
||||
|
||||
- Historically bootstrapped demo streams, learned realism upgrades, and full LOB-style demos stay future work.
|
||||
|
||||
## Dependencies on earlier phases
|
||||
|
||||
- `islandflow-259.1` - Synthetic deterministic spine
|
||||
|
|
|
|||
|
|
@ -8,6 +8,23 @@ Plan future calibration of synthetic generator parameters from historical market
|
|||
|
||||
It is useful to name the future work now so early designs keep calibration hooks in mind. It should not come before deterministic generation, manifests, scenarios, replay, or demo profiles.
|
||||
|
||||
## Source documents
|
||||
|
||||
- Architecture plan: [`docs/plans/synthetic-market-data-architecture-review.md`](../../plans/synthetic-market-data-architecture-review.md)
|
||||
- Research report: [`docs/research-docs/synthetic-market-data-generation.md`](../../research-docs/synthetic-market-data-generation.md)
|
||||
|
||||
These documents are rationale, not added scope. This future phase is the place to turn research ideas into scoped calibration work after MVP.
|
||||
|
||||
## Research basis
|
||||
|
||||
- Once historical data exists, calibration should fit arrival curves, spread states, size mixtures, venue shares, and options-chain activity weights.
|
||||
- Replay-plus-mutation can improve realism while preserving deterministic test intent.
|
||||
- Calibration should layer onto the deterministic engine rather than replace it wholesale.
|
||||
|
||||
## Deferred research ideas
|
||||
|
||||
- Generative ML, learned LOB simulators, and agent-based models remain later research tracks unless a future Beads issue scopes them explicitly.
|
||||
|
||||
## Dependencies on earlier phases
|
||||
|
||||
- `islandflow-259.5` - Synthetic demo and load profiles
|
||||
|
|
|
|||
|
|
@ -4,10 +4,17 @@
|
|||
|
||||
No source code was modified. The current architecture is **not suitable as-is**, but it is **close enough to refactor, not rewrite**. The stack is right; the domain language and pipeline shape are not.
|
||||
|
||||
Research direction: direct observation → inference → hypothesis, with preserved evidence and visible uncertainty. See [smart-flow-market-mechanics.md](/Users/kell/dev/islandflow/docs/research-docs/smart-flow-market-mechanics.md:7).
|
||||
Research direction: direct observation → inference → hypothesis, with preserved evidence and visible uncertainty.
|
||||
|
||||
Key code evidence: `FlowPacket` is a generic feature bag in [events.ts](/Users/kell/dev/islandflow/packages/types/src/events.ts:193), `SmartMoneyEvent` already has useful score/abstention fields in [events.ts](/Users/kell/dev/islandflow/packages/types/src/events.ts:283), compute emits smart-money events then compatibility hits/alerts in [index.ts](/Users/kell/dev/islandflow/services/compute/src/index.ts:1086), storage keeps core hypothesis detail as JSON in [smart-money-events.ts](/Users/kell/dev/islandflow/packages/storage/src/smart-money-events.ts:24), and replay currently replays raw market streams rather than validating the whole derived pipeline in [replay/index.ts](/Users/kell/dev/islandflow/services/replay/src/index.ts:69).
|
||||
|
||||
## Source Documents
|
||||
|
||||
- Research report: [`docs/research-docs/smart-flow-market-mechanics.md`](../research-docs/smart-flow-market-mechanics.md)
|
||||
- Research architecture review copy: [`docs/research-docs/smart-flow-architecture-review.md`](../research-docs/smart-flow-architecture-review.md)
|
||||
|
||||
These research documents explain the rationale. They are background, not implementation scope; execution scope lives in the Beads issue and the relevant phase document.
|
||||
|
||||
## Area Classification
|
||||
|
||||
| Area | Call | Architecture Review |
|
||||
|
|
|
|||
|
|
@ -5,6 +5,13 @@
|
|||
- Recommendation: **Option B — Refactor**. Conservative work would trap determinism inside ingest adapters; full redesign is premature. Refactor makes synthetic generation first-class while keeping the useful NATS, ClickHouse, compute, API, and web stack.
|
||||
- Core direction: build a no-history, seeded, manifest-driven synthetic event engine with canonical real event types, separate labels/manifests, deterministic replay, fixture generation, load profiles, and demo scenarios.
|
||||
|
||||
## Source Documents
|
||||
|
||||
- Research report: [`docs/research-docs/synthetic-market-data-generation.md`](../research-docs/synthetic-market-data-generation.md)
|
||||
- Research architecture review copy: [`docs/research-docs/synthetic-data-architecture-review.md`](../research-docs/synthetic-data-architecture-review.md)
|
||||
|
||||
These research documents explain the rationale. They are background, not implementation scope; execution scope lives in the Beads issue and the relevant phase document.
|
||||
|
||||
## Direct Answers
|
||||
1. Synthetic generation should be a **combination**: a reusable `@islandflow/synthetic-market` package, a CLI for fixture/run generation, replay-source integration, test fixture helpers, and demo presets. A service should be only a thin live/demo emitter.
|
||||
2. Synthetic events should map to existing canonical event types: `OptionPrint`, `OptionNBBO`, `EquityPrint`, and `EquityQuote`. Do not create parallel synthetic-only market event types for the main pipeline.
|
||||
|
|
|
|||
81
docs/research-docs/synthetic-data-architecture-review.md
Normal file
81
docs/research-docs/synthetic-data-architecture-review.md
Normal file
|
|
@ -0,0 +1,81 @@
|
|||
# Synthetic Market-Data Architecture Review
|
||||
|
||||
## Summary
|
||||
- Target file: `docs/plans/synthetic-market-data-architecture-review.md`. No files were changed in this Plan Mode pass.
|
||||
- Recommendation: **Option B — Refactor**. Conservative work would trap determinism inside ingest adapters; full redesign is premature. Refactor makes synthetic generation first-class while keeping the useful NATS, ClickHouse, compute, API, and web stack.
|
||||
- Core direction: build a no-history, seeded, manifest-driven synthetic event engine with canonical real event types, separate labels/manifests, deterministic replay, fixture generation, load profiles, and demo scenarios.
|
||||
|
||||
## Direct Answers
|
||||
1. Synthetic generation should be a **combination**: a reusable `@islandflow/synthetic-market` package, a CLI for fixture/run generation, replay-source integration, test fixture helpers, and demo presets. A service should be only a thin live/demo emitter.
|
||||
2. Synthetic events should map to existing canonical event types: `OptionPrint`, `OptionNBBO`, `EquityPrint`, and `EquityQuote`. Do not create parallel synthetic-only market event types for the main pipeline.
|
||||
3. Use **metadata plus isolation**, not permanent separate business schemas. Add provenance such as `source_kind`, `run_id`, `parameter_snapshot_hash`, and optional `scenario_id`; use run-scoped subjects/databases for tests and load runs when isolation matters.
|
||||
4. Ground-truth labels should be separate label records keyed by `run_id`, `scenario_id`, event IDs/trace IDs, expected class, expected direction, confidence band, required/forbidden evidence, and false-positive penalties. Do not expose hidden labels on emitted market events.
|
||||
5. Expected-output manifests should be versioned JSON/YAML artifacts produced by the CLI. They should pin seed bundle, generator version, parameter snapshot hash, generated event hashes, replay ordering, expected derived events, alert/no-alert expectations, and evidence requirements.
|
||||
6. Deterministic replay should consume either generated fixture files directly or materialized ClickHouse rows through the same replay ordering: event time, ingest time, seq, stable event ID. Replay should support a `synthetic` source/run selector.
|
||||
7. Tests should use synthetic data at three levels: pure package invariants, small golden manifests through compute batch logic, and optional infra-backed NATS/ClickHouse integration tests. `bun test` should not require Docker.
|
||||
8. Demos should use named demo runs/scenarios, not ambient live randomness. Keep the hosted synthetic control drawer for live demo tuning, but add deterministic demo run selection/replay.
|
||||
9. First-class domain objects: `SyntheticRun`, `SeedBundle`, `ParameterSnapshot`, `SymbolProfile`, `LiquidityProfile`, `VolatilityRegime`, `OptionChainProfile`, `ScenarioInjection`, `GroundTruthLabel`, `ExpectedOutputManifest`, `GeneratedEventBatch`, `ReplayPlan`, `LoadProfile`, and `DemoProfile`.
|
||||
10. Implementation details: PRNG algorithm internals, sampling formulas, placement heuristics, adapter timers, NATS consumer names, Redis rolling windows, ClickHouse loader mechanics, UI labels, and cache policy.
|
||||
|
||||
## Area Classification
|
||||
- Existing replay architecture: **refactor**. Keep event-time merge and stream publishing; add generated-stream sources, run IDs, manifests, and deterministic output comparison.
|
||||
- Event schemas: **refactor**. Keep canonical raw/derived event shapes; add provenance metadata and separate label/manifest schemas.
|
||||
- Service boundaries: **refactor**. Move generator logic out of ingest adapters into a package; adapters become thin emitters.
|
||||
- Test structure: **redesign**. Current tests are unit-heavy and adapter-local; add fixture manifests, golden outputs, and batch replay checks.
|
||||
- ClickHouse fixture strategy: **refactor**. Keep storage helpers; add run-scoped fixture loaders and optional run metadata, not permanent synthetic clone tables.
|
||||
- NATS/JetStream: **keep/refactor**. Keep canonical subjects for production behavior; support isolated subject prefixes or disposable streams for tests/load.
|
||||
- Redis baseline interaction: **refactor**. Keep Redis for live rolling state; golden tests should use in-memory/resettable baselines.
|
||||
- UI/demo needs: **refactor**. Keep replay UI and synthetic admin rail; add named deterministic demo modes and scenario selectors.
|
||||
- CI feasibility: **keep/refactor**. Keep fast Bun CI; make synthetic package/golden tests infra-free and defer Docker integration to a separate job.
|
||||
|
||||
## Option A — Conservative
|
||||
- Summary: wrap the current synthetic ingest adapters with minimal metadata, a small fixture CLI, and a few golden tests.
|
||||
- Pros: fastest, least migration, preserves current demos.
|
||||
- Cons: determinism remains mixed with wall-clock timers and live adapter behavior; labels/manifests stay bolted on.
|
||||
- Complexity: low to medium. Migration risk: low.
|
||||
- Better: quick smoke fixtures, basic provenance, modest replay demos.
|
||||
- Worse: long-term generator quality, test reliability, scenario authoring.
|
||||
- Kept: current ingest adapters, bus/storage/API/web mostly unchanged.
|
||||
- Rewritten: small parts of synthetic adapters and tests.
|
||||
- Deleted/deferred: deep replay refactor, new package boundary, batch harness.
|
||||
- PR sequence: add metadata schemas; add CLI wrapper; add fixture files; add basic replay filters; add initial golden tests.
|
||||
|
||||
## Option B — Refactor
|
||||
- Summary: create `@islandflow/synthetic-market` as the deterministic engine; make adapters, CLI, replay, tests, and demos consume it.
|
||||
- Pros: deterministic by design, reusable, testable, demo-friendly, preserves the working stack.
|
||||
- Cons: more up-front movement; current adapter logic must be untangled.
|
||||
- Complexity: medium. Migration risk: medium-low.
|
||||
- Better: seeded runs, profiles, labels, manifests, replay, golden tests, load profiles.
|
||||
- Worse: short-term churn and some duplicated paths during migration.
|
||||
- Kept: canonical event schemas, NATS subjects, ClickHouse helpers, compute classifiers, API replay endpoints, web replay shell.
|
||||
- Rewritten: synthetic options/equities adapters, synthetic control state, replay source abstraction, tests around synthetic scenarios.
|
||||
- Deleted/deferred: adapter-local scenario catalog after migration; full LOB/agent/ML simulation.
|
||||
- PR sequence: add package and schemas; move current generators behind deterministic API; add CLI manifest generation; refactor adapters to consume package; add replay synthetic source/run filters; add golden fixture tests; add demo selector.
|
||||
|
||||
## Option C — Redesign
|
||||
- Summary: rebuild around a unified deterministic event-log architecture where generation, replay, live demo, storage, and tests all consume run-partitioned event logs.
|
||||
- Pros: cleanest long-term model; excellent determinism, provenance, and replay semantics.
|
||||
- Cons: too much rebuild for pre-alpha; delays product learning.
|
||||
- Complexity: high. Migration risk: high.
|
||||
- Better: architecture purity, reproducible environments, run isolation.
|
||||
- Worse: delivery speed, disruption, operational risk.
|
||||
- Kept: some compute/classifier/domain logic and UI concepts.
|
||||
- Rewritten: replay, ingest, storage partitioning, bus topology, fixture/test harness.
|
||||
- Deleted/deferred: current synthetic adapters, current replay service shape, much of current live/demo plumbing.
|
||||
- PR sequence: define event log/envelope; implement generator; rebuild replay; rebuild storage materialization; port compute; port API/UI; retire old ingest paths.
|
||||
|
||||
## Recommendation
|
||||
Choose **Option B**. Bluntly: Option A is a patch, and it will keep producing impressive-looking but untrustworthy demos. Option C is architecture vanity for a pre-alpha product. Option B is the grown-up move: extract the generator into a deterministic package, keep the useful event pipeline, and make replay/tests/demos consume the same generated runs.
|
||||
|
||||
## Test Plan
|
||||
- Unit: PRNG determinism, profile normalization, tick validity, quote/trade invariants, option chain sparsity, label/manifest schema parsing.
|
||||
- Golden: fixed seed plus manifest produces byte/hash-stable raw events and stable smart-money/alert signatures.
|
||||
- Replay: synthetic source ordering matches manifest; derived outputs match expected-output manifest.
|
||||
- Integration: optional NATS/ClickHouse run-scoped fixture test behind a non-default CI job.
|
||||
- Demo/load: named demo profiles render in replay UI; load profile scales rates without changing event semantics.
|
||||
|
||||
## Assumptions
|
||||
- MVP remains no-history-first.
|
||||
- Canonical real event schemas remain the pipeline contract.
|
||||
- Hidden labels are never embedded directly in market events.
|
||||
- Infra-backed tests are useful, but the first synthetic quality gate must pass in plain `bun test`.
|
||||
Loading…
Add table
Add a link
Reference in a new issue