diff --git a/docs/implementation/index.html b/docs/implementation/index.html deleted file mode 100644 index bfb9380..0000000 --- a/docs/implementation/index.html +++ /dev/null @@ -1,543 +0,0 @@ - - - - - - Implementation Phase Plans - - - -
-
-
-

Implementation Map

-

Implementation Phase Plans

-

- The active planning layer for synthetic market-data and smart-money/smart-flow architecture work. - Architecture reviews and research reports are background; phase documents and Beads issues define execution scope. -

-
- Active scope guide - Synthetic market data - Smart flow -
-
- -
- -
-

Jump to

- -
- -
-

Document Precedence

-
-
-
    -
  1. Current Beads issue
    2. -
  2. Referenced phase document under docs/implementation/
    3. -
  3. Architecture plan under docs/plans/
    4. -
  4. Research report under docs/research-docs/
    5. -
-

- This repository uses docs/research-docs/ for research reports; docs/research/ - is not present. Research reports provide rationale and useful constraints, but they do not add active - implementation scope unless that scope is explicitly pulled into a phase document and Beads issue. -

-
-
-
- -
-

Source Plans

-
- - -
-
- -
-

Planning Rules

-
-
-
    -
  • Prefer small, reviewable PRs.
    • -
  • Do not implement an entire architecture plan at once.
    • -
  • Use Beads issues for execution tracking and dependency management.
    • -
  • Keep durable architecture and phase detail in these docs, not in long Beads descriptions.
    • -
  • Synthetic data must emit canonical market event types, not synthetic-only pipeline event types.
    • -
  • Synthetic labels must remain separate from emitted market events.
    • -
  • Smart-flow logic must distinguish facts, evidence, hypotheses, confidence, and abstention.
    • -
  • Historical calibration is future work, not an MVP dependency.
    • -
  • Early synthetic tests must not require Docker, ClickHouse, NATS, or Redis.
    • -
  • Synthetic foundations should come before demos, UI controls, or live service work.
    • -
-
-
-
- -
-

Beads Map

-
- - - - - - - - - - - - - - - - - - - - -
StreamEpicRoadmap
Synthetic market dataislandflow-259 - Plan synthetic market-data implementation phasesdocs/implementation/synthetic-market-data/00-roadmap.html
Smart money / smart flowislandflow-zxh - Plan smart-money to smart-flow implementation phasesdocs/implementation/smart-money/00-roadmap.html
-
-
- -
-

Dependency Order

-
- - - - - - - - - - - - - - - - - - - - - -
OrderPhaseBeads issueBlocks next because
1ASynthetic deterministic spineislandflow-259.1Establishes seeded raw event generation and provenance assumptions for later synthetic work.
1BSmart-flow contracts and vocabularyislandflow-zxh.1Can safely run in parallel with synthetic phase 01; defines evidence/hypothesis language before scoring work.
2Synthetic manifests, fixtures, and CLIislandflow-259.2Evidence clustering needs deterministic fixtures before broad behavior changes.
3Smart-flow evidence clustering and featuresislandflow-zxh.2Scenario labels need the evidence vocabulary they are expected to exercise.
4Synthetic scenarios, labels, and expected outputsislandflow-259.3Hypothesis scoring needs labeled positive, negative, and abstention cases.
5Smart-flow hypothesis scoring and abstentionislandflow-zxh.3Synthetic replay integration should validate the derived hypothesis pipeline.
6Synthetic replay integrationislandflow-259.4Smart-flow golden tests need replayable synthetic runs.
7Smart-flow replay evaluation and golden testsislandflow-zxh.4Demos should wait until replay proves the semantics.
8Synthetic demo and load profilesislandflow-259.5API/UI explainability should show stable, named, deterministic runs.
9Smart-flow API/UI explainabilityislandflow-zxh.5Final MVP presentation layer after the evidence pipeline is validated.
-
-
- -
-

Future Work

-
-
-
-

Synthetic historical calibration

-

islandflow-259.6 depends on synthetic phase 05, but is not required for MVP.

-
-
-
-
-

Smart-flow calibration

-

islandflow-zxh.6 depends on smart-flow phase 05 and synthetic future calibration, but is not required for MVP.

-
-
-
-
- - - - -
- - diff --git a/docs/implementation/smart-money/00-roadmap.html b/docs/implementation/smart-money/00-roadmap.html deleted file mode 100644 index a783b79..0000000 --- a/docs/implementation/smart-money/00-roadmap.html +++ /dev/null @@ -1,502 +0,0 @@ - - - - - - Smart Money / Smart Flow Roadmap - - - -
-
-
-

Smart Flow Roadmap

-

Smart Money / Smart Flow Roadmap

-

- Implementation-sized phases for turning smart-money detection into smart-flow inference: - observations, evidence clusters, cautious hypotheses, confidence, alternatives, abstention, - replay evaluation, and user-facing insight projections. -

-
- Beads islandflow-zxh - Recommended: Option B - MVP before calibration -
-
- -
- -
-

Jump to

- -
- - - -
-

Core Constraints

-
-
-
    -
  • Do not treat "smart money" as a canonical fact emitted by the system.
    • -
  • Distinguish direct facts, evidence, hypotheses, confidence, alternatives, and abstention.
    • -
  • Preserve evidence and uncertainty in storage, API, websocket, and UI surfaces.
    • -
  • Keep Redis as hot cache only, not hidden baseline truth.
    • -
  • Make replay evaluation the acceptance gate before expanding UI confidence.
    • -
  • Keep historical or research-grade calibration as future work, not an MVP dependency.
    • -
-
-
-
- -
-

Phase Sequence

-
- - - - - - - - - - - - - - - - - -
PhaseBeads issueDepends onPurpose
01 - Contracts and vocabularyislandflow-zxh.1None; safe parallel with islandflow-259.1Define evidence/hypothesis/insight contracts and retire canonical overconfidence.
02 - Evidence clustering and featuresislandflow-zxh.2islandflow-259.2Extract eligibility, evidence facts, clusters, and traceable features.
03 - Hypothesis scoring and abstentionislandflow-zxh.3islandflow-259.3Score cautious hypotheses and represent abstention/alternatives.
04 - Replay evaluation and golden testsislandflow-zxh.4islandflow-259.4Validate derived outputs through deterministic replay and golden fixtures.
05 - API/UI explainabilityislandflow-zxh.5islandflow-259.5Expose evidence-backed insights and uncertainty to API, WS, and UI.
99 - Future calibrationislandflow-zxh.6islandflow-zxh.5, islandflow-259.6Calibrate confidence and policy behavior later with richer datasets.
-
-
- -
-

PR Split Notes

-
-
- Phase 02a -

islandflow-zxh.2.1 - Eligibility and evidence facts

-

Split out the direct fact and eligibility layer before clustering and feature vector work.

-
-
- Phase 02b -

islandflow-zxh.2.2 - Clustering and feature vectors

-

Keep clustering and feature vector changes reviewable after the evidence vocabulary exists.

-
-
- Phase 03a -

islandflow-zxh.3.1 - Hypothesis score vectors

-

Build scoring as a separate semantic layer, not as UI-ready certainty.

-
-
- Phase 03b -

islandflow-zxh.3.2 - Abstention and insight projection

-

Represent alternatives, penalties, and abstention before exposing user-facing insight projections.

-
-
- Phase 05a -

islandflow-zxh.5.1 - Evidence API and websocket surfaces

-

Expose evidence-backed contracts through transport before tuning the presentation layer.

-
-
- Phase 05b -

islandflow-zxh.5.2 - UI explainability surfaces

-

Show evidence quality, confidence vs conviction, alternatives, abstention, and catalyst/noise context.

-
-
-

- If an implementation PR crosses contracts, compute, storage, API, and UI in one change, stop and split it. -

-
- -
-

Matching Beads Epic

-
-
-

islandflow-zxh - Plan smart-money to smart-flow implementation phases.

-
-
-
- - -
- - diff --git a/docs/implementation/smart-money/index.html b/docs/implementation/smart-money/index.html index 7e6bfda..0c6ba0e 100644 --- a/docs/implementation/smart-money/index.html +++ b/docs/implementation/smart-money/index.html @@ -411,9 +411,6 @@ tr:last-child td { border-bottom: 0; } Beads islandflow-zxh 7 source docs No app code - Implementation overview - Roadmap HTML - Architecture HTML diff --git a/docs/implementation/synthetic-market-data/00-roadmap.html b/docs/implementation/synthetic-market-data/00-roadmap.html deleted file mode 100644 index 71978eb..0000000 --- a/docs/implementation/synthetic-market-data/00-roadmap.html +++ /dev/null @@ -1,481 +0,0 @@ - - - - - - Synthetic Market-Data Roadmap - - - -
-
-
-

Synthetic Roadmap

-

Synthetic Market-Data Roadmap

-

- Implementation-sized phases for extracting deterministic synthetic generation into a first-class reusable engine - while keeping the useful NATS, ClickHouse, compute, API, replay, and web stack. -

-
- Beads islandflow-259 - Recommended: Option B - Infra-free early gates -
-
- -
- -
-

Jump to

- -
- - - -
-

Core Constraints

-
-
-
    -
  • Emit canonical market event types: OptionPrint, OptionNBBO, EquityPrint, and EquityQuote.
    • -
  • Do not create synthetic-only market event types for the main pipeline.
    • -
  • Keep hidden ground-truth labels separate from emitted market events.
    • -
  • Keep early quality gates infra-free: bun test should not require Docker, ClickHouse, NATS, or Redis.
    • -
  • Build deterministic foundations before demos, UI controls, or live synthetic service behavior.
    • -
  • Treat historical calibration as future work, not as a dependency for the MVP synthetic generator.
    • -
-
-
-
- -
-

Phase Sequence

-
- - - - - - - - - - - - - - - - - -
PhaseBeads issueDepends onPurpose
01 - Deterministic spineislandflow-259.1NoneCreate the seeded generation foundation and canonical event output contract.
02 - Manifests, fixtures, CLIislandflow-259.2islandflow-zxh.1Turn deterministic generation into durable fixtures and manifests.
03 - Scenarios, labels, expected outputsislandflow-259.3islandflow-zxh.2Author named scenarios, separate labels, and expected derived outputs.
04 - Replay integrationislandflow-259.4islandflow-zxh.3Make replay consume synthetic runs with stable ordering and output comparison.
05 - Demo and load profilesislandflow-259.5islandflow-zxh.4Expose named deterministic demo/load profiles after replay validation.
99 - Future historical calibrationislandflow-259.6islandflow-259.5Calibrate parameters from historical data later, after the MVP is stable.
-
-
- -
-

PR Split Notes

-
-
- Phase 03a -

islandflow-259.3.1 - Scenario catalog and labels

-

Keep scenario authoring and ground-truth label shape focused before expected-output comparison grows around it.

-
-
- Phase 03b -

islandflow-259.3.2 - Expected-output manifests

-

Store expected derived outputs as reviewable artifacts for downstream smart-flow validation.

-
-
-

- If any other phase starts touching unrelated service, API, UI, and storage behavior in one PR, split it before implementation continues. -

-
- -
-

Matching Beads Epic

-
-
-

islandflow-259 - Plan synthetic market-data implementation phases.

-
-
-
- - -
- - diff --git a/docs/implementation/synthetic-market-data/index.html b/docs/implementation/synthetic-market-data/index.html index e85f553..dcb2262 100644 --- a/docs/implementation/synthetic-market-data/index.html +++ b/docs/implementation/synthetic-market-data/index.html @@ -411,9 +411,6 @@ tr:last-child td { border-bottom: 0; } Beads islandflow-259 7 source docs No app code - Implementation overview - Roadmap HTML - Architecture HTML diff --git a/docs/plans/smart-flow-architecture-review.html b/docs/plans/smart-flow-architecture-review.html deleted file mode 100644 index 826fa34..0000000 --- a/docs/plans/smart-flow-architecture-review.html +++ /dev/null @@ -1,814 +0,0 @@ - - - - - - Smart Flow Architecture Review - - - -
-
-
-

Plan Document

-

Evidence-Backed Smart-Flow Detection

-

- A readable architecture review for reshaping Islandflow's smart-flow system around direct observation, - evidence clusters, cautious hypotheses, preserved uncertainty, and replayable validation. -

-
- Smart flow - Architecture review - Refactor, not rewrite - Recommended: Option B - Roadmap HTML - Phase dossier -
-
- -
- -
-

Jump to

- -
- -
-

Summary

-
-
-

- No source code was modified as part of the architecture review. The conclusion is direct: - the current architecture is not suitable as-is, but it is close enough to refactor. - The stack is right; the domain language and pipeline shape are not. -

-

- The research direction should be direct observation to inference to hypothesis, with preserved - evidence and visible uncertainty. The system should stop emitting "smart money" as if it is a - fact, and instead emit cautious, explainable smart-flow hypotheses. -

-
-
-
- -
-

Source Documents

-
- - - -
-
- -
-

Area Classification

-
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
AreaCallArchitecture Review
Domain modelrefactorGood bones, wrong center. Make evidence, hypotheses, scores, and alternatives first-class.
Event taxonomyrefactorRaw/derived split is good; smart_money, dark.inferred, and classifier_hits leak overconfident product language.
Service boundariesrefactorIngest does too much signal policy; compute is too broad. Split pipeline stages before adding more intelligence.
FlowPacketrefactorKeep concept, rename/reframe as FlowEvidenceCluster or FlowCandidate. Not a product domain object.
SmartMoneyEventredesignReplace canonical object with FlowHypothesisEvent; use SmartFlowInsight only as UI/API projection.
Classifier pipelineredesignCurrent rules mix evidence extraction, hypothesis scoring, narrative labels, and alerting. Needs staged outputs.
ClickHouse/storagerefactorRight datastore; raw tables are decent, derived evidence/hypotheses need typed/queryable columns plus JSON sidecars.
Redis baselines/cacherefactorRight hot-state role; wrong as hidden baseline truth. Baselines need replayable snapshots/versioning.
NATS/JetStream subjectsrefactorRight bus; subjects should express stage/version: observations, evidence, hypotheses, insights.
Replay determinismredesignPresent but not central enough. Replay must be the acceptance gate for derived outputs.
API/WebSocketrefactorMechanics are good; public surface should expose evidence bundles and hypotheses, not internal legacy names.
UI evidence modelrefactorDirectionally good, but still foregrounds profile/probability over evidence quality, alternatives, and uncertainty.
Test strategyredesignUnit tests are solid scaffolding; needs fixture replay, false-positive suites, calibration, and end-to-end determinism.
-
-
- -
-

Direct Answers

-
-
-
    -
  1. 01

    Current suitability: no. Useful infrastructure, but not yet an evidence-backed smart-flow architecture.

    2. -
  2. 02

    SmartMoneyEvent: not a good canonical domain object. Use FlowHypothesisEvent. ParticipantHypothesisEvent implies participant identity too strongly. SmartFlowInsight should be a user-facing projection.

    3. -
  3. 03

    FlowPacket: not as named. Keep the abstraction as an internal evidence cluster, rename to FlowEvidenceCluster or FlowCandidate.

    4. -
  4. 04

    Service boundaries: not right. Ingest should normalize only; evidence quality, eligibility, clustering, hypothesis scoring, and insight projection should be separate stages.

    5. -
  5. 05

    ClickHouse/Redis/NATS roles: yes broadly. ClickHouse is the authoritative event/audit store. Redis is hot cache only. NATS is transport, not truth. All three need cleaner contracts.

    6. -
  6. 06

    Replay central enough: no. It should be how every detection change proves itself.

    7. -
  7. 07

    UI uncertainty: partially. It shows evidence refs, profile ladders, abstention, and suppression, but needs confidence vs conviction, alternative explanations, evidence quality, and why-not signals.

    8. -
  8. 08

    First-class domain objects: raw observations, execution context, quote join, eligibility decision, evidence cluster, structure hypothesis, evidence quality score, baseline snapshot, hypothesis score vector, false-positive penalty, catalyst context, flow hypothesis event, smart-flow insight, replay run.

    9. -
  9. 09

    Implementation details: Redis list layout, durable consumer names, current classifier thresholds, ClickHouse batch writer, adapter internals, legacy ClassifierHitEvent, alert severity math, UI cache mechanics.

    10. -
  10. 10

    Delete/defer: canonical smart-money naming, real-time dark-pool certainty, standalone whale-premium alerts, trade-level open/close claims, participant identity claims, simplistic premium alert score, ingest-time signal filtering, retail_whale as a canonical profile unless reframed as attention/lottery flow.

    11. -
-
-
-
- -
-

Objects to Make First-Class

-
- Raw Observation - Execution Context - Quote Join - Eligibility Decision - Flow Evidence Cluster - Structure Hypothesis - Evidence Quality Score - Baseline Snapshot - Hypothesis Score Vector - False-Positive Penalty - Catalyst Context - Flow Hypothesis Event - Smart-Flow Insight - Replay Run -
-
- -
-

Options

-
-
-
- Option A -

Conservative

-

Keep current objects and services; add evidence-quality fields, UI copy fixes, and replay tests.

-
-
-
    -
  • Pros

    Fastest, lowest migration risk, preserves current endpoints and UI.

    • -
  • Cons

    Leaves misleading canonical names and keeps inference tangled in compute.

    • -
  • Complexity

    Low.

    • -
  • Migration Risk

    Low.

    • -
-
    -
  1. Rename UI copy from smart money to smart flow candidate.
    2. -
  2. Add evidence-quality and alternative-explanation fields to existing event.
    3. -
  3. Add replay consistency tests around current outputs.
    4. -
  4. Add typed ClickHouse columns for high-value JSON fields.
    5. -
  5. Deprecate, but do not remove, legacy classifier hit display.
    6. -
-
-
- -
-
- Option B -

Refactor

-

Keep the stack and terminal UI, but rebuild the domain pipeline around evidence clusters and hypothesis events.

-
-
-
    -
  • Pros

    Fixes the product's epistemic spine without wasting useful infrastructure.

    • -
  • Cons

    Requires breaking contract migration across types, storage, compute, API, UI, and tests.

    • -
  • Complexity

    Medium-high.

    • -
  • Migration Risk

    Medium.

    • -
-
    -
  1. Introduce FlowEvidenceCluster, FlowHypothesisEvent, SmartFlowInsight, EvidenceQuality, and version fields with compatibility aliases.
    2. -
  2. Move signal eligibility out of ingest.
    3. -
  3. Split compute into evidence join, cluster/structure, hypothesis scoring, and insight/alert projection.
    4. -
  4. Replace derived JSON-only storage with typed query columns.
    5. -
  5. Add replay-run harness that recomputes derived outputs from raw streams.
    6. -
  6. Add /flow/evidence, /flow/hypotheses, /flow/insights, and WS equivalents.
    7. -
  7. Rework UI drawers/tables around evidence quality, confidence vs conviction, alternatives, abstention, and catalyst/noise context.
    8. -
  8. Add fixture suites for stale quotes, complex spreads, 0DTE/event noise, deep ITM, wide spreads, and off-exchange ambiguity.
    9. -
-
-
- -
-
- Option C -

Redesign

-

Start over with an event-sourced evidence engine and versioned, replayable policies.

-
-
-
    -
  • Pros

    Cleanest long-term architecture and strongest research discipline.

    • -
  • Cons

    Slowest, overkill before product fit, and discards too much working infrastructure.

    • -
  • Complexity

    Very high.

    • -
  • Migration Risk

    High.

    • -
-
    -
  1. Define new canonical event taxonomy and versioned policy registry.
    2. -
  2. Build raw observation lake and deterministic replay runner first.
    3. -
  3. Build evidence extraction and quote/condition eligibility services.
    4. -
  4. Build cluster and structure hypothesis services.
    5. -
  5. Build hypothesis scoring and calibration services.
    6. -
  6. Build insight projection API.
    7. -
  7. Rebuild terminal against new evidence/hypothesis contracts.
    8. -
  8. Backfill or discard old derived data.
    9. -
-
-
-
-
- -
-

Recommendation

-
-

Choose Option B.

-

- Option A is too timid for a pre-alpha product whose current names already fight the research. - Option C is intellectually clean but wastes too much working infrastructure. Option B keeps the - stack and terminal momentum while fixing the core mistake: treating smart money as a thing the - system emits, instead of treating smart flow as a cautious, evidence-backed hypothesis with alternatives. -

-

- The first implementation move should be the contract/naming PR: introduce - FlowHypothesisEvent and FlowEvidenceCluster with compatibility aliases, - then make replay the gate before touching more classifier logic. -

-
-
- - -
- - diff --git a/docs/plans/synthetic-market-data-architecture-review.html b/docs/plans/synthetic-market-data-architecture-review.html index 1823f52..d1a40db 100644 --- a/docs/plans/synthetic-market-data-architecture-review.html +++ b/docs/plans/synthetic-market-data-architecture-review.html @@ -521,8 +521,6 @@ Source: markdown review Mode: Plan Recommendation: Option B - Roadmap HTML - Phase dossier