dirtydishes/islandflow
1
0
Fork 0
Code Issues 2 Pull requests 4 Projects Releases Packages Wiki Activity Actions 1

Add Databento historical replay adapter + symbol mapping; speed up replay UI + completion state

Browse source 
- add Databento options adapter (TS) with Python sidecar and env wiring
- add  to stream historical trades and resolve instrument_id -> raw_symbol via symbology
- include Databento + typing_extensions in ingest-options Python requirements
- expose Databento env settings in ingest-options index (dataset/schema/start/end/stype/limit/price scale/python bin)
- update README with Databento replay usage and env docs
- speed up UI replay polling/drain, add per-card replay time display
- stop replay at end and prevent fallback to synthetic by pinning replay to initial trace source
This commit is contained in:
dirtydishes 2025-12-28 21:30:24 -05:00
parent 6dc279099f
commit baaadcf105
6 changed files with 799 additions and 33 deletions
Download patch file Download diff file Expand all files Collapse all files

14
README.md
View file

@ -17,7 +17,7 @@ Done now (in repo):
- UI: live tapes for options/equities/flow + replay toggle + pause controls
In progress / blocked:
- Real data adapters (requires licensed data source)
- Live data adapters (requires licensed data source)
- Rolling stats and advanced clustering
Not started:
@ -36,12 +36,13 @@ Not started:
## Current Capabilities
- Synthetic options/equity prints with deterministic sequencing
- Ingest adapter seam (env-selected, default `synthetic`) for options and equities
- Ingest adapter seam (env-selected; options default `alpaca`, equities default `synthetic`)
- Raw event persistence in ClickHouse + streaming via NATS JetStream
- Deterministic option FlowPacket clustering (time-window)
- API gateway with REST, WS, and replay endpoints
- UI tapes for options/equities/flow packets with live/replay toggle and pause controls
- Alpaca options adapter (dev-only) with bounded contract selection
- Databento historical replay adapter (options, Python sidecar)
## Planned Capabilities (from PLAN.md)
@ -113,6 +114,15 @@ Alpaca selection policy (dev-only, deterministic):
- For each expiry, select 8 strikes per side closest to ATM within ±6% (fallback to ±10% if needed)
- Subscriptions are built once at startup to keep the stream bounded and repeatable
Databento historical replay adapter (options, via Python `databento`):
- Install Python deps: `python3 -m pip install -r services/ingest-options/py/requirements.txt`
- Set `INGEST_ADAPTER=databento` and configure:
- `DATABENTO_API_KEY`, `DATABENTO_START` (ISO date/time)
- Optional: `DATABENTO_END`, `DATABENTO_DATASET` (default `OPRA.PILLAR`), `DATABENTO_SCHEMA` (default `trades`)
- Optional: `DATABENTO_SYMBOLS` (`ALL` or comma list), `DATABENTO_STYPE_IN`/`DATABENTO_STYPE_OUT` (default `raw_symbol`)
- Optional: `DATABENTO_LIMIT` (record cap), `DATABENTO_PRICE_SCALE` (divide raw price), `DATABENTO_PYTHON_BIN`
- This adapter replays historical data only; live capture will be added later.
Run tests:
- `bun test`