Real-Time Options Flow & Off-Exchange Analysis

This repository contains a Bun + TypeScript monorepo for a personal-use, event-sourced market microstructure research platform focused on:

• options prints + NBBO,
• off-exchange equity prints,
• explainable rule-based flow classification,
• deterministic replay,
• evidence-linked UI inspection.

Current Implementation Status

Implemented now:

• Bun workspaces with shared packages for schemas, bus, config, observability, and ClickHouse access.
• Infra orchestration via Docker Compose (NATS JetStream, ClickHouse, Redis).
• Options ingest service with adapters:
  • synthetic stream,
  • Alpaca options (dev-focused, bounded contracts),
  • IBKR bridge (Python sidecar),
  • Databento historical replay adapter (Python sidecar).
• Equities ingest service with adapters:
  • synthetic stream,
  • Alpaca equities trades/quotes.
• Compute service:
  • deterministic option print clustering into FlowPackets,
  • NBBO join quality features and aggressor-mix metrics,
  • rolling baselines in Redis,
  • structure summarization and structure packet emission,
  • rule-based classifiers + confidence-scored alert events,
  • dark-style inferred events from equity prints/quotes,
  • equity print-to-quote join events.
• Candles service:
  • server-side equity candle aggregation,
  • ClickHouse persistence,
  • optional Redis hot cache,
  • NATS publication.
• Replay service:
  • deterministic republishing from ClickHouse to NATS,
  • multi-stream merge with stable tie-break ordering,
  • speed/start/end controls.
• API service:
  • REST endpoints for recent + cursor pagination,
  • REST range endpoints for chart windows,
  • REST replay-oriented endpoints,
  • WebSocket channels for options, NBBO, equities, quotes, joins, flow, classifier hits, alerts, inferred dark, and candles.
• Next.js web app:
  • live tape/workspace views,
  • replay controls and status,
  • signals and chart-focused routes,
  • evidence-centric terminal UI.
• Refdata + EOD enricher service entrypoints are present but currently scaffolds (lifecycle/logging only).

Planned / not yet complete:

• production-grade licensed feed integrations and entitlement workflow,
• richer refdata/corp-action enrichment,
• secure deployment/auth hardening,
• deeper structure + calibration workflows from PLAN.md.

Core Principles

• Explainability first — inferred outputs are evidence-backed and human-readable.
• Event sourcing — raw and derived events persist to support replay.
• Determinism — replay behavior tracks live pipeline logic.
• Microstructure awareness — bounded joins, confidence scoring, and explicit uncertainty.
• Bun-first tooling — runtime/package/scripts all use Bun.

Monorepo Layout

• apps/web — Next.js UI shell/routes.
• services/ingest-options — options print/NBBO ingest adapters.
• services/ingest-equities — equity print/quote ingest adapters.
• services/compute — clustering, structures, classifiers, alerts, inferred dark.
• services/candles — server-side candle aggregation + cache.
• services/replay — ClickHouse → NATS replay streamer.
• services/api — REST + WebSocket gateway.
• services/refdata — scaffold service.
• services/eod-enricher — scaffold service.
• packages/types — shared event schemas/types.
• packages/storage — ClickHouse tables/queries.
• packages/bus — NATS/JetStream helpers.
• packages/config — env parsing.
• packages/observability — logger + metrics facade.

Build and Run

Install dependencies:

• bun install

Start infrastructure only:

• docker compose up -d

Create env file:

• copy .env.example to .env and set provider credentials as needed.

Start infra + all services + web:

• bun run dev

Start services only (assumes infra is already running):

• bun run dev:services

Start web only:

• bun run dev:web

Environment Configuration

All runtime configuration comes from .env.

Core infrastructure

• NATS_URL (default nats://127.0.0.1:4222)
• CLICKHOUSE_URL (default http://127.0.0.1:8123)
• CLICKHOUSE_DATABASE (default default)
• REDIS_URL (default redis://127.0.0.1:6379)

Ingest adapter selection

• OPTIONS_INGEST_ADAPTER (synthetic | alpaca | ibkr | databento)
• EQUITIES_INGEST_ADAPTER (synthetic | alpaca)
• EMIT_INTERVAL_MS (synthetic emit cadence)
• SYNTHETIC_MARKET_MODE (realistic | active | firehose, default realistic)
• SYNTHETIC_OPTIONS_MODE (optional per-service override; falls back to SYNTHETIC_MARKET_MODE)
• SYNTHETIC_EQUITIES_MODE (optional per-service override; falls back to SYNTHETIC_MARKET_MODE)

Synthetic mode profiles

• realistic is the default local mode. Options produce materially more ordinary prints, fewer repeated bursts, and fewer alert-driving sweeps/spikes. Equities produce smaller batches and less relentless off-exchange activity.
• active is a busier demo mode that still leaves meaningful visible history in the UI.
• firehose is the stress profile for backpressure, hot-window eviction, and Databento-readiness validation.

Options adapter settings

• Alpaca: ALPACA_KEY_ID, ALPACA_SECRET_KEY, ALPACA_REST_URL, ALPACA_WS_BASE_URL, ALPACA_FEED, ALPACA_UNDERLYINGS, ALPACA_STRIKES_PER_SIDE, ALPACA_MAX_DTE_DAYS, ALPACA_MONEYNESS_PCT, ALPACA_MONEYNESS_FALLBACK_PCT, ALPACA_MAX_QUOTES
• Databento: DATABENTO_API_KEY, DATABENTO_DATASET, DATABENTO_SCHEMA, DATABENTO_NBBO_SCHEMA, DATABENTO_START, DATABENTO_END, DATABENTO_SYMBOLS, DATABENTO_STYPE_IN, DATABENTO_STYPE_OUT, DATABENTO_LIMIT, DATABENTO_PRICE_SCALE, DATABENTO_PYTHON_BIN
• IBKR: IBKR_HOST, IBKR_PORT, IBKR_CLIENT_ID, IBKR_SYMBOL, IBKR_EXPIRY, IBKR_STRIKE, IBKR_RIGHT, IBKR_EXCHANGE, IBKR_CURRENCY, IBKR_PYTHON_BIN

Equities adapter settings

• ALPACA_EQUITIES_FEED (iex or sip)

Compute / classifiers / inference

• Delivery and windowing: COMPUTE_DELIVER_POLICY, COMPUTE_CONSUMER_RESET, NBBO_MAX_AGE_MS, ROLLING_WINDOW_SIZE, ROLLING_TTL_SEC
• Classifiers: CLASSIFIER_SWEEP_MIN_PREMIUM, CLASSIFIER_SWEEP_MIN_COUNT, CLASSIFIER_SWEEP_MIN_PREMIUM_Z, CLASSIFIER_SPIKE_MIN_PREMIUM, CLASSIFIER_SPIKE_MIN_SIZE, CLASSIFIER_SPIKE_MIN_PREMIUM_Z, CLASSIFIER_SPIKE_MIN_SIZE_Z, CLASSIFIER_Z_MIN_SAMPLES, CLASSIFIER_MIN_NBBO_COVERAGE, CLASSIFIER_MIN_AGGRESSOR_RATIO, CLASSIFIER_0DTE_MAX_ATM_PCT, CLASSIFIER_0DTE_MIN_PREMIUM, CLASSIFIER_0DTE_MIN_SIZE
• Dark inference: EQUITY_QUOTE_MAX_AGE_MS, DARK_INFER_WINDOW_MS, DARK_INFER_COOLDOWN_MS, DARK_INFER_MIN_BLOCK_SIZE, DARK_INFER_MIN_ACCUM_SIZE, DARK_INFER_MIN_ACCUM_COUNT, DARK_INFER_MIN_PRINT_SIZE, DARK_INFER_MAX_EVIDENCE, DARK_INFER_MAX_SPREAD_PCT

Options signal filtering

• OPTIONS_SIGNAL_MODE (smart-money | balanced | all, default smart-money)
• OPTIONS_SIGNAL_MIN_NOTIONAL (default 10000)
• OPTIONS_SIGNAL_ETF_MIN_NOTIONAL (default 50000)
• OPTIONS_SIGNAL_BID_SIDE_MIN_NOTIONAL (default 25000)
• OPTIONS_SIGNAL_MID_MIN_NOTIONAL (default 20000)
• OPTIONS_SIGNAL_NBBO_MAX_AGE_MS (default 1500)
• OPTIONS_SIGNAL_ETF_UNDERLYINGS (default SPY,QQQ,IWM,DIA,TLT,GLD,SLV,XLF,XLE,XLV,XLI,XLP,XLU,XLY,SMH,ARKK)

Default smart-money behavior:

• reject sub-10k options prints,
• reject ETF prints below 50k,
• reject B / BB prints below 25k,
• reject non-SWEEP / non-ISO MID prints below 20k,
• require 50k when NBBO is missing or stale,
• auto-keep 100k+,
• keep ask-side A / AA prints at 10k+,
• keep SWEEP / ISO prints at 25k+,
• keep 500+ contract prints at 10k+.

balanced uses the same shape with lower thresholds. all marks every option print as signal-passing.

Candles

• CANDLE_INTERVALS_MS, CANDLE_MAX_LATE_MS, CANDLE_CACHE_LIMIT, CANDLE_DELIVER_POLICY, CANDLE_CONSUMER_RESET

API

• API_PORT, REST_DEFAULT_LIMIT, API_DELIVER_POLICY, API_CONSUMER_RESET
• LIVE_LIMIT_OPTIONS, LIVE_LIMIT_NBBO, LIVE_LIMIT_EQUITIES, LIVE_LIMIT_EQUITY_JOINS, LIVE_LIMIT_FLOW, LIVE_LIMIT_CLASSIFIER_HITS, LIVE_LIMIT_ALERTS, LIVE_LIMIT_INFERRED_DARK (bounded live generic cache depths; defaults 10000, max 100000)

Web live retention

• NEXT_PUBLIC_LIVE_HOT_WINDOW (frontend hot live window cap for non-options feeds; default 2000)
• NEXT_PUBLIC_LIVE_HOT_WINDOW_OPTIONS (frontend hot live window cap for options prints; default 25000)
• NEXT_PUBLIC_PINNED_EVIDENCE_TTL_MS (pinned evidence TTL; default 1200000)
• NEXT_PUBLIC_PINNED_EVIDENCE_MAX_ITEMS (pinned evidence cache guardrail; default 4000)
• NEXT_PUBLIC_FLOW_FILTER_PRESET (smart-money | balanced | all, default smart-money)

Replay service

• REPLAY_ENABLED, REPLAY_STREAMS, REPLAY_START_TS, REPLAY_END_TS, REPLAY_SPEED, REPLAY_BATCH_SIZE, REPLAY_LOG_EVERY

Testing-mode throttling

• TESTING_MODE
• TESTING_THROTTLE_MS

Quick Notes

• Python dependencies are required only for IBKR/Databento sidecars (services/ingest-options/py/requirements.txt).
• Candle construction is server-side; the client consumes prebuilt OHLC events.
• Option prints now persist as enriched raw rows and can be queried as either:
  • view=signal — default live/UI path and compute input.
  • view=raw — audit/debug path that preserves every stored print.
• The default Tape page options/packets posture is now stock-only, hides B / BB, keeps calls and puts visible, and applies in-memory min-notional controls immediately.
• Live retention uses a two-tier model:
  • API/Redis maintain a bounded hot cache per live generic channel.
  • UI keeps a bounded hot window for rendering performance around the signal view rather than raw noise.
  • Options prints can use a deeper dedicated cap via NEXT_PUBLIC_LIVE_HOT_WINDOW_OPTIONS without raising every other feed.
  • Alert/drawer evidence is pinned and hydrated by id/trace so details remain inspectable after hot-window eviction.
• Firehose-readiness strategy:
  • preserve raw ingest for storage/replay,
  • feed compute and default live UI from the filtered signal path,
  • add filterable live subscription contracts now so selective delivery can move server-side without reshaping the protocol later.
• This repository is for personal, non-redistributed usage.

Useful Examples

Realistic local demo:

SYNTHETIC_MARKET_MODE=realistic \
OPTIONS_SIGNAL_MODE=smart-money \
bun run dev

Active demo:

SYNTHETIC_MARKET_MODE=active bun run dev

Firehose stress test:

SYNTHETIC_MARKET_MODE=firehose \
NEXT_PUBLIC_LIVE_HOT_WINDOW=2000 \
bun run dev

Show raw options flow for debugging:

/prints/options?view=raw&security=all
/history/options?view=raw&security=all&before_ts=<ts>&before_seq=<seq>
/replay/options?view=raw&security=all&after_ts=<ts>&after_seq=<seq>