Merge pull request #28 from dirtydishes/smart-money-classifiers
Ship smart-money event pipeline, event-calendar enrichment, and terminal UI
This commit is contained in:
dirtydishes
GitHub
commit
4368d6db4c
39 changed files with 3443 additions and 618 deletions
|
|
@ -4,5 +4,13 @@
|
|||
{"_type":"issue","id":"islandflow-ayo","title":"Drop stale backlog events from live fanout","description":"Follow-up to live freshness rollout: /ws/live was still fanning out stale backlog events for freshness-gated channels, which kept tape panes in Live feed behind despite active synthetic ingest. Gate fanout and cache ingest by freshness for options/nbbo/equities/flow.","status":"closed","priority":1,"issue_type":"bug","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T21:26:39Z","created_by":"dirtydishes","updated_at":"2026-04-28T21:26:44Z","started_at":"2026-04-28T21:26:44Z","closed_at":"2026-04-28T21:26:44Z","close_reason":"Completed","dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||
{"_type":"issue","id":"islandflow-0v6","title":"Fix tape freshness, NBBO coverage, pause controls, and filter popup","description":"Implement the tape fixes requested for synthetic options notional sizing, strict live freshness, live-mode pause/resume behavior, stronger NBBO snapshot coverage, and moving flow filters behind a popup. Includes server-side live cache changes, web terminal state/UI changes, and tests for synthetic pricing, live snapshot freshness/NBBO retention, and live pause/filter interactions.","status":"closed","priority":1,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T21:02:52Z","created_by":"dirtydishes","updated_at":"2026-04-28T21:13:38Z","started_at":"2026-04-28T21:02:57Z","closed_at":"2026-04-28T21:13:38Z","close_reason":"Completed","dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||
{"_type":"issue","id":"islandflow-e4r","title":"Implement smart-money flow filtering and synthetic firehose modes","description":"Implement the approved multi-surface plan for named synthetic market profiles, options raw-vs-signal filtering, live/API filter contracts, Tape page client-side flow filters, firehose-readiness improvements, tests, and README updates.","status":"closed","priority":1,"issue_type":"feature","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T20:10:49Z","created_by":"dirtydishes","updated_at":"2026-04-28T20:29:29Z","started_at":"2026-04-28T20:10:53Z","closed_at":"2026-04-28T20:29:29Z","close_reason":"Implemented synthetic market profiles, options signal-path filtering, signal-aware API/replay contracts, Tape page filters, tests, and README updates. Follow-up tracked in islandflow-biq.","dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||
{"_type":"issue","id":"islandflow-vbk","title":"Remove deprecated Alpaca key-pair auth","description":"Remove legacy Alpaca key-pair authentication support and keep ALPACA_API_KEY as the only supported auth method across options/equities ingest and docs.\n","status":"closed","priority":2,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-05T07:19:51Z","created_by":"dirtydishes","updated_at":"2026-05-05T07:21:10Z","started_at":"2026-05-05T07:19:54Z","closed_at":"2026-05-05T07:21:10Z","close_reason":"Removed key-pair auth and kept ALPACA_API_KEY only","dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||
{"_type":"issue","id":"islandflow-h47","title":"Support single-token Alpaca auth","description":"Support single-token Alpaca authentication across ingest adapters using ALPACA_API_KEY with fallback to ALPACA_KEY_ID/ALPACA_SECRET_KEY, and document env usage.\n","status":"closed","priority":2,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-05T07:12:22Z","created_by":"dirtydishes","updated_at":"2026-05-05T07:13:54Z","started_at":"2026-05-05T07:12:25Z","closed_at":"2026-05-05T07:13:54Z","close_reason":"Added ALPACA_API_KEY support with key-pair fallback","dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||
{"_type":"issue","id":"islandflow-neu","title":"Add Alpha Vantage event calendar provider","description":"Add an Alpha Vantage earnings-calendar provider to services/refdata that fetches CSV, normalizes entries, writes the JSON cache consumed by compute, and documents the required env variables.\n","status":"closed","priority":2,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-05T07:00:31Z","created_by":"dirtydishes","updated_at":"2026-05-05T07:02:30Z","started_at":"2026-05-05T07:00:37Z","closed_at":"2026-05-05T07:02:30Z","close_reason":"Added Alpha Vantage event-calendar provider","dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||
{"_type":"issue","id":"islandflow-b6d","title":"Finish smart-money event-calendar enrichment","description":"Finish the smart-money event-calendar provider layer in services/refdata and connect days-to-event / expiry-after-event enrichment into compute using timestamp-available data only.","status":"closed","priority":2,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-04T21:35:26Z","created_by":"dirtydishes","updated_at":"2026-05-04T23:21:09Z","started_at":"2026-05-04T23:18:29Z","closed_at":"2026-05-04T23:21:09Z","close_reason":"Completed event-calendar provider and compute enrichment","dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||
{"_type":"issue","id":"islandflow-e60","title":"Add smart-money replay evaluation harness","description":"Add replay-style live-vs-batch consistency tests plus evaluation utilities for parent-event precision/recall, calibration, abstention rate, and economic sanity checks.","status":"closed","priority":2,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-04T21:35:25Z","created_by":"dirtydishes","updated_at":"2026-05-05T06:08:08Z","started_at":"2026-05-05T06:07:22Z","closed_at":"2026-05-05T06:08:08Z","close_reason":"Completed smart-money replay consistency harness and evaluation utilities.","dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||
{"_type":"issue","id":"islandflow-020","title":"Rebuild synthetic smart-money scenarios","description":"Rework services/ingest-options synthetic generation around labeled parent-event templates for the six core smart-money profiles plus neutral background noise, with deterministic test/demo modes and hidden labels for tests.","status":"closed","priority":2,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-04T21:35:24Z","created_by":"dirtydishes","updated_at":"2026-05-05T05:29:27Z","started_at":"2026-05-05T05:25:39Z","closed_at":"2026-05-05T05:29:27Z","close_reason":"Completed Phase 5 synthetic smart-money scenario rebuild","dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||
{"_type":"issue","id":"islandflow-zs0","title":"Migrate terminal UI to smart-money profiles","description":"Migrate apps/web terminal rendering to consume SmartMoneyEvent directly: primary profile, probability ladder, reason codes, and suppression/abstention state, while preserving legacy alert/classifier displays during the bridge.","status":"closed","priority":2,"issue_type":"task","owner":"dishes@dpdrm.com","created_at":"2026-05-04T21:35:23Z","created_by":"dirtydishes","updated_at":"2026-05-05T05:39:58Z","closed_at":"2026-05-05T05:39:58Z","close_reason":"Completed terminal smart-money profile migration","dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||
{"_type":"issue","id":"islandflow-igk","title":"Add plan mode","description":"Implement a user-facing plan mode in the application so users can switch into planning before taking action. Scope to be clarified from existing app patterns.","status":"closed","priority":2,"issue_type":"feature","owner":"dishes@dpdrm.com","created_at":"2026-05-04T04:22:37Z","created_by":"dirtydishes","updated_at":"2026-05-04T04:26:18Z","started_at":"2026-05-04T04:22:40Z","closed_at":"2026-05-04T04:26:18Z","close_reason":"Implemented as a global pi extension toggled with Shift+P","dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||
{"_type":"issue","id":"islandflow-biq","title":"Finish raw live options delivery and filter/backpressure observability","description":"The smart-money signal path and Tape filters are in place, but the next firehose pass should finish server-side selective raw live delivery for options subscriptions and add explicit filtered-out/backpressure observability for API/web counters. This was discovered while landing islandflow-e4r.\n","status":"in_progress","priority":2,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T20:28:58Z","created_by":"dirtydishes","updated_at":"2026-04-29T03:54:12Z","started_at":"2026-04-29T03:54:12Z","dependencies":[{"issue_id":"islandflow-biq","depends_on_id":"islandflow-e4r","type":"discovered-from","created_at":"2026-04-28T16:28:58Z","created_by":"dirtydishes","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||
{"_type":"issue","id":"islandflow-575","title":"Document smart-money event calendar env","description":"Document smart-money event-calendar environment configuration in env examples and README.\n","status":"closed","priority":3,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-05T06:57:14Z","created_by":"dirtydishes","updated_at":"2026-05-05T06:57:57Z","started_at":"2026-05-05T06:57:17Z","closed_at":"2026-05-05T06:57:57Z","close_reason":"Documented event-calendar env variables","dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||
|
|
|
|||
15
.env.example
15
.env.example
|
|
@ -5,8 +5,7 @@ REDIS_URL=redis://127.0.0.1:6379
|
|||
|
||||
# Options ingest
|
||||
OPTIONS_INGEST_ADAPTER=synthetic
|
||||
ALPACA_KEY_ID=
|
||||
ALPACA_SECRET_KEY=
|
||||
ALPACA_API_KEY=
|
||||
ALPACA_REST_URL=https://data.alpaca.markets
|
||||
ALPACA_WS_BASE_URL=wss://stream.data.alpaca.markets/v1beta1
|
||||
ALPACA_FEED=indicative
|
||||
|
|
@ -79,6 +78,18 @@ CLASSIFIER_0DTE_MAX_ATM_PCT=0.01
|
|||
CLASSIFIER_0DTE_MIN_PREMIUM=20000
|
||||
CLASSIFIER_0DTE_MIN_SIZE=400
|
||||
|
||||
# Smart money refdata
|
||||
# Optional JSON event-calendar cache used by compute for event-driven profile enrichment.
|
||||
SMART_MONEY_EVENT_CALENDAR_PATH=data/event-calendar.json
|
||||
# Refdata service also accepts REFDATA_EVENT_CALENDAR_PATH; if unset it falls back to SMART_MONEY_EVENT_CALENDAR_PATH.
|
||||
REFDATA_EVENT_CALENDAR_PATH=
|
||||
# Set to alpha_vantage to refresh the JSON cache from Alpha Vantage's EARNINGS_CALENDAR CSV endpoint.
|
||||
REFDATA_EVENT_CALENDAR_PROVIDER=
|
||||
ALPHA_VANTAGE_API_KEY=
|
||||
ALPHA_VANTAGE_EARNINGS_HORIZON=3month
|
||||
ALPHA_VANTAGE_EARNINGS_SYMBOL=
|
||||
REFDATA_EVENT_CALENDAR_REFRESH_MS=86400000
|
||||
|
||||
# Replay service
|
||||
REPLAY_ENABLED=false
|
||||
REPLAY_STREAMS=options,nbbo,equities,equity-quotes
|
||||
|
|
|
|||
14
README.md
14
README.md
|
|
@ -148,8 +148,7 @@ Synthetic profile intent:
|
|||
|
||||
| Variable | Default | What it controls |
|
||||
| --- | --- | --- |
|
||||
| `ALPACA_KEY_ID` | empty | Alpaca API key for options/equities adapters. Required when `*_INGEST_ADAPTER=alpaca`. |
|
||||
| `ALPACA_SECRET_KEY` | empty | Alpaca API secret for options/equities adapters. Required when `*_INGEST_ADAPTER=alpaca`. |
|
||||
| `ALPACA_API_KEY` | empty | Single-token Alpaca API auth for options/equities adapters. Use this when your account provides one API key value. |
|
||||
| `ALPACA_REST_URL` | `https://data.alpaca.markets` | Alpaca REST base URL for contract discovery/reference calls. |
|
||||
| `ALPACA_WS_BASE_URL` | `wss://stream.data.alpaca.markets/v1beta1` (options), `wss://stream.data.alpaca.markets` (equities) | Alpaca websocket base URL. |
|
||||
| `ALPACA_FEED` | `indicative` | Options feed tier for Alpaca options (`indicative` or `opra`). |
|
||||
|
|
@ -161,6 +160,8 @@ Synthetic profile intent:
|
|||
| `ALPACA_MAX_QUOTES` | `200` | Upper bound on selected Alpaca options contracts/quotes per cycle. |
|
||||
| `ALPACA_EQUITIES_FEED` | `iex` | Alpaca equities feed (`iex` free tier, `sip` paid consolidated feed). |
|
||||
|
||||
For Alpaca adapters, configure `ALPACA_API_KEY`.
|
||||
|
||||
### Databento replay adapter configuration
|
||||
|
||||
| Variable | Default | What it controls |
|
||||
|
|
@ -239,6 +240,15 @@ Default `smart-money` policy rejects lower-information prints and keeps high-con
|
|||
| `CLASSIFIER_0DTE_MAX_ATM_PCT` | `0.01` | Max distance-from-ATM to qualify as near-ATM 0DTE event. |
|
||||
| `CLASSIFIER_0DTE_MIN_PREMIUM` | `20000` | Minimum premium for 0DTE classifier events. |
|
||||
| `CLASSIFIER_0DTE_MIN_SIZE` | `400` | Minimum size for 0DTE classifier events. |
|
||||
| `SMART_MONEY_EVENT_CALENDAR_PATH` | empty | Optional JSON event-calendar file used by compute to enrich event-driven smart-money profile features. |
|
||||
| `REFDATA_EVENT_CALENDAR_PATH` | empty | Optional JSON event-calendar file for refdata service startup validation; falls back to `SMART_MONEY_EVENT_CALENDAR_PATH` when unset. |
|
||||
| `REFDATA_EVENT_CALENDAR_PROVIDER` | empty | Set to `alpha_vantage` to have refdata refresh the calendar cache from Alpha Vantage. |
|
||||
| `ALPHA_VANTAGE_API_KEY` | empty | Alpha Vantage key used when `REFDATA_EVENT_CALENDAR_PROVIDER=alpha_vantage`. |
|
||||
| `ALPHA_VANTAGE_EARNINGS_HORIZON` | `3month` | Alpha Vantage earnings horizon: `3month`, `6month`, or `12month`. |
|
||||
| `ALPHA_VANTAGE_EARNINGS_SYMBOL` | empty | Optional single-symbol Alpha Vantage earnings query; empty fetches the full scheduled earnings list. |
|
||||
| `REFDATA_EVENT_CALENDAR_REFRESH_MS` | `86400000` | Refdata refresh cadence for provider-backed event-calendar cache writes. |
|
||||
|
||||
Event-calendar rows may use `symbol`, `underlying`, or `underlying_id`; `event_date`, `event_time`, or `event_ts`; and `announced_ts`, `available_ts`, `as_of_ts`, or `created_ts`. Compute only uses events already available at the packet timestamp, so missing or unavailable rows leave event-alignment features as neutral `null` values.
|
||||
|
||||
### Candle service configuration
|
||||
|
||||
|
|
|
|||
63
SMART_MONEY_REBUILD_PLAN.md
Normal file
63
SMART_MONEY_REBUILD_PLAN.md
Normal file
|
|
@ -0,0 +1,63 @@
|
|||
# Smart Money Rebuild Plan
|
||||
|
||||
Living implementation tracker for the rules-first smart-money rebuild. Issue tracking remains in `bd`; this file records migration state, acceptance criteria, and handoff notes.
|
||||
|
||||
## Phase Checklists
|
||||
|
||||
### Phase 1: Contracts and Storage
|
||||
- [x] Add `SmartMoneyEvent` contract in `packages/types`.
|
||||
- [x] Add typed features, profile scores, abstention, and suppression metadata.
|
||||
- [x] Extend `AlertEvent` with optional profile metadata.
|
||||
- [x] Add `smart_money_events` ClickHouse storage helpers.
|
||||
- [x] Add bus/live channel names for smart-money events.
|
||||
|
||||
Acceptance: smart-money events round-trip through schema/storage helpers and alerts remain backward-compatible.
|
||||
|
||||
### Phase 2: Parent-Event Reconstruction
|
||||
- [x] Add `services/compute/src/parent-events.ts`.
|
||||
- [x] Convert existing `FlowPacket` clusters and structure packets into deterministic parent events.
|
||||
- [x] Emit deterministic event IDs from packet identity.
|
||||
- [x] Preserve bridge semantics while `FlowPacket` remains an intermediate artifact.
|
||||
|
||||
Acceptance: live and replay produce the same event ID for the same packet.
|
||||
|
||||
### Phase 3: Feature Engineering
|
||||
- [x] Build typed features for aggressor mix, spread/quote quality, timing, strike concentration, DTE, moneyness, structure markers, and event alignment fields.
|
||||
- [x] Keep batch-only validation fields out of live scoring.
|
||||
- [x] Connect an external event-calendar feed through `services/refdata`.
|
||||
|
||||
Acceptance: missing event-calendar fields produce neutral `null` feature values and do not block scoring.
|
||||
|
||||
### Phase 4: Rules Engine
|
||||
- [x] Score the six primary profiles.
|
||||
- [x] Return probabilities, confidence bands, directions, reason codes, and suppression reasons.
|
||||
- [x] Add false-positive guards for stale quotes, complex/special prints, retail-frenzy directional suppression, hedge-reactive 0-2 DTE ATM contexts, and arbitrage symmetry.
|
||||
|
||||
Acceptance: abstained events do not emit legacy classifier hits.
|
||||
|
||||
### Phase 5: Synthetic Market Redesign
|
||||
- [x] Rework synthetic options adapter around labeled parent-event templates.
|
||||
- [x] Add deterministic scenario families for all six profiles.
|
||||
- [x] Add test/demo operating modes with hidden labels.
|
||||
|
||||
Acceptance: scenario tests assert intended profile wins and wrong nearby profiles remain below threshold.
|
||||
|
||||
### Phase 6: Compute, API, and UI Rollout
|
||||
- [x] Emit `SmartMoneyEvent` first in compute.
|
||||
- [x] Derive compatibility `ClassifierHitEvent` and `AlertEvent`.
|
||||
- [x] Add REST/history/replay/ws/live support for smart-money events.
|
||||
- [x] Migrate terminal UI to profile-aware display.
|
||||
|
||||
Acceptance: old classifier and alert endpoints still work while `/flow/smart-money`, `/history/smart-money`, `/replay/smart-money`, and `/ws/smart-money` expose the new model.
|
||||
|
||||
### Phase 7: Evaluation and Replay
|
||||
- [x] Add deterministic unit tests for parent-event scoring and storage.
|
||||
- [x] Add replay-style live-vs-batch consistency tests.
|
||||
- [x] Add evaluation utilities for calibration, abstention rate, and economic sanity checks.
|
||||
|
||||
## Migration Notes
|
||||
|
||||
- `FlowPacket` remains the packet/cluster bridge and is no longer the final semantic alert object.
|
||||
- `ClassifierHitEvent` is now a compatibility surface derived from `SmartMoneyEvent.primary_profile_id`.
|
||||
- `AlertEvent` keeps existing fields and may include `primary_profile_id` plus `profile_scores`.
|
||||
- Existing structure labels such as vertical, straddle, roll, and 0DTE gamma are evidence/reason concepts rather than final business-facing profile IDs.
|
||||
|
|
@ -19,6 +19,8 @@ import {
|
|||
shouldRetainLiveSnapshotHistory,
|
||||
shouldShowEquitiesSilentFeedWarning,
|
||||
selectPrimaryClassifierHit,
|
||||
smartMoneyProfileLabel,
|
||||
smartMoneyToneForProfile,
|
||||
statusLabel,
|
||||
toggleFilterValue
|
||||
} from "./terminal";
|
||||
|
|
@ -318,6 +320,15 @@ describe("classifier row decoration helpers", () => {
|
|||
});
|
||||
});
|
||||
|
||||
describe("smart-money profile helpers", () => {
|
||||
it("labels and colors primary profiles", () => {
|
||||
expect(smartMoneyProfileLabel("institutional_directional")).toBe("Institutional Directional");
|
||||
expect(smartMoneyProfileLabel(null)).toBe("Abstained");
|
||||
expect(smartMoneyToneForProfile("event_driven")).toBe("blue");
|
||||
expect(smartMoneyToneForProfile(null)).toBe("neutral");
|
||||
});
|
||||
});
|
||||
|
||||
describe("flow filter popup helpers", () => {
|
||||
it("opens and closes the popup via toggle and dismiss actions", () => {
|
||||
expect(nextFlowFilterPopoverState(false, "toggle")).toBe(true);
|
||||
|
|
|
|||
|
|
@ -34,7 +34,9 @@ import type {
|
|||
OptionSecurityType,
|
||||
OptionType,
|
||||
OptionNBBO,
|
||||
OptionPrint
|
||||
OptionPrint,
|
||||
SmartMoneyEvent,
|
||||
SmartMoneyProfileId
|
||||
} from "@islandflow/types";
|
||||
import {
|
||||
getSubscriptionKey as getLiveSubscriptionKey,
|
||||
|
|
@ -239,6 +241,7 @@ type MessageType =
|
|||
| "equity-candle"
|
||||
| "equity-join"
|
||||
| "flow-packet"
|
||||
| "smart-money"
|
||||
| "inferred-dark"
|
||||
| "classifier-hit"
|
||||
| "alert";
|
||||
|
|
@ -1006,6 +1009,7 @@ const LIVE_SNAPSHOT_HISTORY_CHANNELS = new Set<LiveSubscription["channel"]>([
|
|||
"nbbo",
|
||||
"equities",
|
||||
"flow",
|
||||
"smart-money",
|
||||
"classifier-hits"
|
||||
]);
|
||||
|
||||
|
|
@ -1052,12 +1056,22 @@ const classifyNbboSide = (price: number, quote: OptionNBBO | null | undefined):
|
|||
};
|
||||
|
||||
type ClassifierDecor = {
|
||||
hit: ClassifierHitEvent;
|
||||
hit?: ClassifierHitEvent;
|
||||
smartMoney?: SmartMoneyEvent;
|
||||
family: string;
|
||||
tone: string;
|
||||
intensity: number;
|
||||
};
|
||||
|
||||
const SMART_MONEY_PROFILE_TONES: Record<SmartMoneyProfileId, string> = {
|
||||
institutional_directional: "green",
|
||||
retail_whale: "amber",
|
||||
event_driven: "blue",
|
||||
vol_seller: "copper",
|
||||
arbitrage: "teal",
|
||||
hedge_reactive: "magenta"
|
||||
};
|
||||
|
||||
const CLASSIFIER_FAMILY_TONES: Record<string, string> = {
|
||||
large_bullish_call_sweep: "green",
|
||||
large_bearish_put_sweep: "red",
|
||||
|
|
@ -1095,6 +1109,12 @@ export const selectPrimaryClassifierHit = (
|
|||
export const classifierToneForFamily = (classifierId: string): string =>
|
||||
CLASSIFIER_FAMILY_TONES[classifierId] ?? "neutral";
|
||||
|
||||
export const smartMoneyToneForProfile = (profileId: SmartMoneyProfileId | null): string =>
|
||||
profileId ? SMART_MONEY_PROFILE_TONES[profileId] ?? "neutral" : "neutral";
|
||||
|
||||
export const smartMoneyProfileLabel = (profileId: SmartMoneyProfileId | null): string =>
|
||||
profileId ? humanizeClassifierId(profileId) : "Abstained";
|
||||
|
||||
const buildClassifierDecor = (hit: ClassifierHitEvent): ClassifierDecor => ({
|
||||
hit,
|
||||
family: hit.classifier_id,
|
||||
|
|
@ -1102,6 +1122,18 @@ const buildClassifierDecor = (hit: ClassifierHitEvent): ClassifierDecor => ({
|
|||
intensity: clamp(hit.confidence, 0.25, 1)
|
||||
});
|
||||
|
||||
const buildSmartMoneyDecor = (event: SmartMoneyEvent): ClassifierDecor => {
|
||||
const primaryScore =
|
||||
event.profile_scores.find((score) => score.profile_id === event.primary_profile_id) ??
|
||||
event.profile_scores[0];
|
||||
return {
|
||||
smartMoney: event,
|
||||
family: event.primary_profile_id ?? primaryScore?.profile_id ?? "abstained",
|
||||
tone: event.abstained ? "neutral" : smartMoneyToneForProfile(event.primary_profile_id),
|
||||
intensity: clamp(primaryScore?.probability ?? 0.25, 0.25, 1)
|
||||
};
|
||||
};
|
||||
|
||||
export const getOptionTableSnapshot = (
|
||||
print: Pick<
|
||||
OptionPrint,
|
||||
|
|
@ -2230,6 +2262,7 @@ type LiveSessionState = {
|
|||
equityQuotes: EquityQuote[];
|
||||
equityJoins: EquityPrintJoin[];
|
||||
flow: FlowPacket[];
|
||||
smartMoney: SmartMoneyEvent[];
|
||||
classifierHits: ClassifierHitEvent[];
|
||||
alerts: AlertEvent[];
|
||||
inferredDark: InferredDarkEvent[];
|
||||
|
|
@ -2249,6 +2282,7 @@ const LIVE_HISTORY_ENDPOINTS: Partial<Record<LiveSubscription["channel"], string
|
|||
"equity-quotes": "/history/equity-quotes",
|
||||
"equity-joins": "/history/equity-joins",
|
||||
flow: "/history/flow",
|
||||
"smart-money": "/history/smart-money",
|
||||
"classifier-hits": "/history/classifier-hits",
|
||||
alerts: "/history/alerts",
|
||||
"inferred-dark": "/history/inferred-dark"
|
||||
|
|
@ -2318,6 +2352,7 @@ export const getLiveManifest = (
|
|||
{ channel: "nbbo" },
|
||||
{ channel: "equities", ...equityScope },
|
||||
{ channel: "flow", filters: flowFilters },
|
||||
{ channel: "smart-money" },
|
||||
{ channel: "classifier-hits" }
|
||||
]);
|
||||
}
|
||||
|
|
@ -2327,6 +2362,7 @@ export const getLiveManifest = (
|
|||
{ channel: "equities", ...equityScope },
|
||||
{ channel: "flow", filters: flowFilters },
|
||||
{ channel: "alerts" },
|
||||
{ channel: "smart-money" },
|
||||
{ channel: "classifier-hits" },
|
||||
{ channel: "inferred-dark" },
|
||||
...chartSubs
|
||||
|
|
@ -2357,6 +2393,7 @@ const useLiveSession = (
|
|||
const [equityQuotes, setEquityQuotes] = useState<EquityQuote[]>([]);
|
||||
const [equityJoins, setEquityJoins] = useState<EquityPrintJoin[]>([]);
|
||||
const [flow, setFlow] = useState<FlowPacket[]>([]);
|
||||
const [smartMoney, setSmartMoney] = useState<SmartMoneyEvent[]>([]);
|
||||
const [classifierHits, setClassifierHits] = useState<ClassifierHitEvent[]>([]);
|
||||
const [alerts, setAlerts] = useState<AlertEvent[]>([]);
|
||||
const [inferredDark, setInferredDark] = useState<InferredDarkEvent[]>([]);
|
||||
|
|
@ -2389,6 +2426,7 @@ const useLiveSession = (
|
|||
setEquityQuotes([]);
|
||||
setEquityJoins([]);
|
||||
setFlow([]);
|
||||
setSmartMoney([]);
|
||||
setClassifierHits([]);
|
||||
setAlerts([]);
|
||||
setInferredDark([]);
|
||||
|
|
@ -2489,6 +2527,9 @@ const useLiveSession = (
|
|||
case "flow":
|
||||
mergeItems(setFlow, items as FlowPacket[]);
|
||||
break;
|
||||
case "smart-money":
|
||||
mergeItems(setSmartMoney, items as SmartMoneyEvent[]);
|
||||
break;
|
||||
case "classifier-hits":
|
||||
mergeItems(setClassifierHits, items as ClassifierHitEvent[]);
|
||||
break;
|
||||
|
|
@ -2757,6 +2798,9 @@ const useLiveSession = (
|
|||
case "flow":
|
||||
mergeOlder(setFlow, LIVE_HOT_WINDOW);
|
||||
break;
|
||||
case "smart-money":
|
||||
mergeOlder(setSmartMoney, LIVE_HOT_WINDOW);
|
||||
break;
|
||||
case "classifier-hits":
|
||||
mergeOlder(setClassifierHits, LIVE_HOT_WINDOW);
|
||||
break;
|
||||
|
|
@ -2801,6 +2845,7 @@ const useLiveSession = (
|
|||
equityQuotes,
|
||||
equityJoins,
|
||||
flow,
|
||||
smartMoney,
|
||||
classifierHits,
|
||||
alerts,
|
||||
inferredDark,
|
||||
|
|
@ -2879,14 +2924,14 @@ type CandleChartProps = {
|
|||
replayTime?: number | null;
|
||||
liveCandles?: EquityCandle[];
|
||||
liveOverlayPrints?: EquityPrint[];
|
||||
classifierHits: ClassifierHitEvent[];
|
||||
smartMoneyEvents: SmartMoneyEvent[];
|
||||
inferredDark: InferredDarkEvent[];
|
||||
onClassifierHitClick: (hit: ClassifierHitEvent) => void;
|
||||
onSmartMoneyClick: (event: SmartMoneyEvent) => void;
|
||||
onInferredDarkClick: (event: InferredDarkEvent) => void;
|
||||
};
|
||||
|
||||
type MarkerAction =
|
||||
| { kind: "hit"; hit: ClassifierHitEvent }
|
||||
| { kind: "smart-money"; event: SmartMoneyEvent }
|
||||
| { kind: "dark"; event: InferredDarkEvent };
|
||||
|
||||
const CandleChart = ({
|
||||
|
|
@ -2896,9 +2941,9 @@ const CandleChart = ({
|
|||
replayTime = null,
|
||||
liveCandles = [],
|
||||
liveOverlayPrints = [],
|
||||
classifierHits,
|
||||
smartMoneyEvents,
|
||||
inferredDark,
|
||||
onClassifierHitClick,
|
||||
onSmartMoneyClick,
|
||||
onInferredDarkClick
|
||||
}: CandleChartProps) => {
|
||||
const containerRef = useRef<HTMLDivElement | null>(null);
|
||||
|
|
@ -2912,7 +2957,7 @@ const CandleChart = ({
|
|||
|
||||
const markerLookupRef = useRef<Map<string, MarkerAction>>(new Map());
|
||||
const [visibleRangeMs, setVisibleRangeMs] = useState<{ from: number; to: number } | null>(null);
|
||||
const onHitClickRef = useRef(onClassifierHitClick);
|
||||
const onSmartMoneyClickRef = useRef(onSmartMoneyClick);
|
||||
const onDarkClickRef = useRef(onInferredDarkClick);
|
||||
|
||||
const overlayCanvasRef = useRef<HTMLCanvasElement | null>(null);
|
||||
|
|
@ -2990,8 +3035,8 @@ const CandleChart = ({
|
|||
}, [drawOverlay, ticker, intervalMs, mode]);
|
||||
|
||||
useEffect(() => {
|
||||
onHitClickRef.current = onClassifierHitClick;
|
||||
}, [onClassifierHitClick]);
|
||||
onSmartMoneyClickRef.current = onSmartMoneyClick;
|
||||
}, [onSmartMoneyClick]);
|
||||
|
||||
useEffect(() => {
|
||||
onDarkClickRef.current = onInferredDarkClick;
|
||||
|
|
@ -3006,8 +3051,8 @@ const CandleChart = ({
|
|||
}
|
||||
|
||||
const { from, to } = visibleRangeMs;
|
||||
const inRangeHits = classifierHits
|
||||
.filter((hit) => hit.source_ts >= from && hit.source_ts <= to)
|
||||
const inRangeSmartMoney = smartMoneyEvents
|
||||
.filter((event) => event.source_ts >= from && event.source_ts <= to)
|
||||
.sort((a, b) => {
|
||||
const delta = a.source_ts - b.source_ts;
|
||||
if (delta !== 0) {
|
||||
|
|
@ -3025,27 +3070,27 @@ const CandleChart = ({
|
|||
return a.seq - b.seq;
|
||||
});
|
||||
|
||||
const MAX_HIT_MARKERS = 220;
|
||||
const MAX_SMART_MONEY_MARKERS = 220;
|
||||
const MAX_DARK_MARKERS = 120;
|
||||
const MAX_TOTAL_MARKERS = 320;
|
||||
|
||||
const cappedHits =
|
||||
inRangeHits.length > MAX_HIT_MARKERS
|
||||
? inRangeHits.slice(inRangeHits.length - MAX_HIT_MARKERS)
|
||||
: inRangeHits;
|
||||
const cappedSmartMoney =
|
||||
inRangeSmartMoney.length > MAX_SMART_MONEY_MARKERS
|
||||
? inRangeSmartMoney.slice(inRangeSmartMoney.length - MAX_SMART_MONEY_MARKERS)
|
||||
: inRangeSmartMoney;
|
||||
const cappedDark =
|
||||
inRangeDark.length > MAX_DARK_MARKERS
|
||||
? inRangeDark.slice(inRangeDark.length - MAX_DARK_MARKERS)
|
||||
: inRangeDark;
|
||||
|
||||
for (const hit of cappedHits) {
|
||||
const direction = normalizeDirection(hit.direction);
|
||||
const markerId = `hit:${hit.trace_id}:${hit.seq}`;
|
||||
lookup.set(markerId, { kind: "hit", hit });
|
||||
for (const event of cappedSmartMoney) {
|
||||
const direction = normalizeDirection(event.primary_direction);
|
||||
const markerId = `smart-money:${event.trace_id}:${event.seq}`;
|
||||
lookup.set(markerId, { kind: "smart-money", event });
|
||||
|
||||
markers.push({
|
||||
id: markerId,
|
||||
time: toChartTime(hit.source_ts),
|
||||
time: toChartTime(event.source_ts),
|
||||
position: direction === "bullish" ? "belowBar" : "aboveBar",
|
||||
color:
|
||||
direction === "bullish"
|
||||
|
|
@ -3059,7 +3104,11 @@ const CandleChart = ({
|
|||
: direction === "bearish"
|
||||
? "arrowDown"
|
||||
: "circle",
|
||||
text: hit.classifier_id ? hit.classifier_id.slice(0, 3).toUpperCase() : "H"
|
||||
text: event.abstained
|
||||
? "ABS"
|
||||
: event.primary_profile_id
|
||||
? event.primary_profile_id.slice(0, 3).toUpperCase()
|
||||
: "SM"
|
||||
});
|
||||
}
|
||||
|
||||
|
|
@ -3105,7 +3154,7 @@ const CandleChart = ({
|
|||
}
|
||||
|
||||
return { markers: cappedMarkers, lookup };
|
||||
}, [classifierHits, inferredDark, visibleRangeMs]);
|
||||
}, [smartMoneyEvents, inferredDark, visibleRangeMs]);
|
||||
|
||||
useEffect(() => {
|
||||
if (!seriesRef.current) {
|
||||
|
|
@ -3221,8 +3270,8 @@ const CandleChart = ({
|
|||
if (!action) {
|
||||
return;
|
||||
}
|
||||
if (action.kind === "hit") {
|
||||
onHitClickRef.current(action.hit);
|
||||
if (action.kind === "smart-money") {
|
||||
onSmartMoneyClickRef.current(action.event);
|
||||
} else {
|
||||
onDarkClickRef.current(action.event);
|
||||
}
|
||||
|
|
@ -3882,6 +3931,109 @@ const ClassifierHitDrawer = ({ hit, flowPacket, evidence, onClose }: ClassifierH
|
|||
);
|
||||
};
|
||||
|
||||
type SmartMoneyDrawerProps = {
|
||||
event: SmartMoneyEvent;
|
||||
flowPacket: FlowPacket | null;
|
||||
evidence: EvidenceItem[];
|
||||
onClose: () => void;
|
||||
};
|
||||
|
||||
const SmartMoneyDrawer = ({ event, flowPacket, evidence, onClose }: SmartMoneyDrawerProps) => {
|
||||
const primaryScore =
|
||||
event.profile_scores.find((score) => score.profile_id === event.primary_profile_id) ??
|
||||
event.profile_scores[0];
|
||||
const direction = normalizeDirection(event.primary_direction);
|
||||
const evidencePrints = evidence.filter((item) => item.kind === "print");
|
||||
const unknownCount = evidence.filter((item) => item.kind === "unknown").length;
|
||||
|
||||
return (
|
||||
<aside className="drawer">
|
||||
<div className="drawer-header">
|
||||
<div>
|
||||
<p className="drawer-eyebrow">Smart money profile</p>
|
||||
<h3>{smartMoneyProfileLabel(event.primary_profile_id)}</h3>
|
||||
<p className="drawer-subtitle">{formatDateTime(event.source_ts)}</p>
|
||||
</div>
|
||||
<button className="drawer-close" type="button" onClick={onClose}>
|
||||
Close
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<div className="drawer-meta">
|
||||
<span className={`pill direction-${direction}`}>{direction}</span>
|
||||
<span className="drawer-chip">
|
||||
Probability {primaryScore ? formatConfidence(primaryScore.probability) : "--"}
|
||||
</span>
|
||||
{event.abstained ? <span className="drawer-chip">Abstained</span> : null}
|
||||
</div>
|
||||
|
||||
<div className="drawer-section">
|
||||
<h4>Profile ladder</h4>
|
||||
<div className="drawer-list">
|
||||
{event.profile_scores.slice(0, 6).map((score) => (
|
||||
<div className="drawer-row" key={`${event.event_id}-${score.profile_id}`}>
|
||||
<div className="drawer-row-title">{smartMoneyProfileLabel(score.profile_id)}</div>
|
||||
<div className="drawer-row-meta">
|
||||
<span className={`pill direction-${normalizeDirection(score.direction)}`}>
|
||||
{normalizeDirection(score.direction)}
|
||||
</span>
|
||||
<span>{formatConfidence(score.probability)}</span>
|
||||
<span>{score.confidence_band}</span>
|
||||
</div>
|
||||
{score.reasons[0] ? <p className="drawer-note">{score.reasons[0]}</p> : null}
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
{event.suppressed_reasons.length > 0 ? (
|
||||
<p className="drawer-empty">Suppressed: {event.suppressed_reasons.join(", ")}</p>
|
||||
) : null}
|
||||
</div>
|
||||
|
||||
<div className="drawer-section">
|
||||
<h4>Parent event</h4>
|
||||
<div className="drawer-row">
|
||||
<div className="drawer-row-title">{event.underlying_id}</div>
|
||||
<div className="drawer-row-meta">
|
||||
<span>{formatFlowMetric(event.features.print_count)} prints</span>
|
||||
<span>{formatFlowMetric(event.features.total_size)} size</span>
|
||||
<span>${formatCompactUsd(event.features.total_premium)}</span>
|
||||
</div>
|
||||
<p className="drawer-note">
|
||||
Window {formatFlowMetric(event.event_window_ms, "ms")} · {event.event_kind}
|
||||
</p>
|
||||
</div>
|
||||
{flowPacket ? (
|
||||
<p className="drawer-note">Flow packet {flowPacket.id}</p>
|
||||
) : null}
|
||||
</div>
|
||||
|
||||
<div className="drawer-section">
|
||||
<h4>Evidence prints</h4>
|
||||
{evidencePrints.length === 0 ? (
|
||||
<p className="drawer-empty">No linked option prints in the live cache yet.</p>
|
||||
) : (
|
||||
<div className="drawer-list">
|
||||
{evidencePrints.slice(0, 6).map((item) => (
|
||||
<div className="drawer-row" key={item.id}>
|
||||
<div className="drawer-row-title">{item.print.option_contract_id}</div>
|
||||
<div className="drawer-row-meta">
|
||||
<span>${formatPrice(item.print.price)}</span>
|
||||
<span>{formatSize(item.print.size)}x</span>
|
||||
<span>{item.print.exchange}</span>
|
||||
</div>
|
||||
<p className="drawer-note">{formatTime(item.print.ts)}</p>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
{unknownCount > 0 ? (
|
||||
<p className="drawer-empty">+{unknownCount} evidence prints not in cache.</p>
|
||||
) : null}
|
||||
</div>
|
||||
</aside>
|
||||
);
|
||||
};
|
||||
|
||||
type DarkDrawerProps = {
|
||||
event: InferredDarkEvent;
|
||||
evidence: DarkEvidenceItem[];
|
||||
|
|
@ -4009,6 +4161,7 @@ const useTerminalState = () => {
|
|||
const [selectedAlert, setSelectedAlert] = useState<AlertEvent | null>(null);
|
||||
const [selectedDarkEvent, setSelectedDarkEvent] = useState<InferredDarkEvent | null>(null);
|
||||
const [selectedClassifierHit, setSelectedClassifierHit] = useState<ClassifierHitEvent | null>(null);
|
||||
const [selectedSmartMoneyEvent, setSelectedSmartMoneyEvent] = useState<SmartMoneyEvent | null>(null);
|
||||
const [selectedInstrument, setSelectedInstrument] = useState<SelectedInstrument>(null);
|
||||
const [filterInput, setFilterInput] = useState<string>("");
|
||||
const [flowFilters, setFlowFilters] = useState<OptionFlowFilters>(() => buildDefaultFlowFilters());
|
||||
|
|
@ -4078,13 +4231,14 @@ const useTerminalState = () => {
|
|||
}, [mode]);
|
||||
|
||||
useEffect(() => {
|
||||
if (!selectedAlert && !selectedClassifierHit && !selectedDarkEvent) {
|
||||
if (!selectedAlert && !selectedClassifierHit && !selectedDarkEvent && !selectedSmartMoneyEvent) {
|
||||
return;
|
||||
}
|
||||
|
||||
const dismissDrawers = () => {
|
||||
setSelectedAlert(null);
|
||||
setSelectedClassifierHit(null);
|
||||
setSelectedSmartMoneyEvent(null);
|
||||
setSelectedDarkEvent(null);
|
||||
};
|
||||
|
||||
|
|
@ -4108,7 +4262,7 @@ const useTerminalState = () => {
|
|||
document.removeEventListener("mousedown", handlePointerDown);
|
||||
document.removeEventListener("keydown", handleKeyDown);
|
||||
};
|
||||
}, [selectedAlert, selectedClassifierHit, selectedDarkEvent]);
|
||||
}, [selectedAlert, selectedClassifierHit, selectedDarkEvent, selectedSmartMoneyEvent]);
|
||||
|
||||
const optionsScroll = useListScroll();
|
||||
const equitiesScroll = useListScroll();
|
||||
|
|
@ -4250,6 +4404,19 @@ const useTerminalState = () => {
|
|||
onNewItems: classifierScroll.onNewItems,
|
||||
getReplayKey: disableReplayGrouping
|
||||
});
|
||||
const smartMoney = useTape<SmartMoneyEvent>({
|
||||
mode,
|
||||
liveEnabled: false,
|
||||
wsPath: "/ws/smart-money",
|
||||
replayPath: "/replay/smart-money",
|
||||
latestPath: "/flow/smart-money",
|
||||
expectedType: "smart-money",
|
||||
batchSize: mode === "replay" ? 120 : undefined,
|
||||
pollMs: mode === "replay" ? 200 : undefined,
|
||||
captureScroll: classifierAnchor.capture,
|
||||
onNewItems: classifierScroll.onNewItems,
|
||||
getReplayKey: disableReplayGrouping
|
||||
});
|
||||
|
||||
const liveOptions = usePausableTapeView<OptionPrint>({
|
||||
enabled: mode === "live",
|
||||
|
|
@ -4302,6 +4469,10 @@ const useTerminalState = () => {
|
|||
mode === "live"
|
||||
? toStaticTapeState(liveSession.status, liveSession.classifierHits, liveSession.lastUpdate)
|
||||
: classifierHits;
|
||||
const smartMoneyFeed =
|
||||
mode === "live"
|
||||
? toStaticTapeState(liveSession.status, liveSession.smartMoney, liveSession.lastUpdate)
|
||||
: smartMoney;
|
||||
const inferredDarkFeed =
|
||||
mode === "live"
|
||||
? toStaticTapeState(liveSession.status, liveSession.inferredDark, liveSession.lastUpdate)
|
||||
|
|
@ -4329,7 +4500,7 @@ const useTerminalState = () => {
|
|||
|
||||
useLayoutEffect(() => {
|
||||
classifierAnchor.apply();
|
||||
}, [classifierHitsFeed.items, classifierAnchor.apply]);
|
||||
}, [smartMoneyFeed.items, classifierHitsFeed.items, classifierAnchor.apply]);
|
||||
|
||||
const nbboMap = useMemo(() => {
|
||||
const map = new Map<string, OptionNBBO>();
|
||||
|
|
@ -4595,6 +4766,7 @@ const useTerminalState = () => {
|
|||
}
|
||||
setSelectedDarkEvent(null);
|
||||
setSelectedClassifierHit(null);
|
||||
setSelectedSmartMoneyEvent(null);
|
||||
}, [mode]);
|
||||
|
||||
const extractPacketContract = useCallback((packet: FlowPacket): string => {
|
||||
|
|
@ -4634,6 +4806,19 @@ const useTerminalState = () => {
|
|||
return map;
|
||||
}, [classifierHitsFeed.items, extractPacketIdFromClassifierHitTrace]);
|
||||
|
||||
const smartMoneyByPacketId = useMemo(() => {
|
||||
const map = new Map<string, SmartMoneyEvent>();
|
||||
for (const event of smartMoneyFeed.items) {
|
||||
for (const packetId of event.packet_ids) {
|
||||
const existing = map.get(packetId);
|
||||
if (!existing || event.source_ts > existing.source_ts || event.seq > existing.seq) {
|
||||
map.set(packetId, event);
|
||||
}
|
||||
}
|
||||
}
|
||||
return map;
|
||||
}, [smartMoneyFeed.items]);
|
||||
|
||||
const packetIdByOptionTraceId = useMemo(() => {
|
||||
const map = new Map<string, string>();
|
||||
for (const packet of flowFeed.items) {
|
||||
|
|
@ -4647,13 +4832,18 @@ const useTerminalState = () => {
|
|||
const classifierDecorByOptionTraceId = useMemo(() => {
|
||||
const map = new Map<string, ClassifierDecor>();
|
||||
for (const [traceId, packetId] of packetIdByOptionTraceId) {
|
||||
const smartMoneyEvent = smartMoneyByPacketId.get(packetId);
|
||||
if (smartMoneyEvent) {
|
||||
map.set(traceId, buildSmartMoneyDecor(smartMoneyEvent));
|
||||
continue;
|
||||
}
|
||||
const primary = selectPrimaryClassifierHit(classifierHitsByPacketId.get(packetId) ?? []);
|
||||
if (primary) {
|
||||
map.set(traceId, buildClassifierDecor(primary));
|
||||
}
|
||||
}
|
||||
return map;
|
||||
}, [classifierHitsByPacketId, packetIdByOptionTraceId]);
|
||||
}, [classifierHitsByPacketId, packetIdByOptionTraceId, smartMoneyByPacketId]);
|
||||
|
||||
const selectedClassifierPacketId = useMemo(() => {
|
||||
if (!selectedClassifierHit) {
|
||||
|
|
@ -4721,6 +4911,90 @@ const useTerminalState = () => {
|
|||
});
|
||||
}, [resolvedFlowPacketMap, resolvedOptionPrintMap, selectedClassifierHit, selectedClassifierPacketId]);
|
||||
|
||||
const selectedSmartMoneyFlowPacket = useMemo(() => {
|
||||
const packetId = selectedSmartMoneyEvent?.packet_ids[0];
|
||||
return packetId ? resolvedFlowPacketMap.get(packetId) ?? null : null;
|
||||
}, [resolvedFlowPacketMap, selectedSmartMoneyEvent]);
|
||||
|
||||
const selectedSmartMoneyEvidence = useMemo((): EvidenceItem[] => {
|
||||
if (!selectedSmartMoneyEvent) {
|
||||
return [];
|
||||
}
|
||||
return selectedSmartMoneyEvent.member_print_ids.map((id) => {
|
||||
const print = resolvedOptionPrintMap.get(id);
|
||||
if (print) {
|
||||
return { kind: "print", id, print };
|
||||
}
|
||||
return { kind: "unknown", id };
|
||||
});
|
||||
}, [resolvedOptionPrintMap, selectedSmartMoneyEvent]);
|
||||
|
||||
useEffect(() => {
|
||||
if (!selectedSmartMoneyEvent || mode !== "live") {
|
||||
return;
|
||||
}
|
||||
|
||||
const missingPacketIds = selectedSmartMoneyEvent.packet_ids.filter((id) => !resolvedFlowPacketMap.has(id));
|
||||
if (missingPacketIds.length > 0) {
|
||||
incrementRetentionMetric("pinnedFetchMisses", missingPacketIds.length);
|
||||
void Promise.all(
|
||||
missingPacketIds.map(async (packetId) => {
|
||||
const response = await fetch(buildApiUrl(`/flow/packets/${encodeURIComponent(packetId)}`));
|
||||
if (!response.ok) {
|
||||
throw new Error(await readErrorDetail(response));
|
||||
}
|
||||
const payload = (await response.json()) as { data?: FlowPacket | null };
|
||||
return payload.data ?? null;
|
||||
})
|
||||
)
|
||||
.then((packets) => {
|
||||
const next = new Map<string, FlowPacket>();
|
||||
for (const packet of packets) {
|
||||
if (packet) {
|
||||
next.set(packet.id, packet);
|
||||
}
|
||||
}
|
||||
if (next.size > 0) {
|
||||
setPinnedFlowPacketMap((prev) => upsertPinnedEntries(prev, next, Date.now()));
|
||||
}
|
||||
})
|
||||
.catch((error) => {
|
||||
incrementRetentionMetric("pinnedFetchFailures", 1);
|
||||
console.warn("Failed to fetch smart-money flow packets", error);
|
||||
});
|
||||
}
|
||||
|
||||
const missingPrintIds = selectedSmartMoneyEvent.member_print_ids.filter((id) => !resolvedOptionPrintMap.has(id));
|
||||
if (missingPrintIds.length === 0) {
|
||||
return;
|
||||
}
|
||||
incrementRetentionMetric("pinnedFetchMisses", missingPrintIds.length);
|
||||
const url = new URL(buildApiUrl("/option-prints/by-trace"));
|
||||
for (const traceId of missingPrintIds) {
|
||||
url.searchParams.append("trace_id", traceId);
|
||||
}
|
||||
void fetch(url.toString())
|
||||
.then(async (response) => {
|
||||
if (!response.ok) {
|
||||
throw new Error(await readErrorDetail(response));
|
||||
}
|
||||
return response.json();
|
||||
})
|
||||
.then((payload: { data?: OptionPrint[] }) => {
|
||||
const next = new Map<string, OptionPrint>();
|
||||
for (const item of payload.data ?? []) {
|
||||
next.set(item.trace_id, item);
|
||||
}
|
||||
if (next.size > 0) {
|
||||
setPinnedOptionPrintMap((prev) => upsertPinnedEntries(prev, next, Date.now()));
|
||||
}
|
||||
})
|
||||
.catch((error) => {
|
||||
incrementRetentionMetric("pinnedFetchFailures", 1);
|
||||
console.warn("Failed to fetch smart-money option prints", error);
|
||||
});
|
||||
}, [mode, resolvedFlowPacketMap, resolvedOptionPrintMap, selectedSmartMoneyEvent]);
|
||||
|
||||
const inferAlertUnderlying = useCallback(
|
||||
(alert: AlertEvent): string | null => {
|
||||
const fromTrace = extractUnderlyingFromTrace(alert.trace_id);
|
||||
|
|
@ -4932,6 +5206,9 @@ const useTerminalState = () => {
|
|||
if (selectedClassifierPacketId) {
|
||||
keys.add(selectedClassifierPacketId);
|
||||
}
|
||||
for (const packetId of selectedSmartMoneyEvent?.packet_ids ?? []) {
|
||||
keys.add(packetId);
|
||||
}
|
||||
for (const alert of visibleAlerts) {
|
||||
const packetId = alert.evidence_refs[0];
|
||||
if (packetId) {
|
||||
|
|
@ -4939,7 +5216,7 @@ const useTerminalState = () => {
|
|||
}
|
||||
}
|
||||
return keys;
|
||||
}, [selectedAlert, selectedClassifierPacketId, visibleAlerts]);
|
||||
}, [selectedAlert, selectedClassifierPacketId, selectedSmartMoneyEvent, visibleAlerts]);
|
||||
|
||||
const activePinnedOptionKeys = useMemo(() => {
|
||||
const keys = new Set<string>();
|
||||
|
|
@ -4953,11 +5230,14 @@ const useTerminalState = () => {
|
|||
keys.add(id);
|
||||
}
|
||||
}
|
||||
for (const id of selectedSmartMoneyEvent?.member_print_ids ?? []) {
|
||||
keys.add(id);
|
||||
}
|
||||
for (const id of visibleAlertEvidenceRefs) {
|
||||
keys.add(id);
|
||||
}
|
||||
return keys;
|
||||
}, [selectedAlert, selectedClassifierFlowPacket, visibleAlertEvidenceRefs]);
|
||||
}, [selectedAlert, selectedClassifierFlowPacket, selectedSmartMoneyEvent, visibleAlertEvidenceRefs]);
|
||||
|
||||
const activePinnedJoinKeys = useMemo(() => {
|
||||
const keys = new Set<string>();
|
||||
|
|
@ -5009,10 +5289,17 @@ const useTerminalState = () => {
|
|||
});
|
||||
}, [classifierHitsFeed.items, extractUnderlyingFromTrace, matchesTicker, tickerSet]);
|
||||
|
||||
const chartClassifierHits = useMemo(() => {
|
||||
const filteredSmartMoneyEvents = useMemo(() => {
|
||||
if (tickerSet.size === 0) {
|
||||
return smartMoneyFeed.items;
|
||||
}
|
||||
return smartMoneyFeed.items.filter((event) => matchesTicker(event.underlying_id));
|
||||
}, [matchesTicker, smartMoneyFeed.items, tickerSet]);
|
||||
|
||||
const chartSmartMoneyEvents = useMemo(() => {
|
||||
const desired = chartTicker.toUpperCase();
|
||||
return classifierHitsFeed.items
|
||||
.filter((hit) => extractUnderlyingFromTrace(hit.trace_id) === desired)
|
||||
return smartMoneyFeed.items
|
||||
.filter((event) => event.underlying_id.toUpperCase() === desired)
|
||||
.sort((a, b) => {
|
||||
const delta = a.source_ts - b.source_ts;
|
||||
if (delta !== 0) {
|
||||
|
|
@ -5020,7 +5307,7 @@ const useTerminalState = () => {
|
|||
}
|
||||
return a.seq - b.seq;
|
||||
});
|
||||
}, [chartTicker, classifierHitsFeed.items, extractUnderlyingFromTrace]);
|
||||
}, [chartTicker, smartMoneyFeed.items]);
|
||||
|
||||
const chartInferredDark = useMemo(() => {
|
||||
const desired = chartTicker.toUpperCase();
|
||||
|
|
@ -5058,27 +5345,37 @@ const useTerminalState = () => {
|
|||
if (alert) {
|
||||
setSelectedClassifierHit(null);
|
||||
setSelectedDarkEvent(null);
|
||||
setSelectedSmartMoneyEvent(null);
|
||||
setSelectedAlert(alert);
|
||||
return;
|
||||
}
|
||||
|
||||
setSelectedAlert(null);
|
||||
setSelectedDarkEvent(null);
|
||||
setSelectedSmartMoneyEvent(null);
|
||||
setSelectedClassifierHit(hit);
|
||||
},
|
||||
[findAlertForClassifierHit]
|
||||
);
|
||||
|
||||
const handleClassifierMarkerClick = useCallback(
|
||||
(hit: ClassifierHitEvent) => {
|
||||
openFromClassifierHit(hit);
|
||||
const openFromSmartMoneyEvent = useCallback((event: SmartMoneyEvent) => {
|
||||
setSelectedAlert(null);
|
||||
setSelectedClassifierHit(null);
|
||||
setSelectedDarkEvent(null);
|
||||
setSelectedSmartMoneyEvent(event);
|
||||
}, []);
|
||||
|
||||
const handleSmartMoneyMarkerClick = useCallback(
|
||||
(event: SmartMoneyEvent) => {
|
||||
openFromSmartMoneyEvent(event);
|
||||
},
|
||||
[openFromClassifierHit]
|
||||
[openFromSmartMoneyEvent]
|
||||
);
|
||||
|
||||
const handleDarkMarkerClick = useCallback((event: InferredDarkEvent) => {
|
||||
setSelectedAlert(null);
|
||||
setSelectedClassifierHit(null);
|
||||
setSelectedSmartMoneyEvent(null);
|
||||
setSelectedDarkEvent(event);
|
||||
}, []);
|
||||
|
||||
|
|
@ -5089,6 +5386,7 @@ const useTerminalState = () => {
|
|||
inferredDarkFeed.lastUpdate,
|
||||
flowFeed.lastUpdate,
|
||||
alertsFeed.lastUpdate,
|
||||
smartMoneyFeed.lastUpdate,
|
||||
classifierHitsFeed.lastUpdate
|
||||
]
|
||||
.filter((value): value is number => value !== null)
|
||||
|
|
@ -5099,6 +5397,7 @@ const useTerminalState = () => {
|
|||
inferredDarkFeed.lastUpdate,
|
||||
flowFeed.lastUpdate,
|
||||
alertsFeed.lastUpdate,
|
||||
smartMoneyFeed.lastUpdate,
|
||||
classifierHitsFeed.lastUpdate
|
||||
]);
|
||||
|
||||
|
|
@ -5113,6 +5412,8 @@ const useTerminalState = () => {
|
|||
setSelectedDarkEvent,
|
||||
selectedClassifierHit,
|
||||
setSelectedClassifierHit,
|
||||
selectedSmartMoneyEvent,
|
||||
setSelectedSmartMoneyEvent,
|
||||
selectedInstrument,
|
||||
setSelectedInstrument,
|
||||
selectedInstrumentLabel,
|
||||
|
|
@ -5135,6 +5436,7 @@ const useTerminalState = () => {
|
|||
inferredDark: inferredDarkFeed,
|
||||
flow: flowFeed,
|
||||
alerts: alertsFeed,
|
||||
smartMoney: smartMoneyFeed,
|
||||
classifierHits: classifierHitsFeed,
|
||||
liveSession,
|
||||
activeTickers,
|
||||
|
|
@ -5155,17 +5457,21 @@ const useTerminalState = () => {
|
|||
selectedClassifierPacketId,
|
||||
selectedClassifierFlowPacket,
|
||||
selectedClassifierEvidence,
|
||||
selectedSmartMoneyFlowPacket,
|
||||
selectedSmartMoneyEvidence,
|
||||
filteredOptions,
|
||||
filteredEquities,
|
||||
equitiesSilentWarning,
|
||||
filteredInferredDark,
|
||||
filteredFlow,
|
||||
filteredAlerts,
|
||||
filteredSmartMoneyEvents,
|
||||
filteredClassifierHits,
|
||||
chartClassifierHits,
|
||||
chartSmartMoneyEvents,
|
||||
chartInferredDark,
|
||||
openFromSmartMoneyEvent,
|
||||
openFromClassifierHit,
|
||||
handleClassifierMarkerClick,
|
||||
handleSmartMoneyMarkerClick,
|
||||
handleDarkMarkerClick,
|
||||
lastSeen,
|
||||
toggleMode: () => {
|
||||
|
|
@ -5618,12 +5924,22 @@ const OptionsPane = ({ limit }: OptionsPaneProps) => {
|
|||
type="button"
|
||||
{...commonProps}
|
||||
key={`${print.trace_id}-${print.seq}`}
|
||||
onClick={() => state.openFromClassifierHit(decor.hit)}
|
||||
onClick={() =>
|
||||
decor.smartMoney
|
||||
? state.openFromSmartMoneyEvent(decor.smartMoney)
|
||||
: decor.hit
|
||||
? state.openFromClassifierHit(decor.hit)
|
||||
: undefined
|
||||
}
|
||||
onKeyDown={(event) => {
|
||||
if (event.key === "Enter" || event.key === " ") {
|
||||
event.preventDefault();
|
||||
if (decor.smartMoney) {
|
||||
state.openFromSmartMoneyEvent(decor.smartMoney);
|
||||
} else if (decor.hit) {
|
||||
state.openFromClassifierHit(decor.hit);
|
||||
}
|
||||
}
|
||||
}}
|
||||
>
|
||||
{cells}
|
||||
|
|
@ -5951,6 +6267,7 @@ const AlertsPane = ({ limit, withStrip = false, className }: AlertsPaneProps) =>
|
|||
onClick={() => {
|
||||
state.setSelectedDarkEvent(null);
|
||||
state.setSelectedClassifierHit(null);
|
||||
state.setSelectedSmartMoneyEvent(null);
|
||||
state.setSelectedAlert(alert);
|
||||
}}
|
||||
>
|
||||
|
|
@ -5982,8 +6299,22 @@ type ClassifierPaneProps = {
|
|||
|
||||
const ClassifierPane = ({ limit, className }: ClassifierPaneProps) => {
|
||||
const state = useTerminal();
|
||||
const items = limit ? state.filteredClassifierHits.slice(0, limit) : state.filteredClassifierHits;
|
||||
const virtual = useVirtualList(items, state.classifierScroll.listRef, !limit, 44);
|
||||
const smartMoneyItems = limit ? state.filteredSmartMoneyEvents.slice(0, limit) : state.filteredSmartMoneyEvents;
|
||||
const legacyItems =
|
||||
smartMoneyItems.length === 0
|
||||
? limit
|
||||
? state.filteredClassifierHits.slice(0, limit)
|
||||
: state.filteredClassifierHits
|
||||
: [];
|
||||
const items: Array<SmartMoneyEvent | ClassifierHitEvent> =
|
||||
smartMoneyItems.length > 0 ? smartMoneyItems : legacyItems;
|
||||
const virtual = useVirtualList<SmartMoneyEvent | ClassifierHitEvent>(
|
||||
items,
|
||||
state.classifierScroll.listRef,
|
||||
!limit,
|
||||
44
|
||||
);
|
||||
const showingSmartMoney = smartMoneyItems.length > 0;
|
||||
|
||||
return (
|
||||
<Pane
|
||||
|
|
@ -5991,19 +6322,19 @@ const ClassifierPane = ({ limit, className }: ClassifierPaneProps) => {
|
|||
title="Rules"
|
||||
status={
|
||||
<TapeStatus
|
||||
status={state.classifierHits.status}
|
||||
lastUpdate={state.classifierHits.lastUpdate}
|
||||
replayTime={state.classifierHits.replayTime}
|
||||
replayComplete={state.classifierHits.replayComplete}
|
||||
paused={state.classifierHits.paused}
|
||||
dropped={state.classifierHits.dropped}
|
||||
status={state.smartMoney.status}
|
||||
lastUpdate={state.smartMoney.lastUpdate ?? state.classifierHits.lastUpdate}
|
||||
replayTime={state.smartMoney.replayTime ?? state.classifierHits.replayTime}
|
||||
replayComplete={state.smartMoney.replayComplete || state.classifierHits.replayComplete}
|
||||
paused={state.smartMoney.paused}
|
||||
dropped={state.smartMoney.dropped}
|
||||
mode={state.mode}
|
||||
/>
|
||||
}
|
||||
actions={
|
||||
<TapeControls
|
||||
paused={state.classifierHits.paused}
|
||||
onTogglePause={state.classifierHits.togglePause}
|
||||
paused={state.smartMoney.paused}
|
||||
onTogglePause={state.smartMoney.togglePause}
|
||||
isAtTop={state.classifierScroll.isAtTop}
|
||||
missed={state.classifierScroll.missed}
|
||||
onJump={state.classifierScroll.jumpToTop}
|
||||
|
|
@ -6016,23 +6347,48 @@ const ClassifierPane = ({ limit, className }: ClassifierPaneProps) => {
|
|||
{state.tickerSet.size > 0
|
||||
? "No classifier hits match the current filter."
|
||||
: state.mode === "live"
|
||||
? "No classifier hits yet. Start compute."
|
||||
? "No smart-money profiles yet. Start compute."
|
||||
: "Replay queue empty. Ensure ClickHouse has data."}
|
||||
</div>
|
||||
) : (
|
||||
<div className="data-table-wrap" ref={state.classifierScroll.setListRef}>
|
||||
<div className="data-table data-table-classifier" role="table" aria-label="Classifier hits">
|
||||
<div className="data-table data-table-classifier" role="table" aria-label="Smart money profiles">
|
||||
<div className="data-table-head" role="row">
|
||||
<span className="data-table-cell">TIME</span>
|
||||
<span className="data-table-cell">RULE</span>
|
||||
<span className="data-table-cell">PROFILE</span>
|
||||
<span className="data-table-cell">DIR</span>
|
||||
<span className="data-table-cell">CONF</span>
|
||||
<span className="data-table-cell">PROB</span>
|
||||
<span className="data-table-cell">NOTE</span>
|
||||
</div>
|
||||
{virtual.topSpacerHeight > 0 ? (
|
||||
<div className="data-table-spacer" style={{ height: `${virtual.topSpacerHeight}px` }} aria-hidden />
|
||||
) : null}
|
||||
{virtual.visibleItems.map((hit) => {
|
||||
{showingSmartMoney ? (virtual.visibleItems as SmartMoneyEvent[]).map((event) => {
|
||||
const primaryScore =
|
||||
event.profile_scores.find((score) => score.profile_id === event.primary_profile_id) ??
|
||||
event.profile_scores[0];
|
||||
const direction = normalizeDirection(event.primary_direction);
|
||||
return (
|
||||
<button
|
||||
className={`data-table-row data-table-row-button data-table-row-classifier data-table-row-direction-${direction}`}
|
||||
key={`${event.trace_id}-${event.seq}`}
|
||||
type="button"
|
||||
onClick={() => state.openFromSmartMoneyEvent(event)}
|
||||
>
|
||||
<span className="data-table-cell data-table-cell-number">{formatTime(event.source_ts)}</span>
|
||||
<span className="data-table-cell">{smartMoneyProfileLabel(event.primary_profile_id)}</span>
|
||||
<span className="data-table-cell">{direction}</span>
|
||||
<span className="data-table-cell data-table-cell-number">
|
||||
{primaryScore ? formatConfidence(primaryScore.probability) : "--"}
|
||||
</span>
|
||||
<span className="data-table-cell">
|
||||
{event.abstained
|
||||
? event.suppressed_reasons[0] ?? "abstained"
|
||||
: primaryScore?.reasons[0] ?? primaryScore?.confidence_band ?? "--"}
|
||||
</span>
|
||||
</button>
|
||||
);
|
||||
}) : (virtual.visibleItems as ClassifierHitEvent[]).map((hit) => {
|
||||
const direction = normalizeDirection(hit.direction);
|
||||
return (
|
||||
<button
|
||||
|
|
@ -6130,6 +6486,7 @@ const DarkPane = ({ limit, className }: DarkPaneProps) => {
|
|||
onClick={() => {
|
||||
state.setSelectedAlert(null);
|
||||
state.setSelectedClassifierHit(null);
|
||||
state.setSelectedSmartMoneyEvent(null);
|
||||
state.setSelectedDarkEvent(event);
|
||||
}}
|
||||
>
|
||||
|
|
@ -6188,9 +6545,9 @@ const ChartPane = ({ title = "Chart" }: ChartPaneProps) => {
|
|||
replayTime={state.equities.replayTime}
|
||||
liveCandles={state.liveSession.chartCandles}
|
||||
liveOverlayPrints={state.liveSession.chartOverlay}
|
||||
classifierHits={state.chartClassifierHits}
|
||||
smartMoneyEvents={state.chartSmartMoneyEvents}
|
||||
inferredDark={state.chartInferredDark}
|
||||
onClassifierHitClick={state.handleClassifierMarkerClick}
|
||||
onSmartMoneyClick={state.handleSmartMoneyMarkerClick}
|
||||
onInferredDarkClick={state.handleDarkMarkerClick}
|
||||
/>
|
||||
</Pane>
|
||||
|
|
@ -6199,7 +6556,7 @@ const ChartPane = ({ title = "Chart" }: ChartPaneProps) => {
|
|||
|
||||
const FocusPane = () => {
|
||||
const state = useTerminal();
|
||||
const hits = state.chartClassifierHits.slice(-10).reverse();
|
||||
const hits = state.chartSmartMoneyEvents.slice(-10).reverse();
|
||||
const dark = state.chartInferredDark.slice(-10).reverse();
|
||||
|
||||
return (
|
||||
|
|
@ -6220,13 +6577,13 @@ const FocusPane = () => {
|
|||
className="row row-button"
|
||||
key={`${hit.trace_id}-${hit.seq}`}
|
||||
type="button"
|
||||
onClick={() => state.openFromClassifierHit(hit)}
|
||||
onClick={() => state.openFromSmartMoneyEvent(hit)}
|
||||
>
|
||||
<div>
|
||||
<div className="contract">{humanizeClassifierId(hit.classifier_id)}</div>
|
||||
<div className="contract">{smartMoneyProfileLabel(hit.primary_profile_id)}</div>
|
||||
<div className="meta">
|
||||
<span className={`pill direction-${normalizeDirection(hit.direction)}`}>
|
||||
{normalizeDirection(hit.direction)}
|
||||
<span className={`pill direction-${normalizeDirection(hit.primary_direction)}`}>
|
||||
{normalizeDirection(hit.primary_direction)}
|
||||
</span>
|
||||
<span>{formatTime(hit.source_ts)}</span>
|
||||
</div>
|
||||
|
|
@ -6396,6 +6753,15 @@ export function TerminalAppShell({ children }: { children: ReactNode }) {
|
|||
/>
|
||||
) : null}
|
||||
|
||||
{state.selectedSmartMoneyEvent ? (
|
||||
<SmartMoneyDrawer
|
||||
event={state.selectedSmartMoneyEvent}
|
||||
flowPacket={state.selectedSmartMoneyFlowPacket}
|
||||
evidence={state.selectedSmartMoneyEvidence}
|
||||
onClose={() => state.setSelectedSmartMoneyEvent(null)}
|
||||
/>
|
||||
) : null}
|
||||
|
||||
{state.selectedDarkEvent ? (
|
||||
<DarkDrawer
|
||||
event={state.selectedDarkEvent}
|
||||
|
|
|
|||
1
bun.lock
1
bun.lock
|
|
@ -81,6 +81,7 @@
|
|||
"@islandflow/bus": "workspace:*",
|
||||
"@islandflow/config": "workspace:*",
|
||||
"@islandflow/observability": "workspace:*",
|
||||
"@islandflow/refdata": "workspace:*",
|
||||
"@islandflow/storage": "workspace:*",
|
||||
"@islandflow/types": "workspace:*",
|
||||
"redis": "^5.10.0",
|
||||
|
|
|
|||
|
|
@ -20,8 +20,7 @@ NEXT_PUBLIC_NBBO_MAX_AGE_MS=1000
|
|||
|
||||
# Options ingest
|
||||
OPTIONS_INGEST_ADAPTER=synthetic
|
||||
ALPACA_KEY_ID=
|
||||
ALPACA_SECRET_KEY=
|
||||
ALPACA_API_KEY=
|
||||
ALPACA_REST_URL=https://data.alpaca.markets
|
||||
ALPACA_WS_BASE_URL=wss://stream.data.alpaca.markets/v1beta1
|
||||
ALPACA_FEED=indicative
|
||||
|
|
@ -96,6 +95,15 @@ CLASSIFIER_0DTE_MAX_ATM_PCT=0.01
|
|||
CLASSIFIER_0DTE_MIN_PREMIUM=20000
|
||||
CLASSIFIER_0DTE_MIN_SIZE=400
|
||||
|
||||
# Smart money refdata
|
||||
SMART_MONEY_EVENT_CALENDAR_PATH=data/event-calendar.json
|
||||
REFDATA_EVENT_CALENDAR_PATH=
|
||||
REFDATA_EVENT_CALENDAR_PROVIDER=
|
||||
ALPHA_VANTAGE_API_KEY=
|
||||
ALPHA_VANTAGE_EARNINGS_HORIZON=3month
|
||||
ALPHA_VANTAGE_EARNINGS_SYMBOL=
|
||||
REFDATA_EVENT_CALENDAR_REFRESH_MS=86400000
|
||||
|
||||
# Candles
|
||||
CANDLE_INTERVALS_MS=60000,300000
|
||||
CANDLE_MAX_LATE_MS=0
|
||||
|
|
|
|||
|
|
@ -2,3 +2,12 @@ TZ=Etc/UTC
|
|||
NPM_ADMIN_BIND_IP=100.87.130.79
|
||||
NPM_EDGE_NETWORK=nextcloud_edge
|
||||
NPM_SHARED_NETWORK=npm-shared
|
||||
|
||||
# Smart money refdata
|
||||
SMART_MONEY_EVENT_CALENDAR_PATH=data/event-calendar.json
|
||||
REFDATA_EVENT_CALENDAR_PATH=
|
||||
REFDATA_EVENT_CALENDAR_PROVIDER=
|
||||
ALPHA_VANTAGE_API_KEY=
|
||||
ALPHA_VANTAGE_EARNINGS_HORIZON=3month
|
||||
ALPHA_VANTAGE_EARNINGS_SYMBOL=
|
||||
REFDATA_EVENT_CALENDAR_REFRESH_MS=86400000
|
||||
|
|
|
|||
|
|
@ -16,6 +16,8 @@ export const STREAM_INFERRED_DARK = "INFERRED_DARK";
|
|||
export const SUBJECT_INFERRED_DARK = "dark.inferred";
|
||||
export const STREAM_FLOW_PACKETS = "FLOW_PACKETS";
|
||||
export const SUBJECT_FLOW_PACKETS = "flow.packets";
|
||||
export const STREAM_SMART_MONEY_EVENTS = "SMART_MONEY_EVENTS";
|
||||
export const SUBJECT_SMART_MONEY_EVENTS = "flow.smart_money";
|
||||
export const STREAM_CLASSIFIER_HITS = "CLASSIFIER_HITS";
|
||||
export const SUBJECT_CLASSIFIER_HITS = "flow.classifier_hits";
|
||||
export const STREAM_ALERTS = "ALERTS";
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
import type { AlertEvent, ClassifierHit } from "@islandflow/types";
|
||||
import type { AlertEvent, ClassifierHit, SmartMoneyProfileScore } from "@islandflow/types";
|
||||
|
||||
export const ALERTS_TABLE = "alerts";
|
||||
|
||||
|
|
@ -11,6 +11,8 @@ export type AlertRecord = {
|
|||
severity: string;
|
||||
hits_json: string;
|
||||
evidence_refs_json: string;
|
||||
primary_profile_id: string;
|
||||
profile_scores_json: string;
|
||||
};
|
||||
|
||||
export const alertsTableDDL = (): string => {
|
||||
|
|
@ -23,13 +25,20 @@ CREATE TABLE IF NOT EXISTS ${ALERTS_TABLE} (
|
|||
score Float64,
|
||||
severity String,
|
||||
hits_json String,
|
||||
evidence_refs_json String
|
||||
evidence_refs_json String,
|
||||
primary_profile_id String DEFAULT '',
|
||||
profile_scores_json String DEFAULT '[]'
|
||||
)
|
||||
ENGINE = MergeTree
|
||||
ORDER BY (source_ts, seq)
|
||||
`;
|
||||
};
|
||||
|
||||
export const alertsTableMigrations = (): string[] => [
|
||||
`ALTER TABLE ${ALERTS_TABLE} ADD COLUMN IF NOT EXISTS primary_profile_id String DEFAULT ''`,
|
||||
`ALTER TABLE ${ALERTS_TABLE} ADD COLUMN IF NOT EXISTS profile_scores_json String DEFAULT '[]'`
|
||||
];
|
||||
|
||||
export const toAlertRecord = (alert: AlertEvent): AlertRecord => {
|
||||
return {
|
||||
source_ts: alert.source_ts,
|
||||
|
|
@ -39,7 +48,9 @@ export const toAlertRecord = (alert: AlertEvent): AlertRecord => {
|
|||
score: alert.score,
|
||||
severity: alert.severity,
|
||||
hits_json: JSON.stringify(alert.hits),
|
||||
evidence_refs_json: JSON.stringify(alert.evidence_refs)
|
||||
evidence_refs_json: JSON.stringify(alert.evidence_refs),
|
||||
primary_profile_id: alert.primary_profile_id ?? "",
|
||||
profile_scores_json: JSON.stringify(alert.profile_scores ?? [])
|
||||
};
|
||||
};
|
||||
|
||||
|
|
@ -79,6 +90,28 @@ const safeStringArray = (value: string): string[] => {
|
|||
return [];
|
||||
};
|
||||
|
||||
const safeProfileScoreArray = (value: string): SmartMoneyProfileScore[] => {
|
||||
try {
|
||||
const parsed = JSON.parse(value);
|
||||
if (Array.isArray(parsed)) {
|
||||
return parsed.map((entry) => {
|
||||
const record = entry as Partial<SmartMoneyProfileScore>;
|
||||
return {
|
||||
profile_id: String(record.profile_id ?? "") as SmartMoneyProfileScore["profile_id"],
|
||||
probability: Number(record.probability ?? 0),
|
||||
confidence_band: String(record.confidence_band ?? "low") as SmartMoneyProfileScore["confidence_band"],
|
||||
direction: String(record.direction ?? "unknown") as SmartMoneyProfileScore["direction"],
|
||||
reasons: Array.isArray(record.reasons) ? record.reasons.map((item) => String(item)) : []
|
||||
};
|
||||
});
|
||||
}
|
||||
} catch {
|
||||
// ignore
|
||||
}
|
||||
|
||||
return [];
|
||||
};
|
||||
|
||||
export const fromAlertRecord = (record: AlertRecord): AlertEvent => {
|
||||
return {
|
||||
source_ts: record.source_ts,
|
||||
|
|
@ -88,6 +121,8 @@ export const fromAlertRecord = (record: AlertRecord): AlertEvent => {
|
|||
score: record.score,
|
||||
severity: record.severity,
|
||||
hits: safeHitArray(record.hits_json),
|
||||
evidence_refs: safeStringArray(record.evidence_refs_json)
|
||||
evidence_refs: safeStringArray(record.evidence_refs_json),
|
||||
...(record.primary_profile_id ? { primary_profile_id: record.primary_profile_id as AlertEvent["primary_profile_id"] } : {}),
|
||||
profile_scores: safeProfileScoreArray(record.profile_scores_json)
|
||||
};
|
||||
};
|
||||
|
|
|
|||
|
|
@ -8,7 +8,8 @@ import {
|
|||
InferredDarkEventSchema,
|
||||
FlowPacketSchema,
|
||||
OptionNBBOSchema,
|
||||
OptionPrintSchema
|
||||
OptionPrintSchema,
|
||||
SmartMoneyEventSchema
|
||||
} from "@islandflow/types";
|
||||
import type {
|
||||
AlertEvent,
|
||||
|
|
@ -19,6 +20,7 @@ import type {
|
|||
EquityPrintJoin,
|
||||
InferredDarkEvent,
|
||||
FlowPacket,
|
||||
SmartMoneyEvent,
|
||||
OptionNBBO,
|
||||
OptionPrint,
|
||||
OptionFlowFilters,
|
||||
|
|
@ -76,11 +78,19 @@ import {
|
|||
} from "./classifier-hits";
|
||||
import {
|
||||
ALERTS_TABLE,
|
||||
alertsTableMigrations,
|
||||
alertsTableDDL,
|
||||
fromAlertRecord,
|
||||
toAlertRecord,
|
||||
type AlertRecord
|
||||
} from "./alerts";
|
||||
import {
|
||||
SMART_MONEY_EVENTS_TABLE,
|
||||
smartMoneyEventsTableDDL,
|
||||
fromSmartMoneyEventRecord,
|
||||
toSmartMoneyEventRecord,
|
||||
type SmartMoneyEventRecord
|
||||
} from "./smart-money-events";
|
||||
|
||||
export type ClickHouseOptions = {
|
||||
url: string;
|
||||
|
|
@ -285,6 +295,14 @@ export const ensureFlowPacketsTable = async (
|
|||
});
|
||||
};
|
||||
|
||||
export const ensureSmartMoneyEventsTable = async (
|
||||
client: ClickHouseClient
|
||||
): Promise<void> => {
|
||||
await client.exec({
|
||||
query: smartMoneyEventsTableDDL()
|
||||
});
|
||||
};
|
||||
|
||||
export const ensureClassifierHitsTable = async (
|
||||
client: ClickHouseClient
|
||||
): Promise<void> => {
|
||||
|
|
@ -297,6 +315,9 @@ export const ensureAlertsTable = async (client: ClickHouseClient): Promise<void>
|
|||
await client.exec({
|
||||
query: alertsTableDDL()
|
||||
});
|
||||
for (const query of alertsTableMigrations()) {
|
||||
await client.exec({ query });
|
||||
}
|
||||
};
|
||||
|
||||
export const insertOptionPrint = async (
|
||||
|
|
@ -395,6 +416,18 @@ export const insertFlowPacket = async (
|
|||
});
|
||||
};
|
||||
|
||||
export const insertSmartMoneyEvent = async (
|
||||
client: ClickHouseClient,
|
||||
event: SmartMoneyEvent
|
||||
): Promise<void> => {
|
||||
const record = toSmartMoneyEventRecord(event);
|
||||
await client.insert({
|
||||
table: SMART_MONEY_EVENTS_TABLE,
|
||||
values: [record],
|
||||
format: "JSONEachRow"
|
||||
});
|
||||
};
|
||||
|
||||
export const insertClassifierHit = async (
|
||||
client: ClickHouseClient,
|
||||
hit: ClassifierHitEvent
|
||||
|
|
@ -777,6 +810,34 @@ const normalizeClassifierHitRow = (row: unknown): ClassifierHitRecord | null =>
|
|||
};
|
||||
};
|
||||
|
||||
const normalizeSmartMoneyEventRow = (row: unknown): SmartMoneyEventRecord | null => {
|
||||
if (!row || typeof row !== "object") {
|
||||
return null;
|
||||
}
|
||||
|
||||
const record = row as Record<string, unknown>;
|
||||
return {
|
||||
source_ts: coerceNumber(record.source_ts) as number,
|
||||
ingest_ts: coerceNumber(record.ingest_ts) as number,
|
||||
seq: coerceNumber(record.seq) as number,
|
||||
trace_id: String(record.trace_id ?? ""),
|
||||
event_id: String(record.event_id ?? ""),
|
||||
packet_ids: Array.isArray(record.packet_ids) ? record.packet_ids.map((value) => String(value)) : [],
|
||||
member_print_ids: Array.isArray(record.member_print_ids)
|
||||
? record.member_print_ids.map((value) => String(value))
|
||||
: [],
|
||||
underlying_id: String(record.underlying_id ?? ""),
|
||||
event_kind: String(record.event_kind ?? ""),
|
||||
event_window_ms: coerceNumber(record.event_window_ms) as number,
|
||||
features_json: String(record.features_json ?? "{}"),
|
||||
profile_scores_json: String(record.profile_scores_json ?? "[]"),
|
||||
primary_profile_id: String(record.primary_profile_id ?? ""),
|
||||
primary_direction: String(record.primary_direction ?? "unknown"),
|
||||
abstained: Boolean(record.abstained),
|
||||
suppressed_reasons_json: String(record.suppressed_reasons_json ?? "[]")
|
||||
};
|
||||
};
|
||||
|
||||
const normalizeAlertRow = (row: unknown): AlertRecord | null => {
|
||||
if (!row || typeof row !== "object") {
|
||||
return null;
|
||||
|
|
@ -791,7 +852,9 @@ const normalizeAlertRow = (row: unknown): AlertRecord | null => {
|
|||
score: Number(coerceNumber(record.score) ?? 0),
|
||||
severity: String(record.severity ?? ""),
|
||||
hits_json: String(record.hits_json ?? "[]"),
|
||||
evidence_refs_json: String(record.evidence_refs_json ?? "[]")
|
||||
evidence_refs_json: String(record.evidence_refs_json ?? "[]"),
|
||||
primary_profile_id: String(record.primary_profile_id ?? ""),
|
||||
profile_scores_json: String(record.profile_scores_json ?? "[]")
|
||||
};
|
||||
};
|
||||
|
||||
|
|
@ -951,6 +1014,23 @@ export const fetchRecentClassifierHits = async (
|
|||
return ClassifierHitEventSchema.array().parse(hits);
|
||||
};
|
||||
|
||||
export const fetchRecentSmartMoneyEvents = async (
|
||||
client: ClickHouseClient,
|
||||
limit: number
|
||||
): Promise<SmartMoneyEvent[]> => {
|
||||
const safeLimit = clampLimit(limit);
|
||||
const result = await client.query({
|
||||
query: `SELECT * FROM ${SMART_MONEY_EVENTS_TABLE} ORDER BY source_ts DESC, seq DESC LIMIT ${safeLimit}`,
|
||||
format: "JSONEachRow"
|
||||
});
|
||||
|
||||
const rows = await result.json<unknown[]>();
|
||||
const records = rows
|
||||
.map(normalizeSmartMoneyEventRow)
|
||||
.filter((record): record is SmartMoneyEventRecord => record !== null);
|
||||
return SmartMoneyEventSchema.array().parse(records.map(fromSmartMoneyEventRecord));
|
||||
};
|
||||
|
||||
export const fetchRecentAlerts = async (
|
||||
client: ClickHouseClient,
|
||||
limit: number
|
||||
|
|
@ -1222,6 +1302,28 @@ export const fetchClassifierHitsAfter = async (
|
|||
return ClassifierHitEventSchema.array().parse(hits);
|
||||
};
|
||||
|
||||
export const fetchSmartMoneyEventsAfter = async (
|
||||
client: ClickHouseClient,
|
||||
afterTs: number,
|
||||
afterSeq: number,
|
||||
limit: number
|
||||
): Promise<SmartMoneyEvent[]> => {
|
||||
const safeLimit = clampLimit(limit);
|
||||
const safeAfterTs = clampCursor(afterTs);
|
||||
const safeAfterSeq = clampCursor(afterSeq);
|
||||
|
||||
const result = await client.query({
|
||||
query: `SELECT * FROM ${SMART_MONEY_EVENTS_TABLE} WHERE (source_ts, seq) > (${safeAfterTs}, ${safeAfterSeq}) ORDER BY source_ts ASC, seq ASC LIMIT ${safeLimit}`,
|
||||
format: "JSONEachRow"
|
||||
});
|
||||
|
||||
const rows = await result.json<unknown[]>();
|
||||
const records = rows
|
||||
.map(normalizeSmartMoneyEventRow)
|
||||
.filter((record): record is SmartMoneyEventRecord => record !== null);
|
||||
return SmartMoneyEventSchema.array().parse(records.map(fromSmartMoneyEventRecord));
|
||||
};
|
||||
|
||||
export const fetchAlertsAfter = async (
|
||||
client: ClickHouseClient,
|
||||
afterTs: number,
|
||||
|
|
@ -1385,6 +1487,25 @@ export const fetchClassifierHitsBefore = async (
|
|||
return ClassifierHitEventSchema.array().parse(records.map(fromClassifierHitRecord));
|
||||
};
|
||||
|
||||
export const fetchSmartMoneyEventsBefore = async (
|
||||
client: ClickHouseClient,
|
||||
beforeTs: number,
|
||||
beforeSeq: number,
|
||||
limit: number
|
||||
): Promise<SmartMoneyEvent[]> => {
|
||||
const safeLimit = clampLimit(limit);
|
||||
const result = await client.query({
|
||||
query: `SELECT * FROM ${SMART_MONEY_EVENTS_TABLE} WHERE ${buildBeforeTupleCondition("source_ts", "seq", beforeTs, beforeSeq)} ORDER BY source_ts DESC, seq DESC LIMIT ${safeLimit}`,
|
||||
format: "JSONEachRow"
|
||||
});
|
||||
|
||||
const rows = await result.json<unknown[]>();
|
||||
const records = rows
|
||||
.map(normalizeSmartMoneyEventRow)
|
||||
.filter((record): record is SmartMoneyEventRecord => record !== null);
|
||||
return SmartMoneyEventSchema.array().parse(records.map(fromSmartMoneyEventRecord));
|
||||
};
|
||||
|
||||
export const fetchAlertsBefore = async (
|
||||
client: ClickHouseClient,
|
||||
beforeTs: number,
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@ export * from "./clickhouse";
|
|||
export * from "./classifier-hits";
|
||||
export * from "./alerts";
|
||||
export * from "./flow-packets";
|
||||
export * from "./smart-money-events";
|
||||
export * from "./equity-prints";
|
||||
export * from "./equity-quotes";
|
||||
export * from "./equity-candles";
|
||||
|
|
|
|||
100
packages/storage/src/smart-money-events.ts
Normal file
100
packages/storage/src/smart-money-events.ts
Normal file
|
|
@ -0,0 +1,100 @@
|
|||
import type { SmartMoneyEvent } from "@islandflow/types";
|
||||
|
||||
export const SMART_MONEY_EVENTS_TABLE = "smart_money_events";
|
||||
|
||||
export type SmartMoneyEventRecord = {
|
||||
source_ts: number;
|
||||
ingest_ts: number;
|
||||
seq: number;
|
||||
trace_id: string;
|
||||
event_id: string;
|
||||
packet_ids: string[];
|
||||
member_print_ids: string[];
|
||||
underlying_id: string;
|
||||
event_kind: string;
|
||||
event_window_ms: number;
|
||||
features_json: string;
|
||||
profile_scores_json: string;
|
||||
primary_profile_id: string;
|
||||
primary_direction: string;
|
||||
abstained: boolean;
|
||||
suppressed_reasons_json: string;
|
||||
};
|
||||
|
||||
export const smartMoneyEventsTableDDL = (): string => {
|
||||
return `
|
||||
CREATE TABLE IF NOT EXISTS ${SMART_MONEY_EVENTS_TABLE} (
|
||||
source_ts UInt64,
|
||||
ingest_ts UInt64,
|
||||
seq UInt64,
|
||||
trace_id String,
|
||||
event_id String,
|
||||
packet_ids Array(String),
|
||||
member_print_ids Array(String),
|
||||
underlying_id String,
|
||||
event_kind String,
|
||||
event_window_ms UInt64,
|
||||
features_json String,
|
||||
profile_scores_json String,
|
||||
primary_profile_id String,
|
||||
primary_direction String,
|
||||
abstained Bool,
|
||||
suppressed_reasons_json String
|
||||
)
|
||||
ENGINE = MergeTree
|
||||
ORDER BY (source_ts, seq)
|
||||
`;
|
||||
};
|
||||
|
||||
export const toSmartMoneyEventRecord = (event: SmartMoneyEvent): SmartMoneyEventRecord => {
|
||||
return {
|
||||
source_ts: event.source_ts,
|
||||
ingest_ts: event.ingest_ts,
|
||||
seq: event.seq,
|
||||
trace_id: event.trace_id,
|
||||
event_id: event.event_id,
|
||||
packet_ids: event.packet_ids,
|
||||
member_print_ids: event.member_print_ids,
|
||||
underlying_id: event.underlying_id,
|
||||
event_kind: event.event_kind,
|
||||
event_window_ms: event.event_window_ms,
|
||||
features_json: JSON.stringify(event.features),
|
||||
profile_scores_json: JSON.stringify(event.profile_scores),
|
||||
primary_profile_id: event.primary_profile_id ?? "",
|
||||
primary_direction: event.primary_direction,
|
||||
abstained: event.abstained,
|
||||
suppressed_reasons_json: JSON.stringify(event.suppressed_reasons)
|
||||
};
|
||||
};
|
||||
|
||||
const safeJson = <T>(value: string, fallback: T): T => {
|
||||
try {
|
||||
return JSON.parse(value) as T;
|
||||
} catch {
|
||||
return fallback;
|
||||
}
|
||||
};
|
||||
|
||||
export const fromSmartMoneyEventRecord = (record: SmartMoneyEventRecord): SmartMoneyEvent => {
|
||||
const primaryProfileId = record.primary_profile_id.trim();
|
||||
return {
|
||||
source_ts: record.source_ts,
|
||||
ingest_ts: record.ingest_ts,
|
||||
seq: record.seq,
|
||||
trace_id: record.trace_id,
|
||||
event_id: record.event_id,
|
||||
packet_ids: record.packet_ids,
|
||||
member_print_ids: record.member_print_ids,
|
||||
underlying_id: record.underlying_id,
|
||||
event_kind: record.event_kind as SmartMoneyEvent["event_kind"],
|
||||
event_window_ms: record.event_window_ms,
|
||||
features: safeJson(record.features_json, {} as SmartMoneyEvent["features"]),
|
||||
profile_scores: safeJson(record.profile_scores_json, [] as SmartMoneyEvent["profile_scores"]),
|
||||
primary_profile_id: primaryProfileId
|
||||
? (primaryProfileId as SmartMoneyEvent["primary_profile_id"])
|
||||
: null,
|
||||
primary_direction: record.primary_direction as SmartMoneyEvent["primary_direction"],
|
||||
abstained: Boolean(record.abstained),
|
||||
suppressed_reasons: safeJson(record.suppressed_reasons_json, [] as string[])
|
||||
};
|
||||
};
|
||||
85
packages/storage/tests/smart-money-events.test.ts
Normal file
85
packages/storage/tests/smart-money-events.test.ts
Normal file
|
|
@ -0,0 +1,85 @@
|
|||
import { describe, expect, it } from "bun:test";
|
||||
import {
|
||||
SMART_MONEY_EVENTS_TABLE,
|
||||
fromSmartMoneyEventRecord,
|
||||
smartMoneyEventsTableDDL,
|
||||
toSmartMoneyEventRecord
|
||||
} from "../src/smart-money-events";
|
||||
import type { SmartMoneyEvent } from "@islandflow/types";
|
||||
|
||||
const event: SmartMoneyEvent = {
|
||||
source_ts: 10,
|
||||
ingest_ts: 20,
|
||||
seq: 1,
|
||||
trace_id: "smartmoney:flowpacket:1",
|
||||
event_id: "smartmoney:single_leg_event:flowpacket:1",
|
||||
packet_ids: ["flowpacket:1"],
|
||||
member_print_ids: ["print:1"],
|
||||
underlying_id: "SPY",
|
||||
event_kind: "single_leg_event",
|
||||
event_window_ms: 500,
|
||||
features: {
|
||||
contract_count: 1,
|
||||
print_count: 3,
|
||||
total_size: 900,
|
||||
total_premium: 75_000,
|
||||
total_notional: 7_500_000,
|
||||
start_ts: 10,
|
||||
end_ts: 10,
|
||||
window_ms: 500,
|
||||
option_contract_id: "SPY-2025-01-17-450-C",
|
||||
option_type: "C",
|
||||
dte_days: 1,
|
||||
moneyness: 1,
|
||||
atm_proximity: 0.01,
|
||||
aggressor_buy_ratio: 0.7,
|
||||
aggressor_sell_ratio: 0.1,
|
||||
aggressor_ratio: 0.8,
|
||||
nbbo_coverage_ratio: 0.9,
|
||||
nbbo_inside_ratio: 0.1,
|
||||
nbbo_stale_ratio: 0,
|
||||
quote_age_ms: 20,
|
||||
venue_count: 2,
|
||||
inter_fill_ms_mean: 100,
|
||||
strike_count: 1,
|
||||
strike_concentration: 1,
|
||||
structure_legs: 0,
|
||||
same_size_leg_symmetry: 0,
|
||||
net_directional_bias: 0.6,
|
||||
synthetic_iv_shock: null,
|
||||
spread_widening: null,
|
||||
underlying_move_bps: null,
|
||||
days_to_event: null,
|
||||
expiry_after_event: null,
|
||||
pre_event_concentration: null,
|
||||
special_print_ratio: 0
|
||||
},
|
||||
profile_scores: [
|
||||
{
|
||||
profile_id: "institutional_directional",
|
||||
probability: 0.74,
|
||||
confidence_band: "high",
|
||||
direction: "bullish",
|
||||
reasons: ["large_parent_event"]
|
||||
}
|
||||
],
|
||||
primary_profile_id: "institutional_directional",
|
||||
primary_direction: "bullish",
|
||||
abstained: false,
|
||||
suppressed_reasons: []
|
||||
};
|
||||
|
||||
describe("smart money event storage helpers", () => {
|
||||
it("includes the correct table name in the DDL", () => {
|
||||
const ddl = smartMoneyEventsTableDDL();
|
||||
expect(ddl).toContain(SMART_MONEY_EVENTS_TABLE);
|
||||
expect(ddl).toContain("profile_scores_json");
|
||||
});
|
||||
|
||||
it("round-trips smart money event records", () => {
|
||||
const restored = fromSmartMoneyEventRecord(toSmartMoneyEventRecord(event));
|
||||
expect(restored.event_id).toBe(event.event_id);
|
||||
expect(restored.profile_scores).toEqual(event.profile_scores);
|
||||
expect(restored.features.total_premium).toBe(event.features.total_premium);
|
||||
});
|
||||
});
|
||||
|
|
@ -135,6 +135,98 @@ export const FlowPacketSchema = EventMetaSchema.merge(
|
|||
|
||||
export type FlowPacket = z.infer<typeof FlowPacketSchema>;
|
||||
|
||||
export const SmartMoneyProfileIdSchema = z.enum([
|
||||
"institutional_directional",
|
||||
"retail_whale",
|
||||
"event_driven",
|
||||
"vol_seller",
|
||||
"arbitrage",
|
||||
"hedge_reactive"
|
||||
]);
|
||||
|
||||
export type SmartMoneyProfileId = z.infer<typeof SmartMoneyProfileIdSchema>;
|
||||
|
||||
export const SmartMoneyDirectionSchema = z.enum(["bullish", "bearish", "neutral", "mixed", "unknown"]);
|
||||
|
||||
export type SmartMoneyDirection = z.infer<typeof SmartMoneyDirectionSchema>;
|
||||
|
||||
export const SmartMoneyEventKindSchema = z.enum(["single_leg_event", "multi_leg_event"]);
|
||||
|
||||
export type SmartMoneyEventKind = z.infer<typeof SmartMoneyEventKindSchema>;
|
||||
|
||||
export const SmartMoneyConfidenceBandSchema = z.enum(["low", "medium", "high"]);
|
||||
|
||||
export type SmartMoneyConfidenceBand = z.infer<typeof SmartMoneyConfidenceBandSchema>;
|
||||
|
||||
export const SmartMoneyFeaturesSchema = z.object({
|
||||
contract_count: z.number().int().nonnegative(),
|
||||
print_count: z.number().int().nonnegative(),
|
||||
total_size: z.number().nonnegative(),
|
||||
total_premium: z.number().nonnegative(),
|
||||
total_notional: z.number().nonnegative(),
|
||||
start_ts: z.number().int().nonnegative(),
|
||||
end_ts: z.number().int().nonnegative(),
|
||||
window_ms: z.number().int().nonnegative(),
|
||||
option_contract_id: z.string().min(1).optional(),
|
||||
option_type: z.enum(["C", "P"]).optional(),
|
||||
dte_days: z.number().nonnegative().nullable(),
|
||||
moneyness: z.number().nullable(),
|
||||
atm_proximity: z.number().nullable(),
|
||||
aggressor_buy_ratio: z.number().min(0).max(1),
|
||||
aggressor_sell_ratio: z.number().min(0).max(1),
|
||||
aggressor_ratio: z.number().min(0).max(1),
|
||||
nbbo_coverage_ratio: z.number().min(0).max(1),
|
||||
nbbo_inside_ratio: z.number().min(0).max(1),
|
||||
nbbo_stale_ratio: z.number().min(0).max(1),
|
||||
quote_age_ms: z.number().nonnegative().nullable(),
|
||||
venue_count: z.number().int().nonnegative(),
|
||||
inter_fill_ms_mean: z.number().nonnegative().nullable(),
|
||||
strike_count: z.number().int().nonnegative(),
|
||||
strike_concentration: z.number().min(0).max(1),
|
||||
structure_type: z.string().optional(),
|
||||
structure_legs: z.number().int().nonnegative(),
|
||||
same_size_leg_symmetry: z.number().min(0).max(1),
|
||||
net_directional_bias: z.number().min(-1).max(1),
|
||||
synthetic_iv_shock: z.number().nullable(),
|
||||
spread_widening: z.number().nullable(),
|
||||
underlying_move_bps: z.number().nullable(),
|
||||
days_to_event: z.number().nullable(),
|
||||
expiry_after_event: z.boolean().nullable(),
|
||||
pre_event_concentration: z.number().min(0).max(1).nullable(),
|
||||
special_print_ratio: z.number().min(0).max(1)
|
||||
});
|
||||
|
||||
export type SmartMoneyFeatures = z.infer<typeof SmartMoneyFeaturesSchema>;
|
||||
|
||||
export const SmartMoneyProfileScoreSchema = z.object({
|
||||
profile_id: SmartMoneyProfileIdSchema,
|
||||
probability: z.number().min(0).max(1),
|
||||
confidence_band: SmartMoneyConfidenceBandSchema,
|
||||
direction: SmartMoneyDirectionSchema,
|
||||
reasons: z.array(z.string().min(1))
|
||||
});
|
||||
|
||||
export type SmartMoneyProfileScore = z.infer<typeof SmartMoneyProfileScoreSchema>;
|
||||
|
||||
export const SmartMoneyEventSchema = EventMetaSchema.merge(
|
||||
z.object({
|
||||
event_id: z.string().min(1),
|
||||
packet_ids: z.array(z.string().min(1)),
|
||||
member_print_ids: z.array(z.string().min(1)),
|
||||
underlying_id: z.string().min(1),
|
||||
event_kind: SmartMoneyEventKindSchema,
|
||||
event_window_ms: z.number().int().nonnegative(),
|
||||
features: SmartMoneyFeaturesSchema,
|
||||
profile_scores: z.array(SmartMoneyProfileScoreSchema),
|
||||
primary_profile_id: SmartMoneyProfileIdSchema.nullable(),
|
||||
primary_direction: SmartMoneyDirectionSchema,
|
||||
abstained: z.boolean(),
|
||||
suppressed_reasons: z.array(z.string().min(1))
|
||||
})
|
||||
);
|
||||
|
||||
export type SmartMoneyEvent = z.infer<typeof SmartMoneyEventSchema>;
|
||||
|
||||
export const ClassifierHitSchema = z.object({
|
||||
classifier_id: z.string().min(1),
|
||||
confidence: z.number().min(0).max(1),
|
||||
|
|
@ -153,7 +245,9 @@ export const AlertEventSchema = EventMetaSchema.merge(
|
|||
score: z.number(),
|
||||
severity: z.string().min(1),
|
||||
hits: z.array(ClassifierHitSchema),
|
||||
evidence_refs: z.array(z.string().min(1))
|
||||
evidence_refs: z.array(z.string().min(1)),
|
||||
primary_profile_id: SmartMoneyProfileIdSchema.optional(),
|
||||
profile_scores: z.array(SmartMoneyProfileScoreSchema).optional()
|
||||
})
|
||||
);
|
||||
|
||||
|
|
|
|||
|
|
@ -9,7 +9,8 @@ import {
|
|||
FlowPacketSchema,
|
||||
InferredDarkEventSchema,
|
||||
OptionNBBOSchema,
|
||||
OptionPrintSchema
|
||||
OptionPrintSchema,
|
||||
SmartMoneyEventSchema
|
||||
} from "./events";
|
||||
import {
|
||||
OptionFlowFiltersSchema,
|
||||
|
|
@ -30,6 +31,7 @@ export const LiveGenericChannelSchema = z.enum([
|
|||
"equity-quotes",
|
||||
"equity-joins",
|
||||
"flow",
|
||||
"smart-money",
|
||||
"classifier-hits",
|
||||
"alerts",
|
||||
"inferred-dark"
|
||||
|
|
@ -42,6 +44,7 @@ export const LiveChannelSchema = z.enum([
|
|||
"equity-quotes",
|
||||
"equity-joins",
|
||||
"flow",
|
||||
"smart-money",
|
||||
"classifier-hits",
|
||||
"alerts",
|
||||
"inferred-dark",
|
||||
|
|
@ -63,6 +66,9 @@ export const LiveSubscriptionSchema = z.discriminatedUnion("channel", [
|
|||
channel: z.literal("flow"),
|
||||
filters: OptionFlowFiltersSchema.optional()
|
||||
}),
|
||||
z.object({
|
||||
channel: z.literal("smart-money")
|
||||
}),
|
||||
z.object({
|
||||
channel: z.enum(["nbbo", "equity-quotes", "equity-joins", "classifier-hits", "alerts", "inferred-dark"])
|
||||
}),
|
||||
|
|
@ -90,6 +96,7 @@ const livePayloadSchemas = {
|
|||
"equity-quotes": EquityQuoteSchema,
|
||||
"equity-joins": EquityPrintJoinSchema,
|
||||
flow: FlowPacketSchema,
|
||||
"smart-money": SmartMoneyEventSchema,
|
||||
"classifier-hits": ClassifierHitEventSchema,
|
||||
alerts: AlertEventSchema,
|
||||
"inferred-dark": InferredDarkEventSchema,
|
||||
|
|
|
|||
|
|
@ -9,6 +9,7 @@ import {
|
|||
SUBJECT_EQUITY_QUOTES,
|
||||
SUBJECT_INFERRED_DARK,
|
||||
SUBJECT_FLOW_PACKETS,
|
||||
SUBJECT_SMART_MONEY_EVENTS,
|
||||
SUBJECT_OPTION_NBBO,
|
||||
SUBJECT_OPTION_SIGNAL_PRINTS,
|
||||
STREAM_ALERTS,
|
||||
|
|
@ -19,6 +20,7 @@ import {
|
|||
STREAM_EQUITY_QUOTES,
|
||||
STREAM_INFERRED_DARK,
|
||||
STREAM_FLOW_PACKETS,
|
||||
STREAM_SMART_MONEY_EVENTS,
|
||||
STREAM_OPTION_NBBO,
|
||||
STREAM_OPTION_SIGNAL_PRINTS,
|
||||
buildDurableConsumer,
|
||||
|
|
@ -36,17 +38,21 @@ import {
|
|||
ensureEquityQuotesTable,
|
||||
ensureInferredDarkTable,
|
||||
ensureFlowPacketsTable,
|
||||
ensureSmartMoneyEventsTable,
|
||||
ensureOptionNBBOTable,
|
||||
ensureOptionPrintsTable,
|
||||
fetchAlertsAfter,
|
||||
fetchAlertsBefore,
|
||||
fetchClassifierHitsAfter,
|
||||
fetchClassifierHitsBefore,
|
||||
fetchSmartMoneyEventsAfter,
|
||||
fetchSmartMoneyEventsBefore,
|
||||
fetchFlowPacketsAfter,
|
||||
fetchFlowPacketById,
|
||||
fetchFlowPacketsBefore,
|
||||
fetchRecentAlerts,
|
||||
fetchRecentClassifierHits,
|
||||
fetchRecentSmartMoneyEvents,
|
||||
fetchRecentEquityPrintJoins,
|
||||
fetchRecentFlowPackets,
|
||||
fetchRecentInferredDark,
|
||||
|
|
@ -95,6 +101,7 @@ import {
|
|||
OptionSecurityTypeSchema,
|
||||
OptionTypeSchema,
|
||||
FlowPacketSchema,
|
||||
SmartMoneyEventSchema,
|
||||
OptionNBBOSchema,
|
||||
OptionPrintSchema,
|
||||
getSubscriptionKey
|
||||
|
|
@ -256,6 +263,7 @@ type Channel =
|
|||
| "equity-joins"
|
||||
| "inferred-dark"
|
||||
| "flow"
|
||||
| "smart-money"
|
||||
| "classifier-hits"
|
||||
| "alerts";
|
||||
|
||||
|
|
@ -278,6 +286,7 @@ const equityQuoteSockets = new Set<LegacySocket>();
|
|||
const equityJoinSockets = new Set<LegacySocket>();
|
||||
const inferredDarkSockets = new Set<LegacySocket>();
|
||||
const flowSockets = new Set<LegacySocket>();
|
||||
const smartMoneySockets = new Set<LegacySocket>();
|
||||
const classifierHitSockets = new Set<LegacySocket>();
|
||||
const alertSockets = new Set<LegacySocket>();
|
||||
const liveSocketSubscriptions = new Map<LiveSocket, Set<string>>();
|
||||
|
|
@ -772,6 +781,19 @@ const run = async () => {
|
|||
num_replicas: 1
|
||||
});
|
||||
|
||||
await ensureStream(jsm, {
|
||||
name: STREAM_SMART_MONEY_EVENTS,
|
||||
subjects: [SUBJECT_SMART_MONEY_EVENTS],
|
||||
retention: "limits",
|
||||
storage: "file",
|
||||
discard: "old",
|
||||
max_msgs_per_subject: -1,
|
||||
max_msgs: -1,
|
||||
max_bytes: -1,
|
||||
max_age: 0,
|
||||
num_replicas: 1
|
||||
});
|
||||
|
||||
await ensureStream(jsm, {
|
||||
name: STREAM_CLASSIFIER_HITS,
|
||||
subjects: [SUBJECT_CLASSIFIER_HITS],
|
||||
|
|
@ -812,6 +834,7 @@ const run = async () => {
|
|||
await ensureEquityPrintJoinsTable(clickhouse);
|
||||
await ensureInferredDarkTable(clickhouse);
|
||||
await ensureFlowPacketsTable(clickhouse);
|
||||
await ensureSmartMoneyEventsTable(clickhouse);
|
||||
await ensureClassifierHitsTable(clickhouse);
|
||||
await ensureAlertsTable(clickhouse);
|
||||
});
|
||||
|
|
@ -918,6 +941,11 @@ const run = async () => {
|
|||
stream: STREAM_FLOW_PACKETS,
|
||||
durableName: "api-flow-packets"
|
||||
},
|
||||
{
|
||||
subject: SUBJECT_SMART_MONEY_EVENTS,
|
||||
stream: STREAM_SMART_MONEY_EVENTS,
|
||||
durableName: "api-smart-money-events"
|
||||
},
|
||||
{
|
||||
subject: SUBJECT_CLASSIFIER_HITS,
|
||||
stream: STREAM_CLASSIFIER_HITS,
|
||||
|
|
@ -1057,18 +1085,24 @@ const run = async () => {
|
|||
consumerBindings[7].durableName
|
||||
);
|
||||
|
||||
const classifierHitSubscription = await subscribeWithReset(
|
||||
const smartMoneySubscription = await subscribeWithReset(
|
||||
consumerBindings[8].subject,
|
||||
consumerBindings[8].stream,
|
||||
consumerBindings[8].durableName
|
||||
);
|
||||
|
||||
const alertSubscription = await subscribeWithReset(
|
||||
const classifierHitSubscription = await subscribeWithReset(
|
||||
consumerBindings[9].subject,
|
||||
consumerBindings[9].stream,
|
||||
consumerBindings[9].durableName
|
||||
);
|
||||
|
||||
const alertSubscription = await subscribeWithReset(
|
||||
consumerBindings[10].subject,
|
||||
consumerBindings[10].stream,
|
||||
consumerBindings[10].durableName
|
||||
);
|
||||
|
||||
const fanoutLive = async (
|
||||
subscription: LiveSubscription,
|
||||
item: unknown,
|
||||
|
|
@ -1269,6 +1303,22 @@ const run = async () => {
|
|||
}
|
||||
};
|
||||
|
||||
const pumpSmartMoney = async () => {
|
||||
for await (const msg of smartMoneySubscription.messages) {
|
||||
try {
|
||||
const payload = SmartMoneyEventSchema.parse(smartMoneySubscription.decode(msg));
|
||||
broadcast(smartMoneySockets, { type: "smart-money", payload });
|
||||
await fanoutLive({ channel: "smart-money" }, payload, "smart-money");
|
||||
msg.ack();
|
||||
} catch (error) {
|
||||
logger.error("failed to process smart money event", {
|
||||
error: error instanceof Error ? error.message : String(error)
|
||||
});
|
||||
msg.term();
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
const pumpClassifierHits = async () => {
|
||||
for await (const msg of classifierHitSubscription.messages) {
|
||||
try {
|
||||
|
|
@ -1309,6 +1359,7 @@ const run = async () => {
|
|||
void pumpEquityJoins();
|
||||
void pumpInferredDark();
|
||||
void pumpFlow();
|
||||
void pumpSmartMoney();
|
||||
void pumpClassifierHits();
|
||||
void pumpAlerts();
|
||||
|
||||
|
|
@ -1429,6 +1480,12 @@ const run = async () => {
|
|||
return jsonResponse({ data });
|
||||
}
|
||||
|
||||
if (req.method === "GET" && url.pathname === "/flow/smart-money") {
|
||||
const limit = parseLimit(url.searchParams.get("limit"));
|
||||
const data = await fetchRecentSmartMoneyEvents(clickhouse, limit);
|
||||
return jsonResponse({ data });
|
||||
}
|
||||
|
||||
if (req.method === "GET" && url.pathname === "/flow/classifier-hits") {
|
||||
const limit = parseLimit(url.searchParams.get("limit"));
|
||||
const data = await fetchRecentClassifierHits(clickhouse, limit);
|
||||
|
|
@ -1507,6 +1564,14 @@ const run = async () => {
|
|||
);
|
||||
}
|
||||
|
||||
if (req.method === "GET" && url.pathname === "/history/smart-money") {
|
||||
const { beforeTs, beforeSeq, limit } = parseBeforeParams(url);
|
||||
const data = await fetchSmartMoneyEventsBefore(clickhouse, beforeTs, beforeSeq, limit);
|
||||
return jsonResponse(
|
||||
buildHistoryResponse(data, (item) => ({ ts: item.source_ts, seq: item.seq }))
|
||||
);
|
||||
}
|
||||
|
||||
if (req.method === "GET" && url.pathname === "/history/classifier-hits") {
|
||||
const { beforeTs, beforeSeq, limit } = parseBeforeParams(url);
|
||||
const data = await fetchClassifierHitsBefore(clickhouse, beforeTs, beforeSeq, limit);
|
||||
|
|
@ -1651,6 +1716,14 @@ const run = async () => {
|
|||
return jsonResponse({ data, next });
|
||||
}
|
||||
|
||||
if (req.method === "GET" && url.pathname === "/replay/smart-money") {
|
||||
const { afterTs, afterSeq, limit } = parseReplayParams(url);
|
||||
const data = await fetchSmartMoneyEventsAfter(clickhouse, afterTs, afterSeq, limit);
|
||||
const last = data.at(-1);
|
||||
const next = last ? { ts: last.source_ts, seq: last.seq } : null;
|
||||
return jsonResponse({ data, next });
|
||||
}
|
||||
|
||||
if (req.method === "GET" && url.pathname === "/replay/classifier-hits") {
|
||||
const { afterTs, afterSeq, limit } = parseReplayParams(url);
|
||||
const data = await fetchClassifierHitsAfter(clickhouse, afterTs, afterSeq, limit);
|
||||
|
|
@ -1739,6 +1812,14 @@ const run = async () => {
|
|||
return jsonResponse({ error: "websocket upgrade failed" }, 400);
|
||||
}
|
||||
|
||||
if (req.method === "GET" && url.pathname === "/ws/smart-money") {
|
||||
if (serverRef.upgrade(req, { data: { channel: "smart-money" } })) {
|
||||
return new Response(null, { status: 101 });
|
||||
}
|
||||
|
||||
return jsonResponse({ error: "websocket upgrade failed" }, 400);
|
||||
}
|
||||
|
||||
if (req.method === "GET" && url.pathname === "/ws/alerts") {
|
||||
if (serverRef.upgrade(req, { data: { channel: "alerts" } })) {
|
||||
return new Response(null, { status: 101 });
|
||||
|
|
@ -1781,6 +1862,8 @@ const run = async () => {
|
|||
inferredDarkSockets.add(socket);
|
||||
} else if (socket.data.channel === "flow") {
|
||||
flowSockets.add(socket);
|
||||
} else if (socket.data.channel === "smart-money") {
|
||||
smartMoneySockets.add(socket);
|
||||
} else if (socket.data.channel === "classifier-hits") {
|
||||
classifierHitSockets.add(socket);
|
||||
} else {
|
||||
|
|
@ -1842,6 +1925,8 @@ const run = async () => {
|
|||
inferredDarkSockets.delete(socket);
|
||||
} else if (socket.data.channel === "flow") {
|
||||
flowSockets.delete(socket);
|
||||
} else if (socket.data.channel === "smart-money") {
|
||||
smartMoneySockets.delete(socket);
|
||||
} else if (socket.data.channel === "classifier-hits") {
|
||||
classifierHitSockets.delete(socket);
|
||||
} else {
|
||||
|
|
|
|||
|
|
@ -9,6 +9,7 @@ import {
|
|||
fetchRecentFlowPackets,
|
||||
fetchRecentInferredDark,
|
||||
fetchRecentOptionNBBO,
|
||||
fetchRecentSmartMoneyEvents,
|
||||
type ClickHouseClient
|
||||
} from "@islandflow/storage";
|
||||
import type { OptionPrintQueryFilters } from "@islandflow/storage";
|
||||
|
|
@ -30,6 +31,7 @@ import {
|
|||
matchesOptionPrintFilters,
|
||||
OptionNBBOSchema,
|
||||
OptionPrintSchema,
|
||||
SmartMoneyEventSchema,
|
||||
type OptionFlowFilters,
|
||||
type Cursor,
|
||||
type EquityCandle,
|
||||
|
|
@ -51,6 +53,7 @@ const GENERIC_LIMIT_ENV_KEYS: Record<LiveGenericChannel, string> = {
|
|||
"equity-quotes": "LIVE_LIMIT_EQUITY_QUOTES",
|
||||
"equity-joins": "LIVE_LIMIT_EQUITY_JOINS",
|
||||
flow: "LIVE_LIMIT_FLOW",
|
||||
"smart-money": "LIVE_LIMIT_SMART_MONEY",
|
||||
"classifier-hits": "LIVE_LIMIT_CLASSIFIER_HITS",
|
||||
alerts: "LIVE_LIMIT_ALERTS",
|
||||
"inferred-dark": "LIVE_LIMIT_INFERRED_DARK"
|
||||
|
|
@ -111,6 +114,7 @@ export const resolveGenericLiveLimits = (env: NodeJS.ProcessEnv = process.env):
|
|||
"equity-quotes": parseGenericLimit(env, "equity-quotes", DEFAULT_GENERIC_LIMIT),
|
||||
"equity-joins": parseGenericLimit(env, "equity-joins", DEFAULT_GENERIC_LIMIT),
|
||||
flow: parseGenericLimit(env, "flow", DEFAULT_GENERIC_LIMIT),
|
||||
"smart-money": parseGenericLimit(env, "smart-money", DEFAULT_GENERIC_LIMIT),
|
||||
"classifier-hits": parseGenericLimit(env, "classifier-hits", DEFAULT_GENERIC_LIMIT),
|
||||
alerts: parseGenericLimit(env, "alerts", DEFAULT_GENERIC_LIMIT),
|
||||
"inferred-dark": parseGenericLimit(env, "inferred-dark", DEFAULT_GENERIC_LIMIT)
|
||||
|
|
@ -185,6 +189,14 @@ const getGenericConfig = (limits: GenericLiveLimits): {
|
|||
cursor: (item) => ({ ts: item.source_ts, seq: item.seq }),
|
||||
fetchRecent: fetchRecentFlowPackets
|
||||
},
|
||||
"smart-money": {
|
||||
redisKey: "live:smart-money",
|
||||
cursorField: "smart-money",
|
||||
limit: limits["smart-money"],
|
||||
parse: (value) => SmartMoneyEventSchema.parse(value),
|
||||
cursor: (item) => ({ ts: item.source_ts, seq: item.seq }),
|
||||
fetchRecent: fetchRecentSmartMoneyEvents
|
||||
},
|
||||
"classifier-hits": {
|
||||
redisKey: "live:classifier-hits",
|
||||
cursorField: "classifier-hits",
|
||||
|
|
|
|||
|
|
@ -154,6 +154,7 @@ describe("LiveStateManager", () => {
|
|||
"equity-quotes": 10000,
|
||||
"equity-joins": 10000,
|
||||
flow: 2,
|
||||
"smart-money": 10000,
|
||||
"classifier-hits": 10000,
|
||||
alerts: 10000,
|
||||
"inferred-dark": 10000
|
||||
|
|
|
|||
|
|
@ -9,6 +9,7 @@
|
|||
"@islandflow/bus": "workspace:*",
|
||||
"@islandflow/config": "workspace:*",
|
||||
"@islandflow/observability": "workspace:*",
|
||||
"@islandflow/refdata": "workspace:*",
|
||||
"@islandflow/storage": "workspace:*",
|
||||
"@islandflow/types": "workspace:*",
|
||||
"redis": "^5.10.0",
|
||||
|
|
|
|||
|
|
@ -1,5 +1,10 @@
|
|||
import { readEnv } from "@islandflow/config";
|
||||
import { createLogger } from "@islandflow/observability";
|
||||
import {
|
||||
createEmptyEventCalendarProvider,
|
||||
loadEventCalendarProviderFromFile,
|
||||
type EventCalendarProvider
|
||||
} from "@islandflow/refdata/event-calendar";
|
||||
import {
|
||||
SUBJECT_ALERTS,
|
||||
SUBJECT_CLASSIFIER_HITS,
|
||||
|
|
@ -8,6 +13,7 @@ import {
|
|||
SUBJECT_EQUITY_QUOTES,
|
||||
SUBJECT_INFERRED_DARK,
|
||||
SUBJECT_FLOW_PACKETS,
|
||||
SUBJECT_SMART_MONEY_EVENTS,
|
||||
SUBJECT_OPTION_NBBO,
|
||||
SUBJECT_OPTION_SIGNAL_PRINTS,
|
||||
STREAM_ALERTS,
|
||||
|
|
@ -17,6 +23,7 @@ import {
|
|||
STREAM_EQUITY_QUOTES,
|
||||
STREAM_INFERRED_DARK,
|
||||
STREAM_FLOW_PACKETS,
|
||||
STREAM_SMART_MONEY_EVENTS,
|
||||
STREAM_OPTION_NBBO,
|
||||
STREAM_OPTION_SIGNAL_PRINTS,
|
||||
buildDurableConsumer,
|
||||
|
|
@ -32,11 +39,13 @@ import {
|
|||
ensureEquityPrintJoinsTable,
|
||||
ensureInferredDarkTable,
|
||||
ensureFlowPacketsTable,
|
||||
ensureSmartMoneyEventsTable,
|
||||
insertAlert,
|
||||
insertClassifierHit,
|
||||
insertEquityPrintJoin,
|
||||
insertInferredDark,
|
||||
insertFlowPacket
|
||||
insertFlowPacket,
|
||||
insertSmartMoneyEvent
|
||||
} from "@islandflow/storage";
|
||||
import {
|
||||
AlertEventSchema,
|
||||
|
|
@ -46,6 +55,7 @@ import {
|
|||
EquityQuoteSchema,
|
||||
InferredDarkEventSchema,
|
||||
FlowPacketSchema,
|
||||
SmartMoneyEventSchema,
|
||||
OptionNBBOSchema,
|
||||
OptionPrintSchema,
|
||||
type AlertEvent,
|
||||
|
|
@ -55,11 +65,16 @@ import {
|
|||
type EquityPrintJoin,
|
||||
type InferredDarkEvent,
|
||||
type FlowPacket,
|
||||
type SmartMoneyEvent,
|
||||
type OptionNBBO,
|
||||
type OptionPrint
|
||||
} from "@islandflow/types";
|
||||
import { z } from "zod";
|
||||
import { evaluateClassifiers, type ClassifierConfig } from "./classifiers";
|
||||
import type { ClassifierConfig } from "./classifiers";
|
||||
import {
|
||||
buildSmartMoneyEventFromPacket,
|
||||
deriveClassifierHitsFromSmartMoneyEvent
|
||||
} from "./parent-events";
|
||||
import { parseContractId } from "./contracts";
|
||||
import {
|
||||
createDarkInferenceState,
|
||||
|
|
@ -125,10 +140,12 @@ const envSchema = z.object({
|
|||
CLASSIFIER_MIN_AGGRESSOR_RATIO: z.coerce.number().min(0).max(1).default(0.55),
|
||||
CLASSIFIER_0DTE_MAX_ATM_PCT: z.coerce.number().min(0).max(1).default(0.01),
|
||||
CLASSIFIER_0DTE_MIN_PREMIUM: z.coerce.number().positive().default(20_000),
|
||||
CLASSIFIER_0DTE_MIN_SIZE: z.coerce.number().int().positive().default(400)
|
||||
CLASSIFIER_0DTE_MIN_SIZE: z.coerce.number().int().positive().default(400),
|
||||
SMART_MONEY_EVENT_CALENDAR_PATH: z.string().optional()
|
||||
});
|
||||
|
||||
const env = readEnv(envSchema);
|
||||
let eventCalendarProvider: EventCalendarProvider = createEmptyEventCalendarProvider();
|
||||
|
||||
const classifierConfig: ClassifierConfig = {
|
||||
sweepMinPremium: env.CLASSIFIER_SWEEP_MIN_PREMIUM,
|
||||
|
|
@ -886,7 +903,32 @@ const emitClassifiers = async (
|
|||
js: Awaited<ReturnType<typeof connectJetStreamWithRetry>>["js"],
|
||||
packet: FlowPacket
|
||||
): Promise<void> => {
|
||||
const hits = evaluateClassifiers(packet, classifierConfig);
|
||||
let smartMoneyEvent: SmartMoneyEvent;
|
||||
try {
|
||||
const underlyingId =
|
||||
typeof packet.features.underlying_id === "string"
|
||||
? packet.features.underlying_id
|
||||
: parseContractId(typeof packet.features.option_contract_id === "string" ? packet.features.option_contract_id : "")?.root;
|
||||
const referenceTs =
|
||||
typeof packet.features.end_ts === "number" && Number.isFinite(packet.features.end_ts)
|
||||
? packet.features.end_ts
|
||||
: packet.source_ts;
|
||||
const eventCalendarMatch = underlyingId ? eventCalendarProvider.findNextEvent(underlyingId, referenceTs) : null;
|
||||
smartMoneyEvent = SmartMoneyEventSchema.parse(buildSmartMoneyEventFromPacket(packet, { eventCalendarMatch }));
|
||||
await insertSmartMoneyEvent(clickhouse, smartMoneyEvent);
|
||||
await publishJson(js, SUBJECT_SMART_MONEY_EVENTS, smartMoneyEvent);
|
||||
} catch (error) {
|
||||
if (isExpectedShutdownNatsError(error)) {
|
||||
return;
|
||||
}
|
||||
logger.error("failed to emit smart money event", {
|
||||
error: error instanceof Error ? error.message : String(error),
|
||||
packet_id: packet.id
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
const hits = deriveClassifierHitsFromSmartMoneyEvent(smartMoneyEvent);
|
||||
if (hits.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
|
@ -922,7 +964,7 @@ const emitClassifiers = async (
|
|||
source_ts: packet.source_ts,
|
||||
ingest_ts: packet.ingest_ts,
|
||||
seq: packet.seq,
|
||||
trace_id: `alert:${packet.id}`,
|
||||
trace_id: `alert:${smartMoneyEvent.event_id}`,
|
||||
score,
|
||||
severity,
|
||||
hits: hitEvents.map((hit) => ({
|
||||
|
|
@ -931,7 +973,11 @@ const emitClassifiers = async (
|
|||
direction: hit.direction,
|
||||
explanations: hit.explanations
|
||||
})),
|
||||
evidence_refs: [packet.id, ...packet.members]
|
||||
evidence_refs: [smartMoneyEvent.event_id, packet.id, ...packet.members],
|
||||
...(smartMoneyEvent.primary_profile_id
|
||||
? { primary_profile_id: smartMoneyEvent.primary_profile_id }
|
||||
: {}),
|
||||
profile_scores: smartMoneyEvent.profile_scores
|
||||
});
|
||||
|
||||
try {
|
||||
|
|
@ -1100,6 +1146,19 @@ const run = async () => {
|
|||
num_replicas: 1
|
||||
});
|
||||
|
||||
await ensureStream(jsm, {
|
||||
name: STREAM_SMART_MONEY_EVENTS,
|
||||
subjects: [SUBJECT_SMART_MONEY_EVENTS],
|
||||
retention: "limits",
|
||||
storage: "file",
|
||||
discard: "old",
|
||||
max_msgs_per_subject: -1,
|
||||
max_msgs: -1,
|
||||
max_bytes: -1,
|
||||
max_age: 0,
|
||||
num_replicas: 1
|
||||
});
|
||||
|
||||
await ensureStream(jsm, {
|
||||
name: STREAM_EQUITY_JOINS,
|
||||
subjects: [SUBJECT_EQUITY_JOINS],
|
||||
|
|
@ -1157,6 +1216,19 @@ const run = async () => {
|
|||
database: env.CLICKHOUSE_DATABASE
|
||||
});
|
||||
|
||||
if (env.SMART_MONEY_EVENT_CALENDAR_PATH) {
|
||||
try {
|
||||
eventCalendarProvider = await loadEventCalendarProviderFromFile(env.SMART_MONEY_EVENT_CALENDAR_PATH);
|
||||
logger.info("smart money event calendar loaded", { path: env.SMART_MONEY_EVENT_CALENDAR_PATH });
|
||||
} catch (error) {
|
||||
eventCalendarProvider = createEmptyEventCalendarProvider();
|
||||
logger.warn("smart money event calendar unavailable; scoring will use neutral event features", {
|
||||
path: env.SMART_MONEY_EVENT_CALENDAR_PATH,
|
||||
error: error instanceof Error ? error.message : String(error)
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
const redis = createRedisClient(env.REDIS_URL);
|
||||
redis.on("error", (error) => {
|
||||
logger.warn("redis client error", { error: error instanceof Error ? error.message : String(error) });
|
||||
|
|
@ -1173,6 +1245,7 @@ const run = async () => {
|
|||
|
||||
await retry("clickhouse table init", 120, 500, async () => {
|
||||
await ensureFlowPacketsTable(clickhouse);
|
||||
await ensureSmartMoneyEventsTable(clickhouse);
|
||||
await ensureEquityPrintJoinsTable(clickhouse);
|
||||
await ensureInferredDarkTable(clickhouse);
|
||||
await ensureClassifierHitsTable(clickhouse);
|
||||
|
|
|
|||
329
services/compute/src/parent-events.ts
Normal file
329
services/compute/src/parent-events.ts
Normal file
|
|
@ -0,0 +1,329 @@
|
|||
import {
|
||||
SmartMoneyEventSchema,
|
||||
type ClassifierHit,
|
||||
type FlowPacket,
|
||||
type SmartMoneyDirection,
|
||||
type SmartMoneyEvent,
|
||||
type SmartMoneyFeatures,
|
||||
type SmartMoneyProfileId,
|
||||
type SmartMoneyProfileScore
|
||||
} from "@islandflow/types";
|
||||
import type { EventCalendarMatch } from "@islandflow/refdata/event-calendar";
|
||||
import { parseContractId } from "./contracts";
|
||||
|
||||
const MS_PER_DAY = 86_400_000;
|
||||
const SPECIAL_CONDITIONS = new Set(["AUCTION", "CROSS", "OPENING", "CLOSING", "COMPLEX", "SPREAD"]);
|
||||
|
||||
const clamp = (value: number, min = 0, max = 1): number => {
|
||||
if (!Number.isFinite(value)) {
|
||||
return min;
|
||||
}
|
||||
return Math.max(min, Math.min(max, value));
|
||||
};
|
||||
|
||||
const numberFeature = (packet: FlowPacket, key: string): number => {
|
||||
const value = packet.features[key];
|
||||
return typeof value === "number" && Number.isFinite(value) ? value : 0;
|
||||
};
|
||||
|
||||
const stringFeature = (packet: FlowPacket, key: string): string => {
|
||||
const value = packet.features[key];
|
||||
return typeof value === "string" ? value : "";
|
||||
};
|
||||
|
||||
const boolFeature = (packet: FlowPacket, key: string): boolean | null => {
|
||||
const value = packet.features[key];
|
||||
return typeof value === "boolean" ? value : null;
|
||||
};
|
||||
|
||||
const confidenceBand = (probability: number): SmartMoneyProfileScore["confidence_band"] => {
|
||||
if (probability >= 0.72) {
|
||||
return "high";
|
||||
}
|
||||
if (probability >= 0.52) {
|
||||
return "medium";
|
||||
}
|
||||
return "low";
|
||||
};
|
||||
|
||||
const score = (
|
||||
profile_id: SmartMoneyProfileId,
|
||||
probability: number,
|
||||
direction: SmartMoneyDirection,
|
||||
reasons: string[]
|
||||
): SmartMoneyProfileScore => ({
|
||||
profile_id,
|
||||
probability: clamp(probability),
|
||||
confidence_band: confidenceBand(probability),
|
||||
direction,
|
||||
reasons
|
||||
});
|
||||
|
||||
const getReferenceTs = (packet: FlowPacket): number => {
|
||||
return numberFeature(packet, "end_ts") || packet.source_ts;
|
||||
};
|
||||
|
||||
const getDteDays = (packet: FlowPacket): number | null => {
|
||||
const contract = parseContractId(stringFeature(packet, "option_contract_id"));
|
||||
if (!contract) {
|
||||
return null;
|
||||
}
|
||||
const expiryTs = Date.parse(`${contract.expiry}T00:00:00Z`);
|
||||
if (!Number.isFinite(expiryTs)) {
|
||||
return null;
|
||||
}
|
||||
const diff = expiryTs - getReferenceTs(packet);
|
||||
return diff >= 0 ? Math.ceil(diff / MS_PER_DAY) : null;
|
||||
};
|
||||
|
||||
const inferDirection = (packet: FlowPacket): SmartMoneyDirection => {
|
||||
const structureRights = stringFeature(packet, "structure_rights");
|
||||
const optionType = stringFeature(packet, "option_type") || parseContractId(stringFeature(packet, "option_contract_id"))?.right;
|
||||
const buy = numberFeature(packet, "nbbo_aggressive_buy_ratio");
|
||||
const sell = numberFeature(packet, "nbbo_aggressive_sell_ratio");
|
||||
const sellDominant = sell >= buy + 0.12;
|
||||
|
||||
if (structureRights === "C") {
|
||||
return sellDominant ? "bearish" : "bullish";
|
||||
}
|
||||
if (structureRights === "P") {
|
||||
return sellDominant ? "bullish" : "bearish";
|
||||
}
|
||||
if (optionType === "C") {
|
||||
return sellDominant ? "bearish" : "bullish";
|
||||
}
|
||||
if (optionType === "P") {
|
||||
return sellDominant ? "bullish" : "bearish";
|
||||
}
|
||||
return "neutral";
|
||||
};
|
||||
|
||||
export type SmartMoneyParentEventOptions = {
|
||||
eventCalendarMatch?: EventCalendarMatch | null;
|
||||
};
|
||||
|
||||
const buildFeatures = (packet: FlowPacket, options: SmartMoneyParentEventOptions = {}): SmartMoneyFeatures => {
|
||||
const contractId = stringFeature(packet, "option_contract_id");
|
||||
const contract = parseContractId(contractId);
|
||||
const underlyingMid = numberFeature(packet, "underlying_mid");
|
||||
const quoteAge = numberFeature(packet, "nbbo_age_ms") || numberFeature(packet, "underlying_quote_age_ms");
|
||||
const printCount = Math.max(0, Math.round(numberFeature(packet, "count") || packet.members.length));
|
||||
const staleCount = numberFeature(packet, "nbbo_stale_count");
|
||||
const missingCount = numberFeature(packet, "nbbo_missing_count");
|
||||
const structureLegs = Math.max(0, Math.round(numberFeature(packet, "structure_legs")));
|
||||
const strikeCount = Math.max(1, Math.round(numberFeature(packet, "structure_strikes") || (contract ? 1 : 0)));
|
||||
const specialCount = numberFeature(packet, "special_print_count");
|
||||
const calendarEventTs = options.eventCalendarMatch?.event_ts ?? null;
|
||||
const eventTs = calendarEventTs ?? numberFeature(packet, "corporate_event_ts");
|
||||
const referenceTs = getReferenceTs(packet);
|
||||
const expiryTs = contract ? Date.parse(`${contract.expiry}T00:00:00Z`) : Number.NaN;
|
||||
|
||||
const atmProximity =
|
||||
contract && underlyingMid > 0 ? Math.abs(contract.strike - underlyingMid) / underlyingMid : null;
|
||||
|
||||
return {
|
||||
contract_count: Math.max(1, structureLegs || 1),
|
||||
print_count: printCount,
|
||||
total_size: numberFeature(packet, "total_size"),
|
||||
total_premium: numberFeature(packet, "total_premium"),
|
||||
total_notional: numberFeature(packet, "total_notional"),
|
||||
start_ts: numberFeature(packet, "start_ts") || packet.source_ts,
|
||||
end_ts: numberFeature(packet, "end_ts") || packet.source_ts,
|
||||
window_ms: Math.max(0, Math.round(numberFeature(packet, "window_ms"))),
|
||||
...(contractId ? { option_contract_id: contractId } : {}),
|
||||
...(contract?.right === "C" || contract?.right === "P" ? { option_type: contract.right } : {}),
|
||||
dte_days: getDteDays(packet),
|
||||
moneyness: contract && underlyingMid > 0 ? contract.strike / underlyingMid : null,
|
||||
atm_proximity: atmProximity,
|
||||
aggressor_buy_ratio: clamp(numberFeature(packet, "nbbo_aggressive_buy_ratio")),
|
||||
aggressor_sell_ratio: clamp(numberFeature(packet, "nbbo_aggressive_sell_ratio")),
|
||||
aggressor_ratio: clamp(numberFeature(packet, "nbbo_aggressive_ratio")),
|
||||
nbbo_coverage_ratio: clamp(numberFeature(packet, "nbbo_coverage_ratio")),
|
||||
nbbo_inside_ratio: clamp(numberFeature(packet, "nbbo_inside_ratio")),
|
||||
nbbo_stale_ratio: printCount > 0 ? clamp((staleCount + missingCount) / printCount) : 0,
|
||||
quote_age_ms: quoteAge > 0 ? quoteAge : null,
|
||||
venue_count: Math.max(1, Math.round(numberFeature(packet, "venue_count") || 1)),
|
||||
inter_fill_ms_mean: printCount > 1 ? numberFeature(packet, "window_ms") / Math.max(1, printCount - 1) : null,
|
||||
strike_count: strikeCount,
|
||||
strike_concentration: strikeCount > 0 ? clamp(1 / strikeCount) : 0,
|
||||
...(stringFeature(packet, "structure_type") ? { structure_type: stringFeature(packet, "structure_type") } : {}),
|
||||
structure_legs: structureLegs,
|
||||
same_size_leg_symmetry: clamp(numberFeature(packet, "same_size_leg_symmetry")),
|
||||
net_directional_bias: clamp(
|
||||
numberFeature(packet, "nbbo_aggressive_buy_ratio") - numberFeature(packet, "nbbo_aggressive_sell_ratio"),
|
||||
-1,
|
||||
1
|
||||
),
|
||||
synthetic_iv_shock: numberFeature(packet, "execution_iv_shock") || null,
|
||||
spread_widening: numberFeature(packet, "nbbo_spread_z") || null,
|
||||
underlying_move_bps: numberFeature(packet, "underlying_move_bps") || null,
|
||||
days_to_event: eventTs > 0 ? (eventTs - referenceTs) / MS_PER_DAY : null,
|
||||
expiry_after_event: eventTs > 0 && Number.isFinite(expiryTs) ? expiryTs >= eventTs : null,
|
||||
pre_event_concentration: eventTs > 0 && eventTs >= referenceTs ? clamp(1 - (eventTs - referenceTs) / (21 * MS_PER_DAY)) : null,
|
||||
special_print_ratio: printCount > 0 ? clamp(specialCount / printCount) : 0
|
||||
};
|
||||
};
|
||||
|
||||
const detectSuppression = (packet: FlowPacket, features: SmartMoneyFeatures): string[] => {
|
||||
const reasons: string[] = [];
|
||||
const conditions = String(packet.features.conditions ?? "")
|
||||
.split(",")
|
||||
.map((item) => item.trim().toUpperCase())
|
||||
.filter(Boolean);
|
||||
if (conditions.some((condition) => SPECIAL_CONDITIONS.has(condition)) || features.special_print_ratio >= 0.34) {
|
||||
reasons.push("special_print_or_complex_context");
|
||||
}
|
||||
if (features.nbbo_coverage_ratio < 0.35 || features.nbbo_stale_ratio >= 0.5) {
|
||||
reasons.push("stale_or_missing_quote_context");
|
||||
}
|
||||
if (features.nbbo_inside_ratio >= 0.7 && features.aggressor_ratio < 0.35) {
|
||||
reasons.push("inside_market_or_cross_like_execution");
|
||||
}
|
||||
return reasons;
|
||||
};
|
||||
|
||||
const evaluateProfiles = (
|
||||
packet: FlowPacket,
|
||||
features: SmartMoneyFeatures,
|
||||
suppressed: string[]
|
||||
): SmartMoneyProfileScore[] => {
|
||||
const direction = inferDirection(packet);
|
||||
const dte = features.dte_days ?? 999;
|
||||
const structure = features.structure_type ?? "";
|
||||
const isStructure = features.structure_legs >= 2 || Boolean(structure);
|
||||
const buy = features.aggressor_buy_ratio;
|
||||
const sell = features.aggressor_sell_ratio;
|
||||
const premiumFactor = clamp(features.total_premium / 120_000);
|
||||
const sizeFactor = clamp(features.total_size / 1800);
|
||||
const burstFactor = clamp(features.print_count / 8);
|
||||
const quality = clamp(features.nbbo_coverage_ratio - features.nbbo_stale_ratio);
|
||||
const shortDatedOtm =
|
||||
dte <= 7 && features.atm_proximity !== null && features.atm_proximity >= 0.05 && features.option_type === "C";
|
||||
const nearAtm = features.atm_proximity !== null && features.atm_proximity <= 0.015;
|
||||
const preEvent =
|
||||
features.days_to_event !== null &&
|
||||
features.days_to_event >= 0 &&
|
||||
features.days_to_event <= 21 &&
|
||||
features.expiry_after_event === true;
|
||||
|
||||
const scores = [
|
||||
score(
|
||||
"institutional_directional",
|
||||
suppressed.length > 0 || shortDatedOtm
|
||||
? 0.18
|
||||
: 0.2 + premiumFactor * 0.25 + burstFactor * 0.18 + quality * 0.16 + (buy >= 0.58 || sell >= 0.58 ? 0.12 : 0),
|
||||
direction,
|
||||
[
|
||||
"large_parent_event",
|
||||
"directional_aggressor_mix",
|
||||
...(shortDatedOtm ? ["retail_frenzy_guard"] : []),
|
||||
...suppressed
|
||||
]
|
||||
),
|
||||
score(
|
||||
"retail_whale",
|
||||
0.12 +
|
||||
(shortDatedOtm ? 0.28 : 0) +
|
||||
burstFactor * 0.18 +
|
||||
clamp(features.synthetic_iv_shock ?? 0, 0, 0.2) +
|
||||
(features.total_premium < 100_000 ? 0.1 : 0),
|
||||
direction,
|
||||
["short_dated_otm_attention_flow", "burst_print_pattern"]
|
||||
),
|
||||
score(
|
||||
"event_driven",
|
||||
0.12 + (preEvent ? 0.32 : 0) + premiumFactor * 0.14 + clamp(features.spread_widening ?? 0, 0, 0.16),
|
||||
direction === "unknown" ? "neutral" : direction,
|
||||
["event_calendar_alignment", "expiry_after_event", "pre_event_concentration"]
|
||||
),
|
||||
score(
|
||||
"vol_seller",
|
||||
0.12 + (sell >= 0.58 ? 0.24 : 0) + (structure === "straddle" || structure === "strangle" ? 0.2 : 0) + premiumFactor * 0.14,
|
||||
"neutral",
|
||||
["sell_side_premium", "short_vol_structure_evidence"]
|
||||
),
|
||||
score(
|
||||
"arbitrage",
|
||||
0.08 +
|
||||
(isStructure ? 0.18 : 0) +
|
||||
(features.same_size_leg_symmetry >= 0.7 ? 0.24 : 0) +
|
||||
(Math.abs(features.net_directional_bias) <= 0.15 ? 0.18 : 0),
|
||||
"neutral",
|
||||
["matched_leg_symmetry", "near_flat_directional_exposure"]
|
||||
),
|
||||
score(
|
||||
"hedge_reactive",
|
||||
0.1 +
|
||||
(dte <= 2 && nearAtm ? 0.32 : 0) +
|
||||
clamp(Math.abs(features.underlying_move_bps ?? 0) / 80, 0, 0.18) +
|
||||
sizeFactor * 0.12,
|
||||
direction,
|
||||
["short_dated_atm_gamma_context", "underlying_move_linkage"]
|
||||
)
|
||||
];
|
||||
|
||||
return scores.sort((a, b) => b.probability - a.probability);
|
||||
};
|
||||
|
||||
export const buildSmartMoneyEventFromPacket = (
|
||||
packet: FlowPacket,
|
||||
options: SmartMoneyParentEventOptions = {}
|
||||
): SmartMoneyEvent => {
|
||||
const features = buildFeatures(packet, options);
|
||||
const suppressed = detectSuppression(packet, features);
|
||||
const profileScores = evaluateProfiles(packet, features, suppressed);
|
||||
const primary = profileScores[0] ?? null;
|
||||
const abstained = !primary || primary.probability < 0.42 || suppressed.includes("stale_or_missing_quote_context");
|
||||
const underlying = stringFeature(packet, "underlying_id") || parseContractId(features.option_contract_id ?? "")?.root || "UNKNOWN";
|
||||
const eventKind = features.structure_legs >= 2 || stringFeature(packet, "packet_kind") === "structure"
|
||||
? "multi_leg_event"
|
||||
: "single_leg_event";
|
||||
|
||||
return SmartMoneyEventSchema.parse({
|
||||
source_ts: packet.source_ts,
|
||||
ingest_ts: packet.ingest_ts,
|
||||
seq: packet.seq,
|
||||
trace_id: `smartmoney:${packet.id}`,
|
||||
event_id: `smartmoney:${eventKind}:${packet.id}`,
|
||||
packet_ids: [packet.id],
|
||||
member_print_ids: packet.members,
|
||||
underlying_id: underlying,
|
||||
event_kind: eventKind,
|
||||
event_window_ms: features.window_ms,
|
||||
features,
|
||||
profile_scores: profileScores,
|
||||
primary_profile_id: abstained ? null : primary?.profile_id ?? null,
|
||||
primary_direction: abstained ? "unknown" : primary?.direction ?? "unknown",
|
||||
abstained,
|
||||
suppressed_reasons: suppressed
|
||||
});
|
||||
};
|
||||
|
||||
const LEGACY_PROFILE_MAP: Record<SmartMoneyProfileId, string> = {
|
||||
institutional_directional: "smart_money_institutional_directional",
|
||||
retail_whale: "smart_money_retail_whale",
|
||||
event_driven: "smart_money_event_driven",
|
||||
vol_seller: "smart_money_vol_seller",
|
||||
arbitrage: "smart_money_arbitrage",
|
||||
hedge_reactive: "smart_money_hedge_reactive"
|
||||
};
|
||||
|
||||
export const deriveClassifierHitsFromSmartMoneyEvent = (event: SmartMoneyEvent): ClassifierHit[] => {
|
||||
if (event.abstained || !event.primary_profile_id) {
|
||||
return [];
|
||||
}
|
||||
|
||||
return event.profile_scores
|
||||
.filter((entry) => entry.profile_id === event.primary_profile_id || entry.probability >= 0.5)
|
||||
.slice(0, 3)
|
||||
.map((entry) => ({
|
||||
classifier_id: LEGACY_PROFILE_MAP[entry.profile_id],
|
||||
confidence: entry.probability,
|
||||
direction: entry.direction,
|
||||
explanations: [
|
||||
`Profile ${entry.profile_id} probability ${(entry.probability * 100).toFixed(0)}%.`,
|
||||
...entry.reasons,
|
||||
...event.suppressed_reasons.map((reason) => `Suppression guard: ${reason}.`)
|
||||
]
|
||||
}));
|
||||
};
|
||||
242
services/compute/src/smart-money-evaluation.ts
Normal file
242
services/compute/src/smart-money-evaluation.ts
Normal file
|
|
@ -0,0 +1,242 @@
|
|||
import type { FlowPacket, SmartMoneyDirection, SmartMoneyEvent, SmartMoneyProfileId } from "@islandflow/types";
|
||||
import { buildSmartMoneyEventFromPacket, type SmartMoneyParentEventOptions } from "./parent-events";
|
||||
|
||||
export type SmartMoneyLabel = {
|
||||
event_id: string;
|
||||
profile_id: SmartMoneyProfileId | null;
|
||||
direction?: Exclude<SmartMoneyDirection, "unknown">;
|
||||
realized_return_bps?: number;
|
||||
};
|
||||
|
||||
export type ReplayConsistencyMismatch = {
|
||||
event_id: string;
|
||||
field: "missing_live" | "missing_batch" | "signature";
|
||||
live?: SmartMoneyEventSignature;
|
||||
batch?: SmartMoneyEventSignature;
|
||||
};
|
||||
|
||||
export type ReplayConsistencyReport = {
|
||||
live_count: number;
|
||||
batch_count: number;
|
||||
matched_count: number;
|
||||
mismatches: ReplayConsistencyMismatch[];
|
||||
consistent: boolean;
|
||||
};
|
||||
|
||||
export type SmartMoneyEventSignature = {
|
||||
event_id: string;
|
||||
primary_profile_id: SmartMoneyProfileId | null;
|
||||
primary_direction: SmartMoneyDirection;
|
||||
abstained: boolean;
|
||||
suppressed_reasons: string[];
|
||||
profile_scores: Array<{
|
||||
profile_id: SmartMoneyProfileId;
|
||||
probability: number;
|
||||
confidence_band: SmartMoneyEvent["profile_scores"][number]["confidence_band"];
|
||||
direction: SmartMoneyDirection;
|
||||
}>;
|
||||
};
|
||||
|
||||
export type CalibrationBucket = {
|
||||
min_probability: number;
|
||||
max_probability: number;
|
||||
count: number;
|
||||
average_probability: number;
|
||||
accuracy: number | null;
|
||||
};
|
||||
|
||||
export type SmartMoneyEvaluationReport = {
|
||||
sample_count: number;
|
||||
labeled_count: number;
|
||||
emitted_count: number;
|
||||
abstained_count: number;
|
||||
abstention_rate: number;
|
||||
profile_precision: Partial<Record<SmartMoneyProfileId, number | null>>;
|
||||
profile_recall: Partial<Record<SmartMoneyProfileId, number | null>>;
|
||||
calibration: CalibrationBucket[];
|
||||
economic_sanity: {
|
||||
directional_count: number;
|
||||
direction_hit_rate: number | null;
|
||||
average_signed_return_bps: number | null;
|
||||
};
|
||||
};
|
||||
|
||||
const PROFILES: SmartMoneyProfileId[] = [
|
||||
"institutional_directional",
|
||||
"retail_whale",
|
||||
"event_driven",
|
||||
"vol_seller",
|
||||
"arbitrage",
|
||||
"hedge_reactive"
|
||||
];
|
||||
|
||||
const directionalSign = (direction: SmartMoneyDirection): number => {
|
||||
if (direction === "bullish") {
|
||||
return 1;
|
||||
}
|
||||
if (direction === "bearish") {
|
||||
return -1;
|
||||
}
|
||||
return 0;
|
||||
};
|
||||
|
||||
const round = (value: number, digits = 4): number => {
|
||||
if (!Number.isFinite(value)) {
|
||||
return 0;
|
||||
}
|
||||
return Number(value.toFixed(digits));
|
||||
};
|
||||
|
||||
export const smartMoneyEventSignature = (event: SmartMoneyEvent): SmartMoneyEventSignature => ({
|
||||
event_id: event.event_id,
|
||||
primary_profile_id: event.primary_profile_id,
|
||||
primary_direction: event.primary_direction,
|
||||
abstained: event.abstained,
|
||||
suppressed_reasons: [...event.suppressed_reasons].sort(),
|
||||
profile_scores: event.profile_scores.map((entry) => ({
|
||||
profile_id: entry.profile_id,
|
||||
probability: round(entry.probability, 6),
|
||||
confidence_band: entry.confidence_band,
|
||||
direction: entry.direction
|
||||
}))
|
||||
});
|
||||
|
||||
export const buildSmartMoneyEventsForReplay = (
|
||||
packets: FlowPacket[],
|
||||
optionsByPacketId: Record<string, SmartMoneyParentEventOptions | undefined> = {}
|
||||
): SmartMoneyEvent[] => {
|
||||
return packets
|
||||
.slice()
|
||||
.sort((a, b) => a.source_ts - b.source_ts || a.seq - b.seq || a.id.localeCompare(b.id))
|
||||
.map((packet) => buildSmartMoneyEventFromPacket(packet, optionsByPacketId[packet.id]));
|
||||
};
|
||||
|
||||
export const compareSmartMoneyReplayOutputs = (
|
||||
liveEvents: SmartMoneyEvent[],
|
||||
batchEvents: SmartMoneyEvent[]
|
||||
): ReplayConsistencyReport => {
|
||||
const liveById = new Map(liveEvents.map((event) => [event.event_id, smartMoneyEventSignature(event)]));
|
||||
const batchById = new Map(batchEvents.map((event) => [event.event_id, smartMoneyEventSignature(event)]));
|
||||
const ids = [...new Set([...liveById.keys(), ...batchById.keys()])].sort();
|
||||
const mismatches: ReplayConsistencyMismatch[] = [];
|
||||
|
||||
for (const id of ids) {
|
||||
const live = liveById.get(id);
|
||||
const batch = batchById.get(id);
|
||||
if (!live) {
|
||||
mismatches.push({ event_id: id, field: "missing_live", batch });
|
||||
continue;
|
||||
}
|
||||
if (!batch) {
|
||||
mismatches.push({ event_id: id, field: "missing_batch", live });
|
||||
continue;
|
||||
}
|
||||
if (JSON.stringify(live) !== JSON.stringify(batch)) {
|
||||
mismatches.push({ event_id: id, field: "signature", live, batch });
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
live_count: liveEvents.length,
|
||||
batch_count: batchEvents.length,
|
||||
matched_count: ids.length - mismatches.length,
|
||||
mismatches,
|
||||
consistent: mismatches.length === 0
|
||||
};
|
||||
};
|
||||
|
||||
export const evaluateSmartMoneyEvents = (
|
||||
events: SmartMoneyEvent[],
|
||||
labels: SmartMoneyLabel[],
|
||||
bucketCount = 5
|
||||
): SmartMoneyEvaluationReport => {
|
||||
const labelsById = new Map(labels.map((label) => [label.event_id, label]));
|
||||
const labeledEvents = events
|
||||
.map((event) => ({ event, label: labelsById.get(event.event_id) }))
|
||||
.filter((entry): entry is { event: SmartMoneyEvent; label: SmartMoneyLabel } => Boolean(entry.label));
|
||||
|
||||
const emitted = events.filter((event) => !event.abstained && event.primary_profile_id);
|
||||
const profilePrecision: SmartMoneyEvaluationReport["profile_precision"] = {};
|
||||
const profileRecall: SmartMoneyEvaluationReport["profile_recall"] = {};
|
||||
|
||||
for (const profile of PROFILES) {
|
||||
const predicted = labeledEvents.filter((entry) => entry.event.primary_profile_id === profile);
|
||||
const actual = labeledEvents.filter((entry) => entry.label.profile_id === profile);
|
||||
const truePositive = predicted.filter((entry) => entry.label.profile_id === profile).length;
|
||||
profilePrecision[profile] = predicted.length > 0 ? round(truePositive / predicted.length) : null;
|
||||
profileRecall[profile] = actual.length > 0 ? round(truePositive / actual.length) : null;
|
||||
}
|
||||
|
||||
const calibration = buildCalibration(labeledEvents, Math.max(1, Math.floor(bucketCount)));
|
||||
const economic = buildEconomicSanity(labeledEvents);
|
||||
|
||||
return {
|
||||
sample_count: events.length,
|
||||
labeled_count: labeledEvents.length,
|
||||
emitted_count: emitted.length,
|
||||
abstained_count: events.filter((event) => event.abstained).length,
|
||||
abstention_rate: events.length > 0 ? round(events.filter((event) => event.abstained).length / events.length) : 0,
|
||||
profile_precision: profilePrecision,
|
||||
profile_recall: profileRecall,
|
||||
calibration,
|
||||
economic_sanity: economic
|
||||
};
|
||||
};
|
||||
|
||||
const buildCalibration = (
|
||||
entries: Array<{ event: SmartMoneyEvent; label: SmartMoneyLabel }>,
|
||||
bucketCount: number
|
||||
): CalibrationBucket[] => {
|
||||
const buckets = Array.from({ length: bucketCount }, (_, index) => ({
|
||||
min_probability: round(index / bucketCount),
|
||||
max_probability: round((index + 1) / bucketCount),
|
||||
probabilities: [] as number[],
|
||||
correct: 0
|
||||
}));
|
||||
|
||||
for (const { event, label } of entries) {
|
||||
const probability = event.profile_scores.find((entry) => entry.profile_id === event.primary_profile_id)?.probability ?? 0;
|
||||
const index = Math.min(bucketCount - 1, Math.floor(probability * bucketCount));
|
||||
buckets[index].probabilities.push(probability);
|
||||
if (!event.abstained && event.primary_profile_id === label.profile_id) {
|
||||
buckets[index].correct += 1;
|
||||
}
|
||||
}
|
||||
|
||||
return buckets.map((bucket) => ({
|
||||
min_probability: bucket.min_probability,
|
||||
max_probability: bucket.max_probability,
|
||||
count: bucket.probabilities.length,
|
||||
average_probability:
|
||||
bucket.probabilities.length > 0
|
||||
? round(bucket.probabilities.reduce((sum, value) => sum + value, 0) / bucket.probabilities.length)
|
||||
: 0,
|
||||
accuracy: bucket.probabilities.length > 0 ? round(bucket.correct / bucket.probabilities.length) : null
|
||||
}));
|
||||
};
|
||||
|
||||
const buildEconomicSanity = (
|
||||
entries: Array<{ event: SmartMoneyEvent; label: SmartMoneyLabel }>
|
||||
): SmartMoneyEvaluationReport["economic_sanity"] => {
|
||||
const directional = entries
|
||||
.map(({ event, label }) => ({
|
||||
sign: directionalSign(event.primary_direction),
|
||||
realized: label.realized_return_bps
|
||||
}))
|
||||
.filter((entry): entry is { sign: number; realized: number } => entry.sign !== 0 && Number.isFinite(entry.realized));
|
||||
|
||||
if (directional.length === 0) {
|
||||
return {
|
||||
directional_count: 0,
|
||||
direction_hit_rate: null,
|
||||
average_signed_return_bps: null
|
||||
};
|
||||
}
|
||||
|
||||
const signedReturns = directional.map((entry) => entry.sign * entry.realized);
|
||||
return {
|
||||
directional_count: directional.length,
|
||||
direction_hit_rate: round(signedReturns.filter((value) => value > 0).length / directional.length),
|
||||
average_signed_return_bps: round(signedReturns.reduce((sum, value) => sum + value, 0) / signedReturns.length, 2)
|
||||
};
|
||||
};
|
||||
112
services/compute/tests/parent-events.test.ts
Normal file
112
services/compute/tests/parent-events.test.ts
Normal file
|
|
@ -0,0 +1,112 @@
|
|||
import { describe, expect, it } from "bun:test";
|
||||
import {
|
||||
buildSmartMoneyEventFromPacket,
|
||||
deriveClassifierHitsFromSmartMoneyEvent
|
||||
} from "../src/parent-events";
|
||||
import { buildFlowPacket } from "./helpers";
|
||||
|
||||
describe("smart money parent events", () => {
|
||||
it("scores institutional directional parent events and derives legacy hits", () => {
|
||||
const packet = buildFlowPacket({
|
||||
id: "flowpacket:institutional",
|
||||
source_ts: Date.parse("2025-01-15T15:00:00Z"),
|
||||
features: {
|
||||
option_contract_id: "SPY-2025-02-21-450-C",
|
||||
underlying_id: "SPY",
|
||||
count: 8,
|
||||
window_ms: 450,
|
||||
total_size: 2200,
|
||||
total_premium: 180_000,
|
||||
total_notional: 18_000_000,
|
||||
nbbo_coverage_ratio: 0.92,
|
||||
nbbo_aggressive_ratio: 0.82,
|
||||
nbbo_aggressive_buy_ratio: 0.78,
|
||||
nbbo_aggressive_sell_ratio: 0.04,
|
||||
nbbo_inside_ratio: 0.08,
|
||||
underlying_mid: 448
|
||||
}
|
||||
});
|
||||
|
||||
const event = buildSmartMoneyEventFromPacket(packet);
|
||||
expect(event.event_kind).toBe("single_leg_event");
|
||||
expect(event.primary_profile_id).toBe("institutional_directional");
|
||||
expect(event.primary_direction).toBe("bullish");
|
||||
|
||||
const hits = deriveClassifierHitsFromSmartMoneyEvent(event);
|
||||
expect(hits[0]?.classifier_id).toBe("smart_money_institutional_directional");
|
||||
});
|
||||
|
||||
it("abstains when quote context is stale or missing", () => {
|
||||
const packet = buildFlowPacket({
|
||||
id: "flowpacket:stale",
|
||||
features: {
|
||||
option_contract_id: "SPY-2025-02-21-450-C",
|
||||
count: 8,
|
||||
window_ms: 450,
|
||||
total_size: 2200,
|
||||
total_premium: 180_000,
|
||||
nbbo_coverage_ratio: 0.1,
|
||||
nbbo_missing_count: 8
|
||||
}
|
||||
});
|
||||
|
||||
const event = buildSmartMoneyEventFromPacket(packet);
|
||||
expect(event.abstained).toBe(true);
|
||||
expect(event.primary_profile_id).toBeNull();
|
||||
expect(event.suppressed_reasons).toContain("stale_or_missing_quote_context");
|
||||
});
|
||||
|
||||
it("uses timestamp-available event calendar matches for event-driven scoring", () => {
|
||||
const packet = buildFlowPacket({
|
||||
id: "flowpacket:event-driven",
|
||||
source_ts: Date.parse("2025-01-15T15:00:00Z"),
|
||||
features: {
|
||||
option_contract_id: "AAPL-2025-02-07-225-C",
|
||||
underlying_id: "AAPL",
|
||||
count: 1,
|
||||
window_ms: 450,
|
||||
total_size: 1800,
|
||||
total_premium: 160_000,
|
||||
total_notional: 16_000_000,
|
||||
nbbo_coverage_ratio: 0.5,
|
||||
nbbo_aggressive_ratio: 0.4,
|
||||
nbbo_aggressive_buy_ratio: 0.4,
|
||||
nbbo_aggressive_sell_ratio: 0.1,
|
||||
nbbo_inside_ratio: 0.08,
|
||||
underlying_mid: 224
|
||||
}
|
||||
});
|
||||
|
||||
const event = buildSmartMoneyEventFromPacket(packet, {
|
||||
eventCalendarMatch: {
|
||||
underlying_id: "AAPL",
|
||||
event_ts: Date.parse("2025-01-31T21:00:00Z"),
|
||||
event_kind: "earnings",
|
||||
announced_ts: Date.parse("2024-12-20T21:00:00Z"),
|
||||
days_to_event: 16.25
|
||||
}
|
||||
});
|
||||
|
||||
expect(event.features.days_to_event).toBeCloseTo(16.25);
|
||||
expect(event.features.expiry_after_event).toBe(true);
|
||||
expect(event.primary_profile_id).toBe("event_driven");
|
||||
});
|
||||
|
||||
it("keeps event-calendar features neutral when no match is available", () => {
|
||||
const packet = buildFlowPacket({
|
||||
id: "flowpacket:no-calendar",
|
||||
source_ts: Date.parse("2025-01-15T15:00:00Z"),
|
||||
features: {
|
||||
option_contract_id: "AAPL-2025-02-07-225-C",
|
||||
underlying_id: "AAPL",
|
||||
total_premium: 160_000,
|
||||
nbbo_coverage_ratio: 0.92
|
||||
}
|
||||
});
|
||||
|
||||
const event = buildSmartMoneyEventFromPacket(packet);
|
||||
expect(event.features.days_to_event).toBeNull();
|
||||
expect(event.features.expiry_after_event).toBeNull();
|
||||
expect(event.features.pre_event_concentration).toBeNull();
|
||||
});
|
||||
});
|
||||
153
services/compute/tests/smart-money-evaluation.test.ts
Normal file
153
services/compute/tests/smart-money-evaluation.test.ts
Normal file
|
|
@ -0,0 +1,153 @@
|
|||
import { describe, expect, it } from "bun:test";
|
||||
import { buildSmartMoneyEventFromPacket } from "../src/parent-events";
|
||||
import {
|
||||
buildSmartMoneyEventsForReplay,
|
||||
compareSmartMoneyReplayOutputs,
|
||||
evaluateSmartMoneyEvents
|
||||
} from "../src/smart-money-evaluation";
|
||||
import { buildFlowPacket } from "./helpers";
|
||||
|
||||
const institutionalPacket = buildFlowPacket({
|
||||
id: "flowpacket:eval-institutional",
|
||||
seq: 2,
|
||||
source_ts: Date.parse("2025-01-15T15:00:01Z"),
|
||||
features: {
|
||||
option_contract_id: "SPY-2025-02-21-450-C",
|
||||
underlying_id: "SPY",
|
||||
count: 8,
|
||||
window_ms: 450,
|
||||
total_size: 2200,
|
||||
total_premium: 180_000,
|
||||
total_notional: 18_000_000,
|
||||
nbbo_coverage_ratio: 0.92,
|
||||
nbbo_aggressive_ratio: 0.82,
|
||||
nbbo_aggressive_buy_ratio: 0.78,
|
||||
nbbo_aggressive_sell_ratio: 0.04,
|
||||
nbbo_inside_ratio: 0.08,
|
||||
underlying_mid: 448
|
||||
}
|
||||
});
|
||||
|
||||
const eventDrivenPacket = buildFlowPacket({
|
||||
id: "flowpacket:eval-event-driven",
|
||||
seq: 1,
|
||||
source_ts: Date.parse("2025-01-15T15:00:00Z"),
|
||||
features: {
|
||||
option_contract_id: "AAPL-2025-02-07-225-C",
|
||||
underlying_id: "AAPL",
|
||||
count: 1,
|
||||
window_ms: 450,
|
||||
total_size: 1800,
|
||||
total_premium: 160_000,
|
||||
total_notional: 16_000_000,
|
||||
nbbo_coverage_ratio: 0.5,
|
||||
nbbo_aggressive_ratio: 0.4,
|
||||
nbbo_aggressive_buy_ratio: 0.4,
|
||||
nbbo_aggressive_sell_ratio: 0.1,
|
||||
nbbo_inside_ratio: 0.08,
|
||||
underlying_mid: 224
|
||||
}
|
||||
});
|
||||
|
||||
const stalePacket = buildFlowPacket({
|
||||
id: "flowpacket:eval-stale",
|
||||
seq: 3,
|
||||
source_ts: Date.parse("2025-01-15T15:00:02Z"),
|
||||
features: {
|
||||
option_contract_id: "SPY-2025-02-21-450-C",
|
||||
underlying_id: "SPY",
|
||||
count: 8,
|
||||
window_ms: 450,
|
||||
total_size: 2200,
|
||||
total_premium: 180_000,
|
||||
nbbo_coverage_ratio: 0.1,
|
||||
nbbo_missing_count: 8
|
||||
}
|
||||
});
|
||||
|
||||
const calendarOptions = {
|
||||
"flowpacket:eval-event-driven": {
|
||||
eventCalendarMatch: {
|
||||
underlying_id: "AAPL",
|
||||
event_ts: Date.parse("2025-01-31T21:00:00Z"),
|
||||
event_kind: "earnings",
|
||||
announced_ts: Date.parse("2024-12-20T21:00:00Z"),
|
||||
days_to_event: 16.25
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
describe("smart money evaluation utilities", () => {
|
||||
it("compares replay-style live and batch outputs with stable event signatures", () => {
|
||||
const liveEvents = [institutionalPacket, eventDrivenPacket, stalePacket].map((packet) =>
|
||||
buildSmartMoneyEventFromPacket(packet, calendarOptions[packet.id])
|
||||
);
|
||||
const batchEvents = buildSmartMoneyEventsForReplay(
|
||||
[stalePacket, institutionalPacket, eventDrivenPacket],
|
||||
calendarOptions
|
||||
);
|
||||
|
||||
const report = compareSmartMoneyReplayOutputs(liveEvents, batchEvents);
|
||||
expect(report.consistent).toBe(true);
|
||||
expect(report.live_count).toBe(3);
|
||||
expect(report.batch_count).toBe(3);
|
||||
expect(report.matched_count).toBe(3);
|
||||
expect(report.mismatches).toEqual([]);
|
||||
});
|
||||
|
||||
it("reports signature mismatches when live and batch scoring diverge", () => {
|
||||
const liveEvent = buildSmartMoneyEventFromPacket(institutionalPacket);
|
||||
const batchEvent = {
|
||||
...liveEvent,
|
||||
primary_profile_id: "retail_whale" as const
|
||||
};
|
||||
|
||||
const report = compareSmartMoneyReplayOutputs([liveEvent], [batchEvent]);
|
||||
expect(report.consistent).toBe(false);
|
||||
expect(report.mismatches).toHaveLength(1);
|
||||
expect(report.mismatches[0]?.field).toBe("signature");
|
||||
});
|
||||
|
||||
it("summarizes precision, recall, calibration, abstention rate, and economic sanity", () => {
|
||||
const events = buildSmartMoneyEventsForReplay(
|
||||
[institutionalPacket, eventDrivenPacket, stalePacket],
|
||||
calendarOptions
|
||||
);
|
||||
const report = evaluateSmartMoneyEvents(
|
||||
events,
|
||||
[
|
||||
{
|
||||
event_id: "smartmoney:single_leg_event:flowpacket:eval-institutional",
|
||||
profile_id: "institutional_directional",
|
||||
direction: "bullish",
|
||||
realized_return_bps: 42
|
||||
},
|
||||
{
|
||||
event_id: "smartmoney:single_leg_event:flowpacket:eval-event-driven",
|
||||
profile_id: "event_driven",
|
||||
direction: "bullish",
|
||||
realized_return_bps: 18
|
||||
},
|
||||
{
|
||||
event_id: "smartmoney:single_leg_event:flowpacket:eval-stale",
|
||||
profile_id: null,
|
||||
realized_return_bps: -12
|
||||
}
|
||||
],
|
||||
4
|
||||
);
|
||||
|
||||
expect(report.sample_count).toBe(3);
|
||||
expect(report.labeled_count).toBe(3);
|
||||
expect(report.emitted_count).toBe(2);
|
||||
expect(report.abstained_count).toBe(1);
|
||||
expect(report.abstention_rate).toBeCloseTo(1 / 3);
|
||||
expect(report.profile_precision.institutional_directional).toBe(1);
|
||||
expect(report.profile_recall.event_driven).toBe(1);
|
||||
expect(report.calibration).toHaveLength(4);
|
||||
expect(report.calibration.reduce((sum, bucket) => sum + bucket.count, 0)).toBe(3);
|
||||
expect(report.economic_sanity.directional_count).toBe(2);
|
||||
expect(report.economic_sanity.direction_hit_rate).toBe(1);
|
||||
expect(report.economic_sanity.average_signed_return_bps).toBe(30);
|
||||
});
|
||||
});
|
||||
|
|
@ -6,8 +6,7 @@ import WebSocket from "ws";
|
|||
export type AlpacaEquitiesFeed = "iex" | "sip";
|
||||
|
||||
export type AlpacaEquitiesAdapterConfig = {
|
||||
keyId: string;
|
||||
secretKey: string;
|
||||
apiKey: string;
|
||||
restUrl: string;
|
||||
wsBaseUrl: string;
|
||||
feed: AlpacaEquitiesFeed;
|
||||
|
|
@ -63,10 +62,11 @@ const normalizeSymbols = (symbols: string[]): string[] => {
|
|||
return result;
|
||||
};
|
||||
|
||||
const buildHeaders = (config: AlpacaEquitiesAdapterConfig): Record<string, string> => ({
|
||||
"APCA-API-KEY-ID": config.keyId,
|
||||
"APCA-API-SECRET-KEY": config.secretKey
|
||||
});
|
||||
const buildHeaders = (config: AlpacaEquitiesAdapterConfig): Record<string, string> => {
|
||||
return {
|
||||
Authorization: `Bearer ${config.apiKey}`
|
||||
};
|
||||
};
|
||||
|
||||
const parseTimestamp = (value: string): number => {
|
||||
const parsed = Date.parse(value);
|
||||
|
|
@ -184,8 +184,8 @@ export const createAlpacaEquitiesAdapter = (
|
|||
return {
|
||||
name: "alpaca",
|
||||
start: async (handlers: EquityIngestHandlers) => {
|
||||
if (!config.keyId || !config.secretKey) {
|
||||
throw new Error("Alpaca equities adapter requires ALPACA_KEY_ID and ALPACA_SECRET_KEY.");
|
||||
if (!config.apiKey) {
|
||||
throw new Error("Alpaca equities adapter requires ALPACA_API_KEY.");
|
||||
}
|
||||
|
||||
const symbols = normalizeSymbols(config.symbols);
|
||||
|
|
@ -195,7 +195,9 @@ export const createAlpacaEquitiesAdapter = (
|
|||
|
||||
const exchangeNameMap = await fetchExchangeMeta(config);
|
||||
const wsUrl = buildWsUrl(config.wsBaseUrl, config.feed);
|
||||
const ws = new WebSocket(wsUrl);
|
||||
const ws = new WebSocket(wsUrl, {
|
||||
headers: buildHeaders(config)
|
||||
});
|
||||
|
||||
let seq = 0;
|
||||
let stopped = false;
|
||||
|
|
@ -205,8 +207,8 @@ export const createAlpacaEquitiesAdapter = (
|
|||
ws.send(
|
||||
JSON.stringify({
|
||||
action: "auth",
|
||||
key: config.keyId,
|
||||
secret: config.secretKey
|
||||
key: config.apiKey,
|
||||
secret: ""
|
||||
})
|
||||
);
|
||||
});
|
||||
|
|
|
|||
|
|
@ -41,8 +41,7 @@ const envSchema = z.object({
|
|||
SYNTHETIC_EQUITIES_MODE: z.string().default(""),
|
||||
|
||||
// Alpaca (equities)
|
||||
ALPACA_KEY_ID: z.string().default(""),
|
||||
ALPACA_SECRET_KEY: z.string().default(""),
|
||||
ALPACA_API_KEY: z.string().default(""),
|
||||
ALPACA_REST_URL: z.string().default("https://data.alpaca.markets"),
|
||||
ALPACA_WS_BASE_URL: z.string().default("wss://stream.data.alpaca.markets"),
|
||||
ALPACA_UNDERLYINGS: z.string().default("SPY,NVDA,AAPL"),
|
||||
|
|
@ -167,9 +166,13 @@ const selectAdapter = (name: string): EquityIngestAdapter => {
|
|||
}
|
||||
|
||||
if (name === "alpaca") {
|
||||
if (!env.ALPACA_API_KEY) {
|
||||
logger.warn("alpaca credentials missing; set ALPACA_API_KEY");
|
||||
throw new Error("ALPACA_API_KEY is required for the alpaca adapter.");
|
||||
}
|
||||
|
||||
return createAlpacaEquitiesAdapter({
|
||||
keyId: env.ALPACA_KEY_ID,
|
||||
secretKey: env.ALPACA_SECRET_KEY,
|
||||
apiKey: env.ALPACA_API_KEY,
|
||||
restUrl: env.ALPACA_REST_URL,
|
||||
wsBaseUrl: env.ALPACA_WS_BASE_URL,
|
||||
feed: env.ALPACA_EQUITIES_FEED,
|
||||
|
|
|
|||
|
|
@ -6,8 +6,7 @@ import WebSocket from "ws";
|
|||
type AlpacaFeed = "indicative" | "opra";
|
||||
|
||||
type AlpacaOptionsAdapterConfig = {
|
||||
keyId: string;
|
||||
secretKey: string;
|
||||
apiKey: string;
|
||||
restUrl: string;
|
||||
wsBaseUrl: string;
|
||||
feed: AlpacaFeed;
|
||||
|
|
@ -148,10 +147,11 @@ const normalizeUnderlyings = (value: string[]): string[] => {
|
|||
return result;
|
||||
};
|
||||
|
||||
const buildHeaders = (config: AlpacaOptionsAdapterConfig): Record<string, string> => ({
|
||||
"APCA-API-KEY-ID": config.keyId,
|
||||
"APCA-API-SECRET-KEY": config.secretKey
|
||||
});
|
||||
const buildHeaders = (config: AlpacaOptionsAdapterConfig): Record<string, string> => {
|
||||
return {
|
||||
Authorization: `Bearer ${config.apiKey}`
|
||||
};
|
||||
};
|
||||
|
||||
const fetchJson = async <T>(
|
||||
url: URL,
|
||||
|
|
@ -398,8 +398,8 @@ export const createAlpacaOptionsAdapter = (
|
|||
return {
|
||||
name: "alpaca",
|
||||
start: async (handlers: OptionIngestHandlers) => {
|
||||
if (!config.keyId || !config.secretKey) {
|
||||
throw new Error("Alpaca adapter requires ALPACA_KEY_ID and ALPACA_SECRET_KEY.");
|
||||
if (!config.apiKey) {
|
||||
throw new Error("Alpaca adapter requires ALPACA_API_KEY.");
|
||||
}
|
||||
|
||||
const underlyings = normalizeUnderlyings(config.underlyings);
|
||||
|
|
|
|||
|
|
@ -1,7 +1,9 @@
|
|||
import {
|
||||
SP500_SYMBOLS,
|
||||
type FlowPacket,
|
||||
type OptionNBBO,
|
||||
type OptionPrint,
|
||||
type SmartMoneyProfileId,
|
||||
type SyntheticMarketMode
|
||||
} from "@islandflow/types";
|
||||
import type { OptionIngestAdapter, OptionIngestHandlers } from "./types";
|
||||
|
|
@ -23,7 +25,9 @@ type Burst = {
|
|||
printCount: number;
|
||||
priceStep: number;
|
||||
scenarioId: string;
|
||||
label: SyntheticScenarioLabel;
|
||||
seed: number;
|
||||
flowFeatures: FlowPacket["features"];
|
||||
};
|
||||
|
||||
export type SyntheticContractIvState = {
|
||||
|
|
@ -58,73 +62,157 @@ type WeightedValue<T> = {
|
|||
type Scenario = {
|
||||
id: string;
|
||||
weight: number;
|
||||
label: SyntheticScenarioLabel;
|
||||
right: "C" | "P" | "either";
|
||||
countRange: [number, number];
|
||||
sizeRange: [number, number];
|
||||
targetNotionalRange: [number, number];
|
||||
priceTrend: "up" | "down" | "flat";
|
||||
expiryOffsets?: number[];
|
||||
underlying?: number;
|
||||
strikeMoneyness?: number;
|
||||
flowFeatures: FlowPacket["features"];
|
||||
conditions?: string[];
|
||||
};
|
||||
|
||||
export type SyntheticScenarioLabel = SmartMoneyProfileId | "neutral_noise";
|
||||
|
||||
export type SyntheticSmartMoneyScenario = {
|
||||
id: string;
|
||||
label: SyntheticScenarioLabel;
|
||||
hiddenLabel: SyntheticScenarioLabel;
|
||||
};
|
||||
|
||||
const SMART_MONEY_SCENARIO_IDS = [
|
||||
"institutional_directional",
|
||||
"retail_whale",
|
||||
"event_driven",
|
||||
"vol_seller",
|
||||
"arbitrage",
|
||||
"hedge_reactive",
|
||||
"neutral_noise"
|
||||
] as const;
|
||||
|
||||
const REALISTIC_SCENARIOS: Scenario[] = [
|
||||
{
|
||||
id: "ask_lift",
|
||||
weight: 18,
|
||||
label: "institutional_directional",
|
||||
right: "either",
|
||||
countRange: [1, 2],
|
||||
sizeRange: [30, 180],
|
||||
targetNotionalRange: [9_000, 35_000],
|
||||
priceTrend: "flat",
|
||||
flowFeatures: {
|
||||
nbbo_coverage_ratio: 0.88,
|
||||
nbbo_aggressive_ratio: 0.7,
|
||||
nbbo_aggressive_buy_ratio: 0.66,
|
||||
nbbo_aggressive_sell_ratio: 0.08,
|
||||
nbbo_inside_ratio: 0.12,
|
||||
venue_count: 2
|
||||
},
|
||||
conditions: ["FILL"]
|
||||
},
|
||||
{
|
||||
id: "mid_block",
|
||||
weight: 14,
|
||||
label: "arbitrage",
|
||||
right: "either",
|
||||
countRange: [1, 2],
|
||||
sizeRange: [120, 480],
|
||||
targetNotionalRange: [12_000, 45_000],
|
||||
priceTrend: "flat",
|
||||
flowFeatures: {
|
||||
structure_type: "vertical",
|
||||
structure_legs: 2,
|
||||
structure_strikes: 2,
|
||||
same_size_leg_symmetry: 0.74,
|
||||
nbbo_coverage_ratio: 0.82,
|
||||
nbbo_aggressive_ratio: 0.26,
|
||||
nbbo_aggressive_buy_ratio: 0.3,
|
||||
nbbo_aggressive_sell_ratio: 0.24,
|
||||
nbbo_inside_ratio: 0.42,
|
||||
venue_count: 2
|
||||
},
|
||||
conditions: ["FILL"]
|
||||
},
|
||||
{
|
||||
id: "bullish_sweep",
|
||||
weight: 8,
|
||||
label: "institutional_directional",
|
||||
right: "C",
|
||||
countRange: [2, 3],
|
||||
sizeRange: [180, 520],
|
||||
targetNotionalRange: [25_000, 90_000],
|
||||
priceTrend: "up",
|
||||
flowFeatures: {
|
||||
nbbo_coverage_ratio: 0.9,
|
||||
nbbo_aggressive_ratio: 0.82,
|
||||
nbbo_aggressive_buy_ratio: 0.78,
|
||||
nbbo_aggressive_sell_ratio: 0.04,
|
||||
nbbo_inside_ratio: 0.08,
|
||||
venue_count: 4
|
||||
},
|
||||
conditions: ["SWEEP"]
|
||||
},
|
||||
{
|
||||
id: "bearish_sweep",
|
||||
weight: 8,
|
||||
label: "institutional_directional",
|
||||
right: "P",
|
||||
countRange: [2, 3],
|
||||
sizeRange: [180, 520],
|
||||
targetNotionalRange: [25_000, 90_000],
|
||||
priceTrend: "up",
|
||||
flowFeatures: {
|
||||
nbbo_coverage_ratio: 0.9,
|
||||
nbbo_aggressive_ratio: 0.82,
|
||||
nbbo_aggressive_buy_ratio: 0.78,
|
||||
nbbo_aggressive_sell_ratio: 0.04,
|
||||
nbbo_inside_ratio: 0.08,
|
||||
venue_count: 4
|
||||
},
|
||||
conditions: ["SWEEP"]
|
||||
},
|
||||
{
|
||||
id: "contract_spike",
|
||||
weight: 6,
|
||||
label: "retail_whale",
|
||||
right: "either",
|
||||
countRange: [2, 3],
|
||||
sizeRange: [500, 900],
|
||||
targetNotionalRange: [18_000, 70_000],
|
||||
priceTrend: "flat",
|
||||
expiryOffsets: [0, 1, 7],
|
||||
strikeMoneyness: 1.08,
|
||||
flowFeatures: {
|
||||
nbbo_coverage_ratio: 0.76,
|
||||
nbbo_aggressive_ratio: 0.68,
|
||||
nbbo_aggressive_buy_ratio: 0.62,
|
||||
nbbo_aggressive_sell_ratio: 0.08,
|
||||
nbbo_inside_ratio: 0.12,
|
||||
execution_iv_shock: 0.16,
|
||||
venue_count: 3
|
||||
},
|
||||
conditions: ["ISO"]
|
||||
},
|
||||
{
|
||||
id: "noise",
|
||||
weight: 46,
|
||||
label: "neutral_noise",
|
||||
right: "either",
|
||||
countRange: [1, 2],
|
||||
sizeRange: [5, 60],
|
||||
targetNotionalRange: [500, 6_000],
|
||||
priceTrend: "flat",
|
||||
flowFeatures: {
|
||||
nbbo_coverage_ratio: 0.76,
|
||||
nbbo_aggressive_ratio: 0.24,
|
||||
nbbo_aggressive_buy_ratio: 0.24,
|
||||
nbbo_aggressive_sell_ratio: 0.18,
|
||||
nbbo_inside_ratio: 0.52,
|
||||
venue_count: 1
|
||||
},
|
||||
conditions: ["FILL"]
|
||||
}
|
||||
];
|
||||
|
|
@ -133,41 +221,246 @@ const ACTIVE_SCENARIOS: Scenario[] = [
|
|||
{
|
||||
id: "bullish_sweep",
|
||||
weight: 35,
|
||||
label: "institutional_directional",
|
||||
right: "C",
|
||||
countRange: [7, 10],
|
||||
sizeRange: [600, 1800],
|
||||
targetNotionalRange: [120_000, 240_000],
|
||||
priceTrend: "up",
|
||||
flowFeatures: {
|
||||
nbbo_coverage_ratio: 0.94,
|
||||
nbbo_aggressive_ratio: 0.86,
|
||||
nbbo_aggressive_buy_ratio: 0.82,
|
||||
nbbo_aggressive_sell_ratio: 0.03,
|
||||
nbbo_inside_ratio: 0.06,
|
||||
venue_count: 5
|
||||
},
|
||||
conditions: ["SWEEP"]
|
||||
},
|
||||
{
|
||||
id: "bearish_sweep",
|
||||
weight: 35,
|
||||
label: "institutional_directional",
|
||||
right: "P",
|
||||
countRange: [7, 10],
|
||||
sizeRange: [600, 1800],
|
||||
targetNotionalRange: [120_000, 240_000],
|
||||
priceTrend: "up",
|
||||
flowFeatures: {
|
||||
nbbo_coverage_ratio: 0.94,
|
||||
nbbo_aggressive_ratio: 0.86,
|
||||
nbbo_aggressive_buy_ratio: 0.82,
|
||||
nbbo_aggressive_sell_ratio: 0.03,
|
||||
nbbo_inside_ratio: 0.06,
|
||||
venue_count: 5
|
||||
},
|
||||
conditions: ["SWEEP"]
|
||||
},
|
||||
{
|
||||
id: "contract_spike",
|
||||
weight: 20,
|
||||
label: "retail_whale",
|
||||
right: "either",
|
||||
countRange: [5, 8],
|
||||
sizeRange: [1200, 3200],
|
||||
targetNotionalRange: [60_000, 140_000],
|
||||
priceTrend: "flat",
|
||||
expiryOffsets: [0, 1, 7],
|
||||
strikeMoneyness: 1.08,
|
||||
flowFeatures: {
|
||||
nbbo_coverage_ratio: 0.78,
|
||||
nbbo_aggressive_ratio: 0.72,
|
||||
nbbo_aggressive_buy_ratio: 0.66,
|
||||
nbbo_aggressive_sell_ratio: 0.06,
|
||||
nbbo_inside_ratio: 0.1,
|
||||
execution_iv_shock: 0.19,
|
||||
venue_count: 4
|
||||
},
|
||||
conditions: ["ISO"]
|
||||
},
|
||||
{
|
||||
id: "noise",
|
||||
weight: 10,
|
||||
label: "neutral_noise",
|
||||
right: "either",
|
||||
countRange: [2, 4],
|
||||
sizeRange: [10, 200],
|
||||
targetNotionalRange: [500, 5000],
|
||||
priceTrend: "flat",
|
||||
flowFeatures: {
|
||||
nbbo_coverage_ratio: 0.72,
|
||||
nbbo_aggressive_ratio: 0.24,
|
||||
nbbo_aggressive_buy_ratio: 0.24,
|
||||
nbbo_aggressive_sell_ratio: 0.2,
|
||||
nbbo_inside_ratio: 0.52,
|
||||
venue_count: 1
|
||||
},
|
||||
conditions: ["FILL"]
|
||||
}
|
||||
];
|
||||
|
||||
const SMART_MONEY_TEMPLATE_SCENARIOS: Scenario[] = [
|
||||
{
|
||||
id: "institutional_directional",
|
||||
weight: 18,
|
||||
label: "institutional_directional",
|
||||
right: "C",
|
||||
countRange: [8, 10],
|
||||
sizeRange: [1600, 2400],
|
||||
targetNotionalRange: [170_000, 230_000],
|
||||
priceTrend: "up",
|
||||
expiryOffsets: [28, 45],
|
||||
strikeMoneyness: 1.01,
|
||||
flowFeatures: {
|
||||
nbbo_coverage_ratio: 0.94,
|
||||
nbbo_aggressive_ratio: 0.86,
|
||||
nbbo_aggressive_buy_ratio: 0.82,
|
||||
nbbo_aggressive_sell_ratio: 0.04,
|
||||
nbbo_inside_ratio: 0.06,
|
||||
venue_count: 5
|
||||
},
|
||||
conditions: ["SWEEP"]
|
||||
},
|
||||
{
|
||||
id: "retail_whale",
|
||||
weight: 14,
|
||||
label: "retail_whale",
|
||||
right: "C",
|
||||
countRange: [9, 12],
|
||||
sizeRange: [450, 850],
|
||||
targetNotionalRange: [35_000, 75_000],
|
||||
priceTrend: "up",
|
||||
expiryOffsets: [1, 7],
|
||||
strikeMoneyness: 1.1,
|
||||
flowFeatures: {
|
||||
nbbo_coverage_ratio: 0.82,
|
||||
nbbo_aggressive_ratio: 0.74,
|
||||
nbbo_aggressive_buy_ratio: 0.68,
|
||||
nbbo_aggressive_sell_ratio: 0.04,
|
||||
nbbo_inside_ratio: 0.08,
|
||||
execution_iv_shock: 0.19,
|
||||
venue_count: 4
|
||||
},
|
||||
conditions: ["ISO"]
|
||||
},
|
||||
{
|
||||
id: "event_driven",
|
||||
weight: 12,
|
||||
label: "event_driven",
|
||||
right: "C",
|
||||
countRange: [1, 2],
|
||||
sizeRange: [700, 1100],
|
||||
targetNotionalRange: [72_000, 88_000],
|
||||
priceTrend: "flat",
|
||||
expiryOffsets: [28, 45],
|
||||
strikeMoneyness: 1.0,
|
||||
flowFeatures: {
|
||||
corporate_event_ts_offset_days: 14,
|
||||
nbbo_coverage_ratio: 0.38,
|
||||
nbbo_aggressive_ratio: 0.32,
|
||||
nbbo_aggressive_buy_ratio: 0.3,
|
||||
nbbo_aggressive_sell_ratio: 0.08,
|
||||
nbbo_inside_ratio: 0.28,
|
||||
nbbo_spread_z: 0.12,
|
||||
venue_count: 2
|
||||
},
|
||||
conditions: ["FILL"]
|
||||
},
|
||||
{
|
||||
id: "vol_seller",
|
||||
weight: 12,
|
||||
label: "vol_seller",
|
||||
right: "either",
|
||||
countRange: [4, 6],
|
||||
sizeRange: [1300, 2100],
|
||||
targetNotionalRange: [150_000, 210_000],
|
||||
priceTrend: "down",
|
||||
expiryOffsets: [28, 45],
|
||||
strikeMoneyness: 1.0,
|
||||
flowFeatures: {
|
||||
structure_type: "straddle",
|
||||
structure_legs: 2,
|
||||
structure_strikes: 1,
|
||||
structure_rights: "CP",
|
||||
conditions: "COMPLEX",
|
||||
nbbo_coverage_ratio: 0.9,
|
||||
nbbo_aggressive_ratio: 0.72,
|
||||
nbbo_aggressive_buy_ratio: 0.08,
|
||||
nbbo_aggressive_sell_ratio: 0.7,
|
||||
nbbo_inside_ratio: 0.1,
|
||||
same_size_leg_symmetry: 0.66,
|
||||
venue_count: 3
|
||||
},
|
||||
conditions: ["FILL"]
|
||||
},
|
||||
{
|
||||
id: "arbitrage",
|
||||
weight: 12,
|
||||
label: "arbitrage",
|
||||
right: "either",
|
||||
countRange: [4, 6],
|
||||
sizeRange: [900, 1400],
|
||||
targetNotionalRange: [70_000, 115_000],
|
||||
priceTrend: "flat",
|
||||
expiryOffsets: [28, 45],
|
||||
strikeMoneyness: 1.0,
|
||||
flowFeatures: {
|
||||
structure_type: "vertical",
|
||||
structure_legs: 2,
|
||||
structure_strikes: 2,
|
||||
structure_rights: "CP",
|
||||
conditions: "COMPLEX",
|
||||
nbbo_coverage_ratio: 0.86,
|
||||
nbbo_aggressive_ratio: 0.4,
|
||||
nbbo_aggressive_buy_ratio: 0.42,
|
||||
nbbo_aggressive_sell_ratio: 0.38,
|
||||
nbbo_inside_ratio: 0.32,
|
||||
same_size_leg_symmetry: 0.92,
|
||||
venue_count: 3
|
||||
},
|
||||
conditions: ["FILL"]
|
||||
},
|
||||
{
|
||||
id: "hedge_reactive",
|
||||
weight: 12,
|
||||
label: "hedge_reactive",
|
||||
right: "P",
|
||||
countRange: [1, 2],
|
||||
sizeRange: [2600, 3400],
|
||||
targetNotionalRange: [35_000, 50_000],
|
||||
priceTrend: "up",
|
||||
expiryOffsets: [0, 1],
|
||||
strikeMoneyness: 1.0,
|
||||
flowFeatures: {
|
||||
nbbo_coverage_ratio: 0.86,
|
||||
nbbo_aggressive_ratio: 0.58,
|
||||
nbbo_aggressive_buy_ratio: 0.54,
|
||||
nbbo_aggressive_sell_ratio: 0.12,
|
||||
nbbo_inside_ratio: 0.16,
|
||||
underlying_move_bps: -72,
|
||||
venue_count: 3
|
||||
},
|
||||
conditions: ["FILL"]
|
||||
},
|
||||
{
|
||||
id: "neutral_noise",
|
||||
weight: 20,
|
||||
label: "neutral_noise",
|
||||
right: "either",
|
||||
countRange: [1, 2],
|
||||
sizeRange: [10, 70],
|
||||
targetNotionalRange: [800, 7_000],
|
||||
priceTrend: "flat",
|
||||
expiryOffsets: [14, 28, 45, 60],
|
||||
strikeMoneyness: 1.02,
|
||||
flowFeatures: {
|
||||
nbbo_coverage_ratio: 0.78,
|
||||
nbbo_aggressive_ratio: 0.22,
|
||||
nbbo_aggressive_buy_ratio: 0.22,
|
||||
nbbo_aggressive_sell_ratio: 0.18,
|
||||
nbbo_inside_ratio: 0.58,
|
||||
venue_count: 1
|
||||
},
|
||||
conditions: ["FILL"]
|
||||
}
|
||||
];
|
||||
|
|
@ -292,6 +585,25 @@ const SYNTHETIC_PROFILES: Record<SyntheticMarketMode, SyntheticOptionsProfile> =
|
|||
}
|
||||
};
|
||||
|
||||
const SMART_MONEY_TEMPLATE_PROFILE: SyntheticOptionsProfile = {
|
||||
burstRunRange: [1, 1],
|
||||
scenarios: SMART_MONEY_TEMPLATE_SCENARIOS,
|
||||
pricePlacements: {
|
||||
...ACTIVE_PRICE_PLACEMENTS,
|
||||
institutional_directional: ACTIVE_PRICE_PLACEMENTS.bullish_sweep,
|
||||
retail_whale: ACTIVE_PRICE_PLACEMENTS.contract_spike,
|
||||
event_driven: REALISTIC_PRICE_PLACEMENTS.ask_lift,
|
||||
vol_seller: [
|
||||
{ value: "B", weight: 45 },
|
||||
{ value: "BB", weight: 35 },
|
||||
{ value: "MID", weight: 20 }
|
||||
],
|
||||
arbitrage: REALISTIC_PRICE_PLACEMENTS.mid_block,
|
||||
hedge_reactive: ACTIVE_PRICE_PLACEMENTS.bullish_sweep,
|
||||
neutral_noise: REALISTIC_PRICE_PLACEMENTS.noise
|
||||
}
|
||||
};
|
||||
|
||||
const pick = <T,>(items: T[], seed: number): T => {
|
||||
return items[Math.abs(seed) % items.length];
|
||||
};
|
||||
|
|
@ -414,14 +726,18 @@ const buildBurst = (burstIndex: number, now: number, profile: SyntheticOptionsPr
|
|||
const seed = symbolHash + burstIndex * 7;
|
||||
const scenario = pickWeighted(profile.scenarios, seed);
|
||||
const baseUnderlying = 30 + (symbolHash % 470);
|
||||
const expiryOffset = pick(EXPIRY_OFFSETS, symbolHash + burstIndex);
|
||||
const expiryOffset = pick(scenario.expiryOffsets ?? EXPIRY_OFFSETS, symbolHash + burstIndex);
|
||||
const expiry = formatExpiry(now, expiryOffset);
|
||||
const strikeStep = baseUnderlying >= 200 ? 10 : baseUnderlying >= 100 ? 5 : 2.5;
|
||||
const moneynessSteps = scenario.id === "noise" ? 5 : 2;
|
||||
const strikeOffset = pickInt(-moneynessSteps, moneynessSteps, symbolHash + burstIndex * 11);
|
||||
const templateStrike =
|
||||
scenario.strikeMoneyness !== undefined
|
||||
? Math.round((baseUnderlying * scenario.strikeMoneyness) / strikeStep) * strikeStep
|
||||
: null;
|
||||
const strike = Math.max(
|
||||
1,
|
||||
Math.round(baseUnderlying / strikeStep) * strikeStep + strikeOffset * strikeStep
|
||||
templateStrike ?? Math.round(baseUnderlying / strikeStep) * strikeStep + strikeOffset * strikeStep
|
||||
);
|
||||
const right =
|
||||
scenario.right === "either"
|
||||
|
|
@ -463,6 +779,8 @@ const buildBurst = (burstIndex: number, now: number, profile: SyntheticOptionsPr
|
|||
printCount,
|
||||
priceStep,
|
||||
scenarioId: scenario.id,
|
||||
label: scenario.label,
|
||||
flowFeatures: scenario.flowFeatures,
|
||||
seed
|
||||
};
|
||||
};
|
||||
|
|
@ -473,6 +791,68 @@ export const buildSyntheticBurstForTest = (
|
|||
mode: SyntheticMarketMode
|
||||
): Burst => buildBurst(burstIndex, now, SYNTHETIC_PROFILES[mode]);
|
||||
|
||||
export const listSyntheticSmartMoneyScenariosForTest = (): SyntheticSmartMoneyScenario[] =>
|
||||
SMART_MONEY_SCENARIO_IDS.map((id) => ({
|
||||
id,
|
||||
label: id,
|
||||
hiddenLabel: id
|
||||
}));
|
||||
|
||||
export const buildSyntheticSmartMoneyBurstForTest = (
|
||||
scenarioId: (typeof SMART_MONEY_SCENARIO_IDS)[number],
|
||||
now: number
|
||||
): Burst => {
|
||||
const scenarioIndex = SMART_MONEY_TEMPLATE_SCENARIOS.findIndex((scenario) => scenario.id === scenarioId);
|
||||
if (scenarioIndex < 0) {
|
||||
throw new Error(`Unknown synthetic smart-money scenario: ${scenarioId}`);
|
||||
}
|
||||
return buildBurst(scenarioIndex, now, {
|
||||
...SMART_MONEY_TEMPLATE_PROFILE,
|
||||
scenarios: [SMART_MONEY_TEMPLATE_SCENARIOS[scenarioIndex]]
|
||||
});
|
||||
};
|
||||
|
||||
export const buildSyntheticFlowPacketForTest = (
|
||||
scenarioId: (typeof SMART_MONEY_SCENARIO_IDS)[number],
|
||||
now: number
|
||||
): { packet: FlowPacket; hiddenLabel: SyntheticScenarioLabel } => {
|
||||
const burst = buildSyntheticSmartMoneyBurstForTest(scenarioId, now);
|
||||
const corporateEventOffset = Number(burst.flowFeatures.corporate_event_ts_offset_days ?? 0);
|
||||
const flowFeatures: FlowPacket["features"] = {
|
||||
option_contract_id: burst.contractId,
|
||||
underlying_id: burst.contractId.split("-")[0],
|
||||
underlying_mid: burst.underlying,
|
||||
count: burst.printCount,
|
||||
window_ms: Math.max(0, (burst.printCount - 1) * 45),
|
||||
total_size: burst.baseSize * burst.printCount,
|
||||
total_premium: Number((burst.basePrice * burst.baseSize * burst.printCount * OPTION_CONTRACT_MULTIPLIER).toFixed(2)),
|
||||
total_notional: Number((burst.underlying * burst.baseSize * burst.printCount * OPTION_CONTRACT_MULTIPLIER).toFixed(2)),
|
||||
first_price: burst.basePrice,
|
||||
last_price: Number((burst.basePrice * (1 + burst.priceStep * Math.max(0, burst.printCount - 1))).toFixed(2)),
|
||||
nbbo_missing_count: 0,
|
||||
nbbo_stale_count: 0,
|
||||
...burst.flowFeatures
|
||||
};
|
||||
delete flowFeatures.corporate_event_ts_offset_days;
|
||||
if (corporateEventOffset > 0) {
|
||||
flowFeatures.corporate_event_ts = now + corporateEventOffset * MS_PER_DAY;
|
||||
}
|
||||
|
||||
return {
|
||||
hiddenLabel: burst.label,
|
||||
packet: {
|
||||
source_ts: now,
|
||||
ingest_ts: now,
|
||||
seq: SMART_MONEY_SCENARIO_IDS.indexOf(scenarioId) + 1,
|
||||
trace_id: `synthetic-smart-money:${scenarioId}`,
|
||||
id: `synthetic-smart-money:${scenarioId}:${now}`,
|
||||
members: Array.from({ length: burst.printCount }, (_, index) => `${burst.contractId}:${index + 1}`),
|
||||
features: flowFeatures,
|
||||
join_quality: {}
|
||||
}
|
||||
};
|
||||
};
|
||||
|
||||
export const createSyntheticOptionsAdapter = (
|
||||
config: SyntheticOptionsAdapterConfig
|
||||
): OptionIngestAdapter => {
|
||||
|
|
|
|||
|
|
@ -49,8 +49,7 @@ const envSchema = z.object({
|
|||
CLICKHOUSE_URL: z.string().default("http://127.0.0.1:8123"),
|
||||
CLICKHOUSE_DATABASE: z.string().default("default"),
|
||||
OPTIONS_INGEST_ADAPTER: z.string().min(1).default("synthetic"),
|
||||
ALPACA_KEY_ID: z.string().default(""),
|
||||
ALPACA_SECRET_KEY: z.string().default(""),
|
||||
ALPACA_API_KEY: z.string().default(""),
|
||||
ALPACA_REST_URL: z.string().default("https://data.alpaca.markets"),
|
||||
ALPACA_WS_BASE_URL: z.string().default("wss://stream.data.alpaca.markets/v1beta1"),
|
||||
ALPACA_FEED: z.enum(["indicative", "opra"]).default("indicative"),
|
||||
|
|
@ -229,16 +228,15 @@ const selectAdapter = (name: string): OptionIngestAdapter => {
|
|||
}
|
||||
|
||||
if (name === "alpaca") {
|
||||
if (!env.ALPACA_KEY_ID || !env.ALPACA_SECRET_KEY) {
|
||||
logger.warn("alpaca credentials missing; set ALPACA_KEY_ID and ALPACA_SECRET_KEY");
|
||||
throw new Error("ALPACA_KEY_ID and ALPACA_SECRET_KEY are required for the alpaca adapter.");
|
||||
if (!env.ALPACA_API_KEY) {
|
||||
logger.warn("alpaca credentials missing; set ALPACA_API_KEY");
|
||||
throw new Error("ALPACA_API_KEY is required for the alpaca adapter.");
|
||||
}
|
||||
|
||||
const underlyings = env.ALPACA_UNDERLYINGS.split(",").map((symbol) => symbol.trim());
|
||||
|
||||
return createAlpacaOptionsAdapter({
|
||||
keyId: env.ALPACA_KEY_ID,
|
||||
secretKey: env.ALPACA_SECRET_KEY,
|
||||
apiKey: env.ALPACA_API_KEY,
|
||||
restUrl: env.ALPACA_REST_URL,
|
||||
wsBaseUrl: env.ALPACA_WS_BASE_URL,
|
||||
feed: env.ALPACA_FEED,
|
||||
|
|
|
|||
|
|
@ -1,5 +1,13 @@
|
|||
import { describe, expect, it } from "bun:test";
|
||||
import { buildSyntheticBurstForTest, updateSyntheticIvForTest } from "../src/adapters/synthetic";
|
||||
import type { OptionPrint } from "@islandflow/types";
|
||||
import { buildSmartMoneyEventFromPacket } from "../../compute/src/parent-events";
|
||||
import {
|
||||
buildSyntheticBurstForTest,
|
||||
buildSyntheticFlowPacketForTest,
|
||||
createSyntheticOptionsAdapter,
|
||||
listSyntheticSmartMoneyScenariosForTest,
|
||||
updateSyntheticIvForTest
|
||||
} from "../src/adapters/synthetic";
|
||||
|
||||
const totalBurstNotional = (burst: {
|
||||
basePrice: number;
|
||||
|
|
@ -87,3 +95,76 @@ describe("synthetic options IV model", () => {
|
|||
expect(state.iv).toBeLessThanOrEqual(2.5);
|
||||
});
|
||||
});
|
||||
|
||||
describe("synthetic smart-money scenarios", () => {
|
||||
it("provides deterministic labeled parent-event templates for all core profiles plus noise", () => {
|
||||
const scenarios = listSyntheticSmartMoneyScenariosForTest();
|
||||
|
||||
expect(scenarios.map((scenario) => scenario.id)).toEqual([
|
||||
"institutional_directional",
|
||||
"retail_whale",
|
||||
"event_driven",
|
||||
"vol_seller",
|
||||
"arbitrage",
|
||||
"hedge_reactive",
|
||||
"neutral_noise"
|
||||
]);
|
||||
});
|
||||
|
||||
it("scores each labeled scenario as its intended primary profile", () => {
|
||||
const now = Date.parse("2026-01-02T15:00:00Z");
|
||||
const scenarios = listSyntheticSmartMoneyScenariosForTest().filter(
|
||||
(scenario) => scenario.hiddenLabel !== "neutral_noise"
|
||||
);
|
||||
|
||||
for (const scenario of scenarios) {
|
||||
const { packet, hiddenLabel } = buildSyntheticFlowPacketForTest(scenario.id, now);
|
||||
const event = buildSmartMoneyEventFromPacket(packet);
|
||||
const winningScore = event.profile_scores[0];
|
||||
const nearbyWrongScores = event.profile_scores.filter(
|
||||
(score) => score.profile_id !== hiddenLabel && score.probability >= 0.5
|
||||
);
|
||||
|
||||
expect(event.abstained, scenario.id).toBe(false);
|
||||
expect(event.primary_profile_id, scenario.id).toBe(hiddenLabel);
|
||||
expect(winningScore?.profile_id, scenario.id).toBe(hiddenLabel);
|
||||
expect(winningScore?.probability ?? 0, scenario.id).toBeGreaterThanOrEqual(0.5);
|
||||
expect(nearbyWrongScores, scenario.id).toEqual([]);
|
||||
}
|
||||
});
|
||||
|
||||
it("keeps neutral background noise below the emission threshold", () => {
|
||||
const { packet } = buildSyntheticFlowPacketForTest(
|
||||
"neutral_noise",
|
||||
Date.parse("2026-01-02T15:00:00Z")
|
||||
);
|
||||
|
||||
const event = buildSmartMoneyEventFromPacket(packet);
|
||||
|
||||
expect(event.abstained).toBe(true);
|
||||
expect(event.primary_profile_id).toBeNull();
|
||||
expect(event.profile_scores[0]?.probability ?? 1).toBeLessThan(0.42);
|
||||
});
|
||||
|
||||
it("does not expose hidden labels on emitted option prints", async () => {
|
||||
const adapter = createSyntheticOptionsAdapter({
|
||||
emitIntervalMs: 1,
|
||||
mode: "active"
|
||||
});
|
||||
const trades: OptionPrint[] = [];
|
||||
const stop = adapter.start({
|
||||
onTrade: (trade) => {
|
||||
trades.push(trade);
|
||||
}
|
||||
});
|
||||
|
||||
await new Promise((resolve) => setTimeout(resolve, 25));
|
||||
stop();
|
||||
|
||||
expect(trades.length).toBeGreaterThan(0);
|
||||
for (const trade of trades) {
|
||||
expect("hiddenLabel" in trade).toBe(false);
|
||||
expect("label" in trade).toBe(false);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -2,6 +2,9 @@
|
|||
"name": "@islandflow/refdata",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"exports": {
|
||||
"./event-calendar": "./src/event-calendar.ts"
|
||||
},
|
||||
"scripts": {
|
||||
"dev": "bun run src/index.ts"
|
||||
},
|
||||
|
|
|
|||
231
services/refdata/src/event-calendar.ts
Normal file
231
services/refdata/src/event-calendar.ts
Normal file
|
|
@ -0,0 +1,231 @@
|
|||
import { mkdir } from "node:fs/promises";
|
||||
|
||||
export type EventCalendarKind = "earnings" | "dividend" | "corporate_action" | "m_and_a" | "news" | "other";
|
||||
|
||||
export type EventCalendarEntry = {
|
||||
underlying_id: string;
|
||||
event_ts: number;
|
||||
event_kind: EventCalendarKind;
|
||||
announced_ts: number;
|
||||
source?: string;
|
||||
source_event_id?: string;
|
||||
};
|
||||
|
||||
export type EventCalendarMatch = EventCalendarEntry & {
|
||||
days_to_event: number;
|
||||
};
|
||||
|
||||
export type EventCalendarProvider = {
|
||||
findNextEvent(underlyingId: string, asOfTs: number): EventCalendarMatch | null;
|
||||
};
|
||||
|
||||
export type AlphaVantageEarningsCalendarOptions = {
|
||||
apiKey: string;
|
||||
horizon?: "3month" | "6month" | "12month";
|
||||
symbol?: string;
|
||||
nowTs?: number;
|
||||
fetchFn?: typeof fetch;
|
||||
};
|
||||
|
||||
const MS_PER_DAY = 86_400_000;
|
||||
const ALPHA_VANTAGE_URL = "https://www.alphavantage.co/query";
|
||||
|
||||
const EVENT_KINDS = new Set<EventCalendarKind>([
|
||||
"earnings",
|
||||
"dividend",
|
||||
"corporate_action",
|
||||
"m_and_a",
|
||||
"news",
|
||||
"other"
|
||||
]);
|
||||
|
||||
const normalizeUnderlying = (underlyingId: string): string => underlyingId.trim().toUpperCase();
|
||||
|
||||
const asNumber = (value: unknown): number | null => {
|
||||
if (typeof value === "number" && Number.isFinite(value)) {
|
||||
return value;
|
||||
}
|
||||
if (typeof value === "string" && value.trim()) {
|
||||
const parsed = Number(value);
|
||||
if (Number.isFinite(parsed)) {
|
||||
return parsed;
|
||||
}
|
||||
const ts = Date.parse(value);
|
||||
return Number.isFinite(ts) ? ts : null;
|
||||
}
|
||||
return null;
|
||||
};
|
||||
|
||||
const asString = (value: unknown): string | null => (typeof value === "string" && value.trim() ? value.trim() : null);
|
||||
|
||||
const parseCsvLine = (line: string): string[] => {
|
||||
const values: string[] = [];
|
||||
let current = "";
|
||||
let quoted = false;
|
||||
|
||||
for (let index = 0; index < line.length; index += 1) {
|
||||
const char = line[index];
|
||||
const next = line[index + 1];
|
||||
if (char === '"' && quoted && next === '"') {
|
||||
current += '"';
|
||||
index += 1;
|
||||
} else if (char === '"') {
|
||||
quoted = !quoted;
|
||||
} else if (char === "," && !quoted) {
|
||||
values.push(current);
|
||||
current = "";
|
||||
} else {
|
||||
current += char;
|
||||
}
|
||||
}
|
||||
|
||||
values.push(current);
|
||||
return values.map((value) => value.trim());
|
||||
};
|
||||
|
||||
const parseCsv = (csv: string): Record<string, string>[] => {
|
||||
const lines = csv
|
||||
.split(/\r?\n/)
|
||||
.map((line) => line.trim())
|
||||
.filter(Boolean);
|
||||
const [headerLine, ...dataLines] = lines;
|
||||
if (!headerLine) {
|
||||
return [];
|
||||
}
|
||||
|
||||
const headers = parseCsvLine(headerLine);
|
||||
return dataLines.map((line) => {
|
||||
const values = parseCsvLine(line);
|
||||
return Object.fromEntries(headers.map((header, index) => [header, values[index] ?? ""]));
|
||||
});
|
||||
};
|
||||
|
||||
export const parseAlphaVantageEarningsCalendar = (
|
||||
csv: string,
|
||||
announcedTs: number = Date.now()
|
||||
): EventCalendarEntry[] => {
|
||||
return parseCsv(csv).flatMap((row): EventCalendarEntry[] => {
|
||||
const symbol = asString(row.symbol);
|
||||
const reportDate = asString(row.reportDate);
|
||||
if (!symbol || !reportDate) {
|
||||
return [];
|
||||
}
|
||||
|
||||
const eventTs = Date.parse(`${reportDate}T21:00:00Z`);
|
||||
if (!Number.isFinite(eventTs)) {
|
||||
return [];
|
||||
}
|
||||
|
||||
return [
|
||||
{
|
||||
underlying_id: normalizeUnderlying(symbol),
|
||||
event_ts: eventTs,
|
||||
event_kind: "earnings",
|
||||
announced_ts: Math.trunc(announcedTs),
|
||||
source: "alpha_vantage",
|
||||
source_event_id: `${normalizeUnderlying(symbol)}:${reportDate}:earnings`
|
||||
}
|
||||
];
|
||||
});
|
||||
};
|
||||
|
||||
export const parseEventCalendarEntries = (value: unknown): EventCalendarEntry[] => {
|
||||
const rows = Array.isArray(value) ? value : [];
|
||||
return rows.flatMap((row): EventCalendarEntry[] => {
|
||||
if (!row || typeof row !== "object") {
|
||||
return [];
|
||||
}
|
||||
|
||||
const record = row as Record<string, unknown>;
|
||||
const underlying = asString(record.underlying_id ?? record.underlying ?? record.symbol);
|
||||
const eventTs = asNumber(record.event_ts ?? record.event_time ?? record.event_date);
|
||||
const announcedTs = asNumber(record.announced_ts ?? record.available_ts ?? record.as_of_ts ?? record.created_ts) ?? 0;
|
||||
const rawKind = asString(record.event_kind ?? record.kind ?? record.type) ?? "other";
|
||||
const eventKind = EVENT_KINDS.has(rawKind as EventCalendarKind) ? (rawKind as EventCalendarKind) : "other";
|
||||
|
||||
if (!underlying || eventTs === null || eventTs < 0 || announcedTs < 0) {
|
||||
return [];
|
||||
}
|
||||
|
||||
return [
|
||||
{
|
||||
underlying_id: normalizeUnderlying(underlying),
|
||||
event_ts: Math.trunc(eventTs),
|
||||
event_kind: eventKind,
|
||||
announced_ts: Math.trunc(announcedTs),
|
||||
...(asString(record.source) ? { source: asString(record.source) ?? undefined } : {}),
|
||||
...(asString(record.source_event_id ?? record.id)
|
||||
? { source_event_id: asString(record.source_event_id ?? record.id) ?? undefined }
|
||||
: {})
|
||||
}
|
||||
];
|
||||
});
|
||||
};
|
||||
|
||||
export const createStaticEventCalendarProvider = (entries: EventCalendarEntry[]): EventCalendarProvider => {
|
||||
const byUnderlying = new Map<string, EventCalendarEntry[]>();
|
||||
for (const entry of entries) {
|
||||
const key = normalizeUnderlying(entry.underlying_id);
|
||||
const normalized = { ...entry, underlying_id: key };
|
||||
const bucket = byUnderlying.get(key) ?? [];
|
||||
bucket.push(normalized);
|
||||
byUnderlying.set(key, bucket);
|
||||
}
|
||||
|
||||
for (const bucket of byUnderlying.values()) {
|
||||
bucket.sort((a, b) => a.event_ts - b.event_ts || a.announced_ts - b.announced_ts);
|
||||
}
|
||||
|
||||
return {
|
||||
findNextEvent(underlyingId, asOfTs) {
|
||||
const key = normalizeUnderlying(underlyingId);
|
||||
if (!key || !Number.isFinite(asOfTs)) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const bucket = byUnderlying.get(key) ?? [];
|
||||
const entry = bucket.find((candidate) => candidate.announced_ts <= asOfTs && candidate.event_ts >= asOfTs);
|
||||
return entry ? { ...entry, days_to_event: (entry.event_ts - asOfTs) / MS_PER_DAY } : null;
|
||||
}
|
||||
};
|
||||
};
|
||||
|
||||
export const createEmptyEventCalendarProvider = (): EventCalendarProvider => createStaticEventCalendarProvider([]);
|
||||
|
||||
export const loadEventCalendarProviderFromFile = async (path: string): Promise<EventCalendarProvider> => {
|
||||
const text = await Bun.file(path).text();
|
||||
return createStaticEventCalendarProvider(parseEventCalendarEntries(JSON.parse(text)));
|
||||
};
|
||||
|
||||
export const fetchAlphaVantageEarningsCalendar = async (
|
||||
options: AlphaVantageEarningsCalendarOptions
|
||||
): Promise<EventCalendarEntry[]> => {
|
||||
const horizon = options.horizon ?? "3month";
|
||||
const url = new URL(ALPHA_VANTAGE_URL);
|
||||
url.searchParams.set("function", "EARNINGS_CALENDAR");
|
||||
url.searchParams.set("horizon", horizon);
|
||||
url.searchParams.set("apikey", options.apiKey);
|
||||
if (options.symbol) {
|
||||
url.searchParams.set("symbol", normalizeUnderlying(options.symbol));
|
||||
}
|
||||
|
||||
const response = await (options.fetchFn ?? fetch)(url);
|
||||
const text = await response.text();
|
||||
if (!response.ok) {
|
||||
throw new Error(`Alpha Vantage earnings calendar request failed: ${response.status} ${text.slice(0, 160)}`);
|
||||
}
|
||||
if (/^(?:\s*\{|\s*Thank you for using Alpha Vantage)/i.test(text)) {
|
||||
throw new Error(`Alpha Vantage returned a non-calendar response: ${text.slice(0, 200)}`);
|
||||
}
|
||||
|
||||
return parseAlphaVantageEarningsCalendar(text, options.nowTs ?? Date.now());
|
||||
};
|
||||
|
||||
export const writeEventCalendarEntries = async (path: string, entries: EventCalendarEntry[]): Promise<void> => {
|
||||
const directory = path.includes("/") ? path.slice(0, path.lastIndexOf("/")) : "";
|
||||
if (directory) {
|
||||
await mkdir(directory, { recursive: true });
|
||||
}
|
||||
const file = Bun.file(path);
|
||||
await Bun.write(file, `${JSON.stringify(entries, null, 2)}\n`);
|
||||
};
|
||||
|
|
@ -1,10 +1,98 @@
|
|||
import { createLogger } from "@islandflow/observability";
|
||||
import {
|
||||
createEmptyEventCalendarProvider,
|
||||
fetchAlphaVantageEarningsCalendar,
|
||||
loadEventCalendarProviderFromFile,
|
||||
writeEventCalendarEntries,
|
||||
type AlphaVantageEarningsCalendarOptions
|
||||
} from "./event-calendar";
|
||||
|
||||
const service = "refdata";
|
||||
const logger = createLogger({ service });
|
||||
|
||||
logger.info("service starting");
|
||||
|
||||
const eventCalendarPath = process.env.REFDATA_EVENT_CALENDAR_PATH ?? process.env.SMART_MONEY_EVENT_CALENDAR_PATH;
|
||||
const eventCalendarProvider = process.env.REFDATA_EVENT_CALENDAR_PROVIDER ?? process.env.EVENT_CALENDAR_PROVIDER;
|
||||
const refreshMs = Math.max(0, Number(process.env.REFDATA_EVENT_CALENDAR_REFRESH_MS ?? 86_400_000) || 0);
|
||||
|
||||
const getAlphaVantageOptions = (): AlphaVantageEarningsCalendarOptions | null => {
|
||||
const apiKey = process.env.ALPHA_VANTAGE_API_KEY;
|
||||
if (!apiKey) {
|
||||
logger.warn("alpha vantage event calendar disabled; missing ALPHA_VANTAGE_API_KEY");
|
||||
return null;
|
||||
}
|
||||
|
||||
const horizon = process.env.ALPHA_VANTAGE_EARNINGS_HORIZON;
|
||||
return {
|
||||
apiKey,
|
||||
horizon: horizon === "6month" || horizon === "12month" ? horizon : "3month",
|
||||
symbol: process.env.ALPHA_VANTAGE_EARNINGS_SYMBOL || undefined
|
||||
};
|
||||
};
|
||||
|
||||
const refreshEventCalendar = async (): Promise<void> => {
|
||||
if (!eventCalendarPath) {
|
||||
logger.warn("event calendar refresh disabled; missing SMART_MONEY_EVENT_CALENDAR_PATH or REFDATA_EVENT_CALENDAR_PATH");
|
||||
return;
|
||||
}
|
||||
if (eventCalendarProvider !== "alpha_vantage") {
|
||||
return;
|
||||
}
|
||||
|
||||
const options = getAlphaVantageOptions();
|
||||
if (!options) {
|
||||
return;
|
||||
}
|
||||
|
||||
const entries = await fetchAlphaVantageEarningsCalendar(options);
|
||||
await writeEventCalendarEntries(eventCalendarPath, entries);
|
||||
logger.info("event calendar refreshed", {
|
||||
provider: "alpha_vantage",
|
||||
path: eventCalendarPath,
|
||||
count: entries.length,
|
||||
horizon: options.horizon,
|
||||
symbol: options.symbol ?? "ALL"
|
||||
});
|
||||
};
|
||||
|
||||
if (eventCalendarProvider === "alpha_vantage") {
|
||||
try {
|
||||
await refreshEventCalendar();
|
||||
} catch (error) {
|
||||
logger.warn("event calendar refresh failed", {
|
||||
provider: "alpha_vantage",
|
||||
error: error instanceof Error ? error.message : String(error)
|
||||
});
|
||||
}
|
||||
|
||||
if (refreshMs > 0) {
|
||||
setInterval(() => {
|
||||
refreshEventCalendar().catch((error) => {
|
||||
logger.warn("event calendar refresh failed", {
|
||||
provider: "alpha_vantage",
|
||||
error: error instanceof Error ? error.message : String(error)
|
||||
});
|
||||
});
|
||||
}, refreshMs);
|
||||
}
|
||||
}
|
||||
|
||||
if (eventCalendarPath) {
|
||||
try {
|
||||
await loadEventCalendarProviderFromFile(eventCalendarPath);
|
||||
logger.info("event calendar loaded", { path: eventCalendarPath });
|
||||
} catch (error) {
|
||||
logger.warn("event calendar unavailable", {
|
||||
path: eventCalendarPath,
|
||||
error: error instanceof Error ? error.message : String(error)
|
||||
});
|
||||
}
|
||||
} else {
|
||||
createEmptyEventCalendarProvider();
|
||||
logger.info("event calendar disabled");
|
||||
}
|
||||
|
||||
const shutdown = (signal: string) => {
|
||||
logger.info("service stopping", { signal });
|
||||
process.exit(0);
|
||||
|
|
|
|||
55
services/refdata/tests/event-calendar.test.ts
Normal file
55
services/refdata/tests/event-calendar.test.ts
Normal file
|
|
@ -0,0 +1,55 @@
|
|||
import { describe, expect, it } from "bun:test";
|
||||
import {
|
||||
createStaticEventCalendarProvider,
|
||||
parseAlphaVantageEarningsCalendar,
|
||||
parseEventCalendarEntries
|
||||
} from "../src/event-calendar";
|
||||
|
||||
describe("event calendar refdata", () => {
|
||||
it("parses provider rows and filters by timestamp availability", () => {
|
||||
const entries = parseEventCalendarEntries([
|
||||
{
|
||||
symbol: "aapl",
|
||||
event_date: "2025-01-31T21:00:00Z",
|
||||
event_kind: "earnings",
|
||||
announced_ts: "2025-01-20T21:00:00Z",
|
||||
source: "fixture"
|
||||
},
|
||||
{
|
||||
symbol: "AAPL",
|
||||
event_date: "2025-02-28T21:00:00Z",
|
||||
type: "mystery",
|
||||
announced_ts: "2025-02-01T21:00:00Z"
|
||||
}
|
||||
]);
|
||||
|
||||
const provider = createStaticEventCalendarProvider(entries);
|
||||
const beforeAnnouncement = provider.findNextEvent("AAPL", Date.parse("2025-01-15T15:00:00Z"));
|
||||
const afterAnnouncement = provider.findNextEvent("aapl", Date.parse("2025-01-21T15:00:00Z"));
|
||||
|
||||
expect(beforeAnnouncement).toBeNull();
|
||||
expect(afterAnnouncement?.event_kind).toBe("earnings");
|
||||
expect(afterAnnouncement?.underlying_id).toBe("AAPL");
|
||||
expect(afterAnnouncement?.days_to_event).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it("normalizes Alpha Vantage earnings CSV rows", () => {
|
||||
const entries = parseAlphaVantageEarningsCalendar(
|
||||
[
|
||||
"symbol,name,reportDate,fiscalDateEnding,estimate,currency",
|
||||
"aapl,Apple Inc,2025-01-31,2024-12-31,2.11,USD",
|
||||
"MSFT,Microsoft Corp,2025-02-05,2024-12-31,3.04,USD"
|
||||
].join("\n"),
|
||||
Date.parse("2025-01-15T12:00:00Z")
|
||||
);
|
||||
|
||||
expect(entries).toHaveLength(2);
|
||||
expect(entries[0]).toMatchObject({
|
||||
underlying_id: "AAPL",
|
||||
event_kind: "earnings",
|
||||
announced_ts: Date.parse("2025-01-15T12:00:00Z"),
|
||||
source: "alpha_vantage",
|
||||
source_event_id: "AAPL:2025-01-31:earnings"
|
||||
});
|
||||
});
|
||||
});
|
||||
534
smartmoney.md
Normal file
534
smartmoney.md
Normal file
|
|
@ -0,0 +1,534 @@
|
|||
# Smart Money Options Flow Classifier Playbook
|
||||
|
||||
## Executive Summary
|
||||
|
||||
A usable options-flow classifier should start from one hard truth: there is no single “smart money” footprint. The same aggressive print can come from an informed institutional buyer, a dealer hedging inventory, a retail stampede, a facilitation auction, or a parity trade embedded in a complex order. The public tape is informative, but it is noisy, and the literature is mixed: some studies find that options volume and order imbalance predict future stock returns and volatility, while other work shows that a meaningful share of apparent pre-event options activity is speculative and retail-driven rather than informed.
|
||||
|
||||
The practical implication is that you should not train one monolithic `smart_money = 1` label. Train a taxonomy. First reconstruct parent events from child prints, quotes, condition codes, venue information, and Greeks. Then classify each parent event into one or more participant-style hypotheses such as directional institutional block buy, dealer hedge, professional/prop burst, retail whale chase, event-driven informed flow, volatility seller, or arbitrage desk. That hierarchy matches both the official options market structure and the academic evidence on demand pressure, volatility-information trading, dealer hedging, retail demand, and event-time option selection.
|
||||
|
||||
Your best baseline is an ensemble. Use rules to produce interpretable weak labels, supervised models to learn non-linear interactions, and unsupervised anomaly detection to catch new regimes. Keep the final output probabilistic and multi-label, not categorical and overconfident. That is the only sane way to handle a market where options sometimes lead stocks, but options are also noisier than equities because of wider spreads, temporary price pressure, legging, and event-driven speculation.
|
||||
|
||||
## Market Structure and Data Foundation
|
||||
|
||||
Start with public U.S. listed-options data from Options Price Reporting Authority. The OPRA specification gives you participant ID, last-sale message types, quote messages, best-bid/best-offer appendages, and end-of-day open interest. Participant IDs identify the exchange that originated the message, and OPRA quote appendages identify the exchange posting the best bid or offer. OPRA also carries important last-sale condition detail including ISO executions, auctions, crosses, multi-leg complex trades, stock-option trades, compression trades, late prints, cancels, and out-of-sequence messages. Those fields are the backbone of any classifier worth building.
|
||||
|
||||
Venue and structure matter because the rules explicitly allow complex and special-order handling to look different from plain single-leg urgency. The options order protection plan defines an ISO as a limit order routed together with additional ISOs to satisfy better-priced protected quotations, and it treats complex trades as a specific trade-through exception. Exchange rulebooks also define complex-order books, complex-order auctions, synthetic BBOs for strategies, and legging into simple books. In plain English: a print that looks “too aggressive” versus the simple-leg NBBO may be perfectly normal for a complex strategy or auction.
|
||||
|
||||
Use official strategy definitions from The Options Clearing Corporation to anchor the arbitrage and overwrite classes. OCC materials and related exchange methodology documents give canonical descriptions of covered-call or buy-write structures, put/call parity, conversions, reversals, and box spreads. Those are not trivia; they let you design deterministic detectors for some of the cleanest non-directional “smart money” profiles on the tape.
|
||||
|
||||
Routing data is the next layer. Public routing disclosure under U.S. Securities and Exchange Commission Rule 606 and the FINRA 606 portal can tell you where non-directed listed-options orders are routed and whether payment for order flow or other venue economics may be shaping execution. That is not a per-trade participant flag, but it becomes useful as a prior when you have broker-level logs, customer requests, or controlled execution datasets.
|
||||
|
||||
For quote alignment, use the latest valid NBBO snapshot at or before the trade timestamp, but maintain a correction pipeline because OPRA explicitly documents late trades, out-of-sequence trades, cancels, and sequence resets. For volatility features, derive IV, delta, gamma, and vega from a surface built from contemporaneous quotes; for variance-risk features, use an option-strip estimate of risk-neutral variance and compare it with subsequent realized variance. That is the cleanest way to separate directional demand from pure vol-selling or vol-buying pressure.
|
||||
|
||||
### Data sources and what each is good for
|
||||
|
||||
| Source | Core fields | Best use in classifier | Main limitations |
|
||||
|---|---|---|---|
|
||||
| OPRA time & sales / tape | last sale, condition code, exchange participant ID, contract identifiers | trade reconstruction, urgency, venue, complex/auction/special print filters | no beneficial owner, no true account class |
|
||||
| OPRA quotes / NBBO | bid, ask, sizes, best-bid/best-offer appendages | aggressor-side inference, spread position, quote pressure | quote-trade desync, temporary noise |
|
||||
| Vendor-normalized NBBO & trades such as products from Nasdaq | nanosecond timestamps, OPRA-derived NBBO/trade fields, appendages | production-grade replay, lower engineering friction | still usually lacks owner identity |
|
||||
| Exchange rulebooks and venue specs such as Cboe Options Exchange materials | complex-order logic, auction mechanics, strategy definitions | false-positive mitigation for crosses, auctions, legging | descriptive, not participant labels |
|
||||
| Open interest | end-of-day OI by contract | weak confirmation of opening flow for training and backtests | not real-time |
|
||||
| Broker routing / Rule 606 | venue distribution, non-directed-routing stats, PFOF economics | priors on retail/wholesaler routing, venue fingerprints | not per-trade in public reports |
|
||||
| IV surface and underlying prices | IV, skew, term structure, delta/gamma/vega, realized vol | participant-style separation, especially vol sellers, hedgers, arbs | model choice matters |
|
||||
| Event calendars | earnings, M&A, dividends, corporate actions | event-driven informed-flow labeling | external datasets required |
|
||||
| Broker/account/CAT-like audit data if available | account type, origin, open/close, route chain | strongest labels for retail vs professional vs institutional | usually not publicly available |
|
||||
|
||||
Source basis for the table: OPRA field definitions and condition codes, vendor OPRA-derived feeds, exchange complex-order rules, OCC strategy definitions, SEC Rule 606, and FINRA’s 606 reporting portal.
|
||||
|
||||
The core research takeaway is that the tape can contain real information. Easley, O’Hara, and Srinivas show that signed positive and negative option volume contains information about future stock prices; Pan and Poteshman show that open-buy put/call ratios predict subsequent stock returns; Ni, Pan, and Poteshman show that non-market-maker net demand for volatility predicts future realized volatility beyond implied volatility; and later price-discovery work finds that options reflect new information before stocks roughly one-quarter of the time on average, especially around information events. But equally important, other work shows strongly mixed evidence, including papers arguing that much earnings-related options activity is dominated by speculative retail trading and differences of opinion rather than pure information. Your classifier must be built around that ambiguity, not around internet folklore.
|
||||
|
||||
## Taxonomy of Smart Money Profiles
|
||||
|
||||
The table below is intentionally pragmatic. Where the market does not provide a canonical cutoff, the threshold is marked **unspecified** and I add a **seed** value in parentheses that is meant only as a starting hyperparameter. Tune every seed by symbol liquidity bucket, option price level, spread regime, and event context.
|
||||
|
||||
| Profile | Economic motive | Strongest tape signature | Highest-value measurable features | Suggested thresholds or ranges |
|
||||
|---|---|---|---|---|
|
||||
| Institutional block buyers | Directional or convexity exposure around a thesis or catalyst | Large parent order, mostly aggressive, concentrated in one strike or a tight strike cluster, expiration usually aligned with a catalyst horizon | ask-lift share, spread position, parent notional, strike concentration, DTE, absolute delta, next-day OI change, IV percentile | ask-lift share **unspecified** (seed `> 0.60`); parent notional **unspecified** (seed `>$250k` single names, `>$1m` indexes); same-strike notional share **unspecified** (seed `>0.70`) |
|
||||
| Market makers hedging | Inventory and gamma risk management | Activity is most visible as reactive cross-asset flow, especially in short-dated ATM contracts and the underlying/futures, often reversing with price changes | DTE, ATM proximity, dollar gamma, hedge-link to stock/futures, intraday sign reversals, two-sided prints, quote widening | DTE `0–2` days for strongest signatures; abs(delta) **unspecified** (seed `0.35–0.65`); high dollar gamma **unspecified** (seed `>95th percentile by symbol/DTE`) |
|
||||
| Prop firms / professional customers | Intraday alpha, microstructure taking, liquidity seeking, statistical edge | Rapid child-order bursts across venues or strikes, ISO/sweep-like urgency, low dwell time, often many small or medium clips rather than one giant block | inter-fill milliseconds, venue count, ISO flag, distinct strikes in burst, burst entropy, lot-size dispersion, routing pattern | official “professional customer” threshold is `>390 orders/day` if origin data exists; public burst proxy **unspecified** (seed `>=5` child prints in `<=2s`, `>=2` venues) |
|
||||
| Retail whales | Leveraged speculation in attention-heavy names | Large prints by retail standards, short-dated and often OTM, heavily call-biased in favored names, rising IV, often occurs in the same contracts retail prefers generally | DTE, moneyness, call/put bias, IV shock, venue prior from routing data, concentration in high-attention symbols | DTE **unspecified** (seed `<=7` for single names, often `0DTE/1DTE` in indexes); abs(delta) **unspecified** (seed `0.10–0.35`); notional threshold is account-dependent and thus **unspecified** |
|
||||
| Corporate-event informed flow | Exploit private or superior information about timing and direction of a known upcoming event; do **not** equate this with illegal insider trading | Expiration chosen to land just after the event, high leverage via OTM or near-ATM contracts, unusual pre-event volume, IV and spreads often rise before announcement | event-distance days, expiry alignment, moneyness, spread widening, IV term-structure change, OI growth, low-priced leverage preference | event window **unspecified** (seed `1–30d` before event); expiry alignment **unspecified** (seed “first listed expiry after event”); abs(delta) **unspecified** (seed `0.15–0.40` for directional calls/puts; `0.40–0.60` for straddles/strangles) |
|
||||
| Volatility sellers | Harvest premium, overwrite stock, or short rich implied volatility | Prints are often on the sell side near bid or midpoint, repeated rolled positions, multi-leg short-vol structures, or covered-call / buy-write linkage to stock | signed vega, IV-minus-HV, realized-vs-implied variance spread, roll cadence, covered-call stock ratio, multi-leg flags | sell-side dominance **unspecified** (seed `>0.60` of parent contracts); IV-RV richness **unspecified** (seed z-score `>1.5`); covered-call stock/contract ratio **unspecified** (seed `80–120` shares per contract equivalent) |
|
||||
| Arbitrage desks | Enforce parity, finance/carry trades, or exploit mispricings | Conversions, reversals, boxes, jelly-roll-like structures, same-size matched legs, near-zero net delta, often same expiry/paired strikes, may appear as complex trades | parity residual, matched-leg timing, same-size legs, net delta near zero, net vega near zero, complex flag, European/cash-settled box eligibility | abs(net delta) **unspecified** (seed `<0.05` of equivalent shares after scaling); parity residual **unspecified** (seed `> fees + slippage`); same-size matched legs `exact or within 5%` |
|
||||
|
||||
Evidence anchors for the taxonomy: signed option volume predicts future stock prices, volatility demand predicts future realized volatility, dealer hedging changes spreads and underlier trading, retail demand clusters in short-dated OTM calls and affects IV, informed traders time maturities around earnings/news, professional-customer status begins above 390 orders per day, and arbitrage/overwrite structures are formally defined by OCC and exchange methodology.
|
||||
|
||||
Two profiles deserve special caution. First, market-maker hedging is often better observed in the underlying than in the options tape itself; a dealer can be the passive counterparty in options and the aggressive actor in stock or futures because of delta rebalancing, especially in high-gamma short-dated regimes. Second, “corporate-event informed flow” should be treated as a market-behavior label, not as an accusation of illegal insider trading. The academic and regulatory evidence shows suspicious pre-event patterns can exist, but the public tape is not enough to prove intent or legal status.
|
||||
|
||||
## Feature Engineering and Weak Labeling
|
||||
|
||||
The right unit of analysis is almost never the raw print. It is the reconstructed parent event. Sessionize child prints by contract, side, and time gap; align them to the most recent valid quote; compute whether the parent traded at the ask, at the bid, or through a special mechanism; then aggregate size, notional, strike dispersion, expiry alignment, venue footprint, and Greeks. If you skip parent reconstruction, your classifier will overfit to child-print fragmentation and venue noise.
|
||||
|
||||
### Feature library
|
||||
|
||||
| Feature | How to compute | Data required | Suggested threshold / range |
|
||||
|---|---|---|---|
|
||||
| `order_side_score` | classify child print as buy if `price >= ask - eps`, sell if `price <= bid + eps`, else midpoint/unknown; aggregate parent as ask-lift share or bid-hit share | trades + contemporaneous NBBO | `eps` **unspecified** (seed `0.01` or `0.1 * spread`); buy aggression **unspecified** (seed ask-lift share `>0.60`) |
|
||||
| `spread_position` | `(price - bid) / max(ask - bid, tick)` clipped to `[0,1]` | trades + NBBO | buy-like `>=0.80`, sell-like `<=0.20` |
|
||||
| `inter_fill_ms` | median and max milliseconds between child prints in same parent | trades | urgent burst **unspecified** (seed median `<=500ms`); sweep-like **unspecified** (seed child gap `<=50ms`) |
|
||||
| `parent_notional_usd` | `sum(size * contract_multiplier * trade_price)` over parent | trades + contract multiplier | **unspecified** (seed rank `>=99th pct` by symbol; or absolute `>$250k` single names / `>$1m` indexes) |
|
||||
| `strike_concentration` | largest strike notional share within parent or same-day cluster | trades | **unspecified** (seed `>0.70`) |
|
||||
| `maturity_alignment` | days from trade date to expiry; also distance from expiry to event date | contract metadata + event calendar | directional event flow often `expiry just after event`; hedge flow strongest at `0DTE/1DTE/2DTE`; exact cutoff **unspecified** |
|
||||
| `abs_delta` and `dollar_gamma` | compute from IV surface and spot; scale gamma by contracts and spot | quotes + underlying + surface | event-driven directional flow often abs(delta) **unspecified** (seed `0.15–0.40`); hedge-sensitive flow often `0.35–0.65` |
|
||||
| `iv_minus_hv` / `vrp_signal` | compare contemporaneous IV or synthetic risk-neutral variance with trailing HV or future RV | quotes + underlying history | **unspecified** (seed z-score `>1.5` for “rich IV”, `<-1.5` for “cheap IV`) |
|
||||
| `complex_flag` | true if OPRA condition indicates multi-leg, cross, auction, stock-option, or compression trade | OPRA condition codes | exact by condition code; no threshold |
|
||||
| `venue_count` and `venue_entropy` | count distinct exchanges in parent burst and entropy of prints by exchange | participant ID / exchange | **unspecified** (seed `>=2` venues in `<=1s` = urgency prior) |
|
||||
| `iso_or_sweep_flag` | true if OPRA ISO condition present or if multi-venue ask-lifting occurs in one burst | trade conditions + participant ID + NBBO | ISO is deterministic when flagged; burst-sweep proxy **unspecified** |
|
||||
| `routing_prior` | broker-level probability vector from Rule 606, broker execution logs, or account-specific data | Rule 606 / broker logs | public per-trade threshold is **unspecified** |
|
||||
| `oi_confirmation` | `next_day_OI - prior_day_OI`, optionally scaled by burst size | open interest + trades | **unspecified** (seed `OI delta > 0` or `>=25%` of parent size) |
|
||||
| `underlying_link` | stock/futures buy-sell imbalance in a short window around option parent; for buy-write detect stock buy near call sale | signed stock/futures trades + options | **unspecified** (seed `±5s` window; share/contract ratio `80–120`) |
|
||||
|
||||
Source basis for the feature library: OPRA quote/trade fields, OPRA condition codes, open interest, the options order protection plan, complex-order rules, variance-risk-premium construction from option prices, and academic evidence linking directional, volatility, retail, and event-time flow to specific contracts and maturities.
|
||||
|
||||
### Weak-label seeds for training
|
||||
|
||||
| Profile | Positive seed label | Hard exclusions / downweights |
|
||||
|---|---|---|
|
||||
| Institutional block buyer | large parent notional, ask-lift dominant, concentrated in one strike or narrow cluster, not tagged complex/auction, expiry consistent with thesis horizon, next-day OI rises | complex/auction/cross flags, parity-like matched opposite legs, obvious covered-call stock link |
|
||||
| Market-maker hedge | high dollar gamma in `0DTE–2DTE` near ATM, option parent followed by opposite-direction stock/futures hedge, repeated intraday sign flips, two-sided inventory management | single giant concentrated directional bet with no underlier hedge |
|
||||
| Prop / professional | many child prints fast, multiple venues, ISO or sweep-like urgency, multiple strikes or expiries, high daily order count if origin exists | one slow resting limit order, one-venue block, obvious overwrite/arbitrage |
|
||||
| Retail whale | short-dated OTM call-heavy flow in retail-favored symbol, IV shock, broker/venue prior consistent with retail routing if available | complex parity structures, low-delta institutional put hedges, calm overwrite roll |
|
||||
| Corporate-event informed flow | event within next `1–30d`, expiry just after event, unusual OTM directional exposure or ATM vol exposure, rising IV/spreads, OI expansion | contracts far beyond event horizon, obvious retail-meme chase, special-order cross conditions |
|
||||
| Vol seller | sell-side dominant, short vega, repeated monthly roll or overwrite pattern, IV rich to HV/RV, stock buy link for covered call | strong ask-lifting call/put buys, long-vol straddles, event-aligned convexity buys |
|
||||
| Arbitrage desk | same-size opposite legs, same expiry and parity-linked strikes, near-zero net delta, complex flag or matched-leg timestamps | highly concentrated one-way exposure with large residual delta |
|
||||
|
||||
These labels are intentionally “silver,” not “gold.” They are for weak supervision, self-training, and human review queues. Public OPRA-style data does not identify beneficial owner, open/close intent in real time, or customer/professional/institutional status directly, so hard participant labels require richer private data.
|
||||
|
||||
## Classifier Design and Evaluation
|
||||
|
||||
A strong design is a three-layer ensemble. Layer one is an interpretable rule engine that reconstructs parents, filters special prints, and emits weak labels plus reason codes. Layer two is a supervised event-level model, usually gradient-boosted trees as the baseline and a sequence model only if you truly need temporal microstructure context. Layer three is an unsupervised anomaly detector by symbol and regime to catch novel bursts that the rules and labels miss. Calibrate the final probabilities so downstream systems can set risk-sensitive thresholds instead of blindly trusting raw scores. That structure matches the mixed research evidence and the market’s obvious non-stationarity.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A[Raw options trades and quotes] --> B[Quote alignment and correction handling]
|
||||
B --> C[Parent-order reconstruction]
|
||||
C --> D[Feature engineering]
|
||||
D --> E[Rule engine and weak labels]
|
||||
D --> F[Supervised event model]
|
||||
D --> G[Unsupervised anomaly model]
|
||||
E --> H[Ensemble and calibration]
|
||||
F --> H
|
||||
G --> H
|
||||
H --> I[Profile probabilities plus reason codes]
|
||||
I --> J[Real-time alerts, batch review, backtests]
|
||||
```
|
||||
|
||||
Evaluation should happen at the parent-event level, not the child-print level. Use macro and micro F1, precision, recall, AUROC, AUPRC, Matthews correlation, Brier score, and expected calibration error. Then add profile-specific economic validation. For directional institutional or event-driven flow, test post-signal stock return, IV change, and spread-adjusted PnL. For vol sellers, test realized-versus-implied variance, theta capture, and post-event IV compression. For arbitrage, test parity convergence net of fees and slippage. For market-maker hedges, test whether predicted hedge-linked flow lines up with same-session stock or futures rebalancing.
|
||||
|
||||
Backtests should be walk-forward and purged. Split by time first, then by symbol clusters or sectors if you can, and embargo around the same catalyst so nearly identical event windows do not leak into train and test. Use only information available at the event timestamp in the live feature set. End-of-day open interest can validate labels during offline training, but it must never leak into real-time scoring. Re-run batch labels after cancels, late reports, and sequence repairs. That last step is not optional; the OPRA spec explicitly says those records exist.
|
||||
|
||||
### Common false positives and how to kill them
|
||||
|
||||
| False positive | Why it happens | Mitigation |
|
||||
|---|---|---|
|
||||
| Aggressive buy at ask that is actually an auction or facilitation print | single-leg or complex auctions can print “aggressively” without informed urgency | downweight or exclude auction/cross/complex condition codes before directional classification |
|
||||
| Print above simple-leg NBBO that is actually a complex trade | complex trades have protection exceptions and can leg or price off strategy economics | require `complex_flag = false` for simple directional labels |
|
||||
| Retail frenzy mistaken for institutional conviction | retail demand is heavily concentrated in short-dated OTM calls and can move IV | layer in retail priors, attention proxies, and avoid treating short-dated OTM call bursts as automatically informed |
|
||||
| Dealer re-hedging mistaken for end-user direction | dealer inventory management can create reactive stock/futures flow and two-sided options activity | use stock/futures linkage and estimated dollar gamma |
|
||||
| Parity trades mistaken for “smart bullish calls” | conversions/reversals/boxes include calls, puts, and sometimes stock in matched quantities | net Greeks and parity residual checks |
|
||||
| Late, canceled, or out-of-sequence prints | tape corrections can invert apparent urgency | correction-aware replay and batch relabeling |
|
||||
| Wide-spread illiquid contracts | midpoint and ask/bid inference is unreliable when spreads are huge | liquidity filters, spread normalization, larger `eps`, contract-level confidence score |
|
||||
| Pre-earnings speculation mistaken for information | some literature finds earnings-related options activity is mainly speculative and retail-driven | use event alignment plus cross-checks: pre-event stock returns, spreads, OI, and profile probabilities rather than raw volume alone |
|
||||
|
||||
This table is built directly from OPRA condition rules, exchange complex-order mechanics, retail-flow evidence, dealer-hedging evidence, and the mixed academic results on information versus speculation in options.
|
||||
|
||||
## Implementation and Detection Examples
|
||||
|
||||
For production, keep three stores: raw append-only packet or normalized message storage; a corrected trade-and-quote warehouse; and an event-level feature store keyed by parent ID. Real time should score on the best-available aligned quote state and current IV surface, while batch should replay the day after corrections and open interest land. Storage should be columnar for history and ring-buffered in memory for current-day scoring. Latency matters mostly for parent reconstruction and hedge linkage; model inference itself is cheap compared with quote alignment and surface updates.
|
||||
|
||||
```mermaid
|
||||
timeline
|
||||
title Sample trade sequence for one inferred parent event
|
||||
09:31:02.100: NBBO 1.00 x 1.05
|
||||
09:31:02.120: 1,500 calls print at 1.05 on one venue
|
||||
09:31:02.310: 2,000 calls print at 1.05 on second venue
|
||||
09:31:02.360: 1,200 calls print at 1.06 after ask lifts
|
||||
09:31:02.900: stock/futures buy program starts
|
||||
09:31:03.400: implied vol up and spread widens
|
||||
09:31:04.000: parent burst closes and event is scored
|
||||
```
|
||||
|
||||
### Example pseudocode
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass, field
|
||||
from typing import List, Dict, Optional
|
||||
import math
|
||||
|
||||
@dataclass
|
||||
class ChildTrade:
|
||||
ts_ns: int
|
||||
underlying: str
|
||||
expiry: str
|
||||
strike: float
|
||||
cp: str
|
||||
price: float
|
||||
contracts: int
|
||||
exchange: str
|
||||
cond: str
|
||||
bid: float
|
||||
ask: float
|
||||
spot: float
|
||||
iv: Optional[float] = None
|
||||
delta: Optional[float] = None
|
||||
gamma: Optional[float] = None
|
||||
|
||||
@dataclass
|
||||
class ParentEvent:
|
||||
children: List[ChildTrade] = field(default_factory=list)
|
||||
|
||||
def add(self, t: ChildTrade) -> None:
|
||||
self.children.append(t)
|
||||
|
||||
def features(self) -> Dict[str, float]:
|
||||
if not self.children:
|
||||
return {}
|
||||
|
||||
prices = [c.price for c in self.children]
|
||||
sizes = [c.contracts for c in self.children]
|
||||
notionals = [c.price * c.contracts * 100.0 for c in self.children]
|
||||
spreads = [max(c.ask - c.bid, 0.01) for c in self.children]
|
||||
|
||||
ask_lifts = [
|
||||
1.0 if c.price >= c.ask - min(0.01, 0.1 * (c.ask - c.bid)) else 0.0
|
||||
for c in self.children
|
||||
]
|
||||
bid_hits = [
|
||||
1.0 if c.price <= c.bid + min(0.01, 0.1 * (c.ask - c.bid)) else 0.0
|
||||
for c in self.children
|
||||
]
|
||||
spread_pos = [
|
||||
max(0.0, min(1.0, (c.price - c.bid) / s))
|
||||
for c, s in zip(self.children, spreads)
|
||||
]
|
||||
|
||||
ts = sorted(c.ts_ns for c in self.children)
|
||||
gaps_ms = [(ts[i] - ts[i - 1]) / 1e6 for i in range(1, len(ts))]
|
||||
gap_med_ms = sorted(gaps_ms)[len(gaps_ms) // 2] if gaps_ms else 0.0
|
||||
|
||||
strike_notional: Dict[tuple, float] = {}
|
||||
venue_set = set()
|
||||
complex_flag = 0
|
||||
iso_flag = 0
|
||||
|
||||
gamma_dollar = 0.0
|
||||
delta_equiv_shares = 0.0
|
||||
|
||||
for c, n in zip(self.children, notionals):
|
||||
strike_notional[(c.expiry, c.strike, c.cp)] = strike_notional.get((c.expiry, c.strike, c.cp), 0.0) + n
|
||||
venue_set.add(c.exchange)
|
||||
if c.cond in set("abcdefghijklmnopqrstuvwxyz"):
|
||||
complex_flag = 1
|
||||
if c.cond == "S":
|
||||
iso_flag = 1
|
||||
if c.gamma is not None:
|
||||
gamma_dollar += c.gamma * c.contracts * 100.0 * (c.spot ** 2) * 0.01
|
||||
if c.delta is not None:
|
||||
delta_equiv_shares += c.delta * c.contracts * 100.0
|
||||
|
||||
top_cluster_share = max(strike_notional.values()) / max(sum(notionals), 1.0)
|
||||
|
||||
return {
|
||||
"contracts_total": float(sum(sizes)),
|
||||
"notional_total_usd": float(sum(notionals)),
|
||||
"avg_price": sum(prices) / len(prices),
|
||||
"ask_lift_share": sum(ask_lifts) / len(ask_lifts),
|
||||
"bid_hit_share": sum(bid_hits) / len(bid_hits),
|
||||
"spread_pos_mean": sum(spread_pos) / len(spread_pos),
|
||||
"inter_fill_median_ms": gap_med_ms,
|
||||
"top_strike_cluster_share": top_cluster_share,
|
||||
"venue_count": float(len(venue_set)),
|
||||
"complex_flag": float(complex_flag),
|
||||
"iso_flag": float(iso_flag),
|
||||
"gamma_dollar_per_1pct_move": gamma_dollar,
|
||||
"delta_equiv_shares": delta_equiv_shares,
|
||||
}
|
||||
|
||||
def same_parent(prev: ChildTrade, cur: ChildTrade, max_gap_ms: int = 2000) -> bool:
|
||||
if (cur.ts_ns - prev.ts_ns) / 1e6 > max_gap_ms:
|
||||
return False
|
||||
return (
|
||||
prev.underlying == cur.underlying
|
||||
and prev.cp == cur.cp
|
||||
and prev.expiry == cur.expiry
|
||||
and abs(prev.strike - cur.strike) < 1e-9
|
||||
)
|
||||
|
||||
def score_profile(x: Dict[str, float]) -> Dict[str, float]:
|
||||
# Interpretable weak-score layer. Replace with calibrated model outputs later.
|
||||
scores = {
|
||||
"institutional_block_buy": 0.0,
|
||||
"market_maker_hedge": 0.0,
|
||||
"prop_professional": 0.0,
|
||||
"retail_whale": 0.0,
|
||||
"corporate_event_informed": 0.0,
|
||||
"vol_seller": 0.0,
|
||||
"arbitrage_desk": 0.0,
|
||||
}
|
||||
|
||||
if x["complex_flag"] == 0 and x["ask_lift_share"] > 0.60 and x["top_strike_cluster_share"] > 0.70:
|
||||
scores["institutional_block_buy"] += 0.6
|
||||
if x["inter_fill_median_ms"] <= 500 and x["venue_count"] >= 2:
|
||||
scores["prop_professional"] += 0.5
|
||||
if x["iso_flag"] == 1:
|
||||
scores["prop_professional"] += 0.2
|
||||
if x["complex_flag"] == 1 and abs(x["delta_equiv_shares"]) < 0.05 * max(x["contracts_total"] * 100.0, 1.0):
|
||||
scores["arbitrage_desk"] += 0.5
|
||||
if x["bid_hit_share"] > 0.60:
|
||||
scores["vol_seller"] += 0.4
|
||||
|
||||
# Market-maker-hedge score benefits from separate stock/futures linkage features not shown here.
|
||||
return scores
|
||||
```
|
||||
|
||||
### SQL schema assumption
|
||||
|
||||
The SQL below assumes PostgreSQL and these normalized tables:
|
||||
|
||||
- `option_trades(ts, trade_id, underlying, expiry, strike, cp, price, contracts, exchange, cond)`
|
||||
- `option_nbbo(ts, underlying, expiry, strike, cp, bid, ask, bid_exch, ask_exch)`
|
||||
- `stock_trades_signed(ts, symbol, shares, price, side_est)` where `side_est` is `1` for aggressive buy and `-1` for aggressive sell
|
||||
|
||||
### SQL prework: enrich options prints with the latest NBBO
|
||||
|
||||
```sql
|
||||
CREATE MATERIALIZED VIEW enriched_option_trades AS
|
||||
SELECT
|
||||
t.*,
|
||||
q.bid,
|
||||
q.ask,
|
||||
q.bid_exch,
|
||||
q.ask_exch,
|
||||
CASE
|
||||
WHEN t.price >= q.ask - LEAST(0.01, 0.10 * GREATEST(q.ask - q.bid, 0.01)) THEN 1
|
||||
WHEN t.price <= q.bid + LEAST(0.01, 0.10 * GREATEST(q.ask - q.bid, 0.01)) THEN -1
|
||||
ELSE 0
|
||||
END AS side_est,
|
||||
CASE
|
||||
WHEN q.ask > q.bid THEN (t.price - q.bid) / (q.ask - q.bid)
|
||||
ELSE NULL
|
||||
END AS spread_pos,
|
||||
(t.price * t.contracts * 100.0) AS notional_usd,
|
||||
(t.cond = 'S')::int AS iso_flag
|
||||
FROM option_trades t
|
||||
JOIN LATERAL (
|
||||
SELECT q.*
|
||||
FROM option_nbbo q
|
||||
WHERE q.underlying = t.underlying
|
||||
AND q.expiry = t.expiry
|
||||
AND q.strike = t.strike
|
||||
AND q.cp = t.cp
|
||||
AND q.ts <= t.ts
|
||||
ORDER BY q.ts DESC
|
||||
LIMIT 1
|
||||
) q ON TRUE;
|
||||
```
|
||||
|
||||
### SQL query: buys at or above the ask within seconds
|
||||
|
||||
```sql
|
||||
WITH tagged AS (
|
||||
SELECT
|
||||
*,
|
||||
CASE
|
||||
WHEN LAG(ts) OVER w IS NULL THEN 1
|
||||
WHEN EXTRACT(EPOCH FROM (ts - LAG(ts) OVER w)) > 2 THEN 1
|
||||
ELSE 0
|
||||
END AS new_parent
|
||||
FROM enriched_option_trades
|
||||
WHERE side_est = 1
|
||||
AND cond NOT IN ('a','b','c','d','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v')
|
||||
WINDOW w AS (
|
||||
PARTITION BY underlying, expiry, strike, cp
|
||||
ORDER BY ts
|
||||
)
|
||||
),
|
||||
parents AS (
|
||||
SELECT
|
||||
*,
|
||||
SUM(new_parent) OVER (
|
||||
PARTITION BY underlying, expiry, strike, cp
|
||||
ORDER BY ts
|
||||
ROWS UNBOUNDED PRECEDING
|
||||
) AS parent_id
|
||||
FROM tagged
|
||||
)
|
||||
SELECT
|
||||
underlying, expiry, strike, cp, parent_id,
|
||||
MIN(ts) AS start_ts,
|
||||
MAX(ts) AS end_ts,
|
||||
SUM(contracts) AS contracts_total,
|
||||
SUM(notional_usd) AS notional_total_usd,
|
||||
AVG(spread_pos) AS mean_spread_pos,
|
||||
COUNT(*) AS child_prints
|
||||
FROM parents
|
||||
GROUP BY underlying, expiry, strike, cp, parent_id
|
||||
HAVING SUM(notional_usd) > 250000
|
||||
AND AVG(spread_pos) >= 0.80
|
||||
ORDER BY start_ts;
|
||||
```
|
||||
|
||||
### SQL query: large notional concentrated in a single strike
|
||||
|
||||
```sql
|
||||
WITH daily AS (
|
||||
SELECT
|
||||
DATE(ts) AS trade_date,
|
||||
underlying,
|
||||
expiry,
|
||||
strike,
|
||||
cp,
|
||||
SUM(notional_usd) AS strike_notional,
|
||||
SUM(SUM(notional_usd)) OVER (
|
||||
PARTITION BY DATE(ts), underlying
|
||||
) AS total_notional_underlying
|
||||
FROM enriched_option_trades
|
||||
WHERE side_est = 1
|
||||
GROUP BY DATE(ts), underlying, expiry, strike, cp
|
||||
)
|
||||
SELECT
|
||||
trade_date,
|
||||
underlying,
|
||||
expiry,
|
||||
strike,
|
||||
cp,
|
||||
strike_notional,
|
||||
total_notional_underlying,
|
||||
strike_notional / NULLIF(total_notional_underlying, 0) AS strike_share
|
||||
FROM daily
|
||||
WHERE strike_notional >= 250000
|
||||
AND strike_notional / NULLIF(total_notional_underlying, 0) >= 0.70
|
||||
ORDER BY trade_date, underlying, strike_share DESC;
|
||||
```
|
||||
|
||||
### SQL query: rapid repeated buys across strikes
|
||||
|
||||
```sql
|
||||
WITH bursts AS (
|
||||
SELECT
|
||||
*,
|
||||
CASE
|
||||
WHEN LAG(ts) OVER w IS NULL THEN 1
|
||||
WHEN EXTRACT(EPOCH FROM (ts - LAG(ts) OVER w)) > 2 THEN 1
|
||||
ELSE 0
|
||||
END AS new_burst
|
||||
FROM enriched_option_trades
|
||||
WHERE side_est = 1
|
||||
AND cond NOT IN ('a','b','c','d','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v')
|
||||
WINDOW w AS (
|
||||
PARTITION BY underlying, expiry, cp
|
||||
ORDER BY ts
|
||||
)
|
||||
),
|
||||
clustered AS (
|
||||
SELECT
|
||||
*,
|
||||
SUM(new_burst) OVER (
|
||||
PARTITION BY underlying, expiry, cp
|
||||
ORDER BY ts
|
||||
ROWS UNBOUNDED PRECEDING
|
||||
) AS burst_id
|
||||
FROM bursts
|
||||
)
|
||||
SELECT
|
||||
underlying,
|
||||
expiry,
|
||||
cp,
|
||||
burst_id,
|
||||
MIN(ts) AS start_ts,
|
||||
MAX(ts) AS end_ts,
|
||||
COUNT(*) AS child_prints,
|
||||
COUNT(DISTINCT strike) AS strikes_hit,
|
||||
COUNT(DISTINCT exchange) AS venues_hit,
|
||||
SUM(notional_usd) AS burst_notional_usd
|
||||
FROM clustered
|
||||
GROUP BY underlying, expiry, cp, burst_id
|
||||
HAVING COUNT(DISTINCT strike) >= 3
|
||||
AND COUNT(DISTINCT exchange) >= 2
|
||||
AND SUM(notional_usd) > 250000
|
||||
ORDER BY start_ts;
|
||||
```
|
||||
|
||||
### SQL query: sweeps
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
underlying,
|
||||
expiry,
|
||||
strike,
|
||||
cp,
|
||||
MIN(ts) AS start_ts,
|
||||
MAX(ts) AS end_ts,
|
||||
COUNT(*) AS child_prints,
|
||||
COUNT(DISTINCT exchange) AS venues_hit,
|
||||
SUM(notional_usd) AS notional_total_usd,
|
||||
MAX(iso_flag) AS has_iso_flag
|
||||
FROM enriched_option_trades
|
||||
WHERE side_est = 1
|
||||
GROUP BY underlying, expiry, strike, cp, DATE_TRUNC('second', ts)
|
||||
HAVING MAX(iso_flag) = 1
|
||||
OR (
|
||||
COUNT(DISTINCT exchange) >= 2
|
||||
AND COUNT(*) >= 3
|
||||
AND SUM(notional_usd) > 150000
|
||||
)
|
||||
ORDER BY start_ts;
|
||||
```
|
||||
|
||||
### SQL query: probabilistic buy-write / covered-call indicator
|
||||
|
||||
```sql
|
||||
WITH option_sales AS (
|
||||
SELECT
|
||||
ts,
|
||||
underlying,
|
||||
expiry,
|
||||
strike,
|
||||
cp,
|
||||
contracts,
|
||||
notional_usd
|
||||
FROM enriched_option_trades
|
||||
WHERE cp = 'C'
|
||||
AND side_est = -1
|
||||
AND cond NOT IN ('a','b','c','d','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v')
|
||||
),
|
||||
paired AS (
|
||||
SELECT
|
||||
o.ts AS option_ts,
|
||||
s.ts AS stock_ts,
|
||||
o.underlying,
|
||||
o.expiry,
|
||||
o.strike,
|
||||
o.contracts,
|
||||
s.shares,
|
||||
ABS(s.shares - o.contracts * 100) AS ratio_error
|
||||
FROM option_sales o
|
||||
JOIN stock_trades_signed s
|
||||
ON s.symbol = o.underlying
|
||||
AND s.side_est = 1
|
||||
AND s.ts BETWEEN o.ts - INTERVAL '5 seconds' AND o.ts + INTERVAL '5 seconds'
|
||||
)
|
||||
SELECT
|
||||
option_ts,
|
||||
stock_ts,
|
||||
underlying,
|
||||
expiry,
|
||||
strike,
|
||||
contracts,
|
||||
shares,
|
||||
ratio_error
|
||||
FROM paired
|
||||
WHERE ratio_error <= contracts * 20
|
||||
ORDER BY option_ts;
|
||||
```
|
||||
|
||||
These SQL patterns are deliberately conservative. They are best used to populate candidate event sets for downstream scoring, not as final labels by themselves. That is especially true for sweeps, retail whales, and buy-write detection, where account-level or route-level data can dramatically improve precision.
|
||||
|
||||
## Open Questions and Limitations
|
||||
|
||||
The biggest limitation is identity. Public OPRA-style options data gives venue, timestamps, quotes, trade conditions, and open interest, but not the beneficial owner, true account class, or reliable open/close intent in real time. Exchange and broker systems may carry professional/customer origin codes or route-chain data, but the public tape generally does not. That means participant-style labels from public data are inferential, not definitive.
|
||||
|
||||
The second limitation is that the literature is not unanimous. Some papers find strong informational content in options flow and meaningful options-led price discovery; other papers find that pre-earnings options activity is often speculative and retail-dominated. Treat that disagreement as a feature, not a bug: it is exactly why your production system should output calibrated probabilities, reason codes, and low-confidence abstentions instead of pretending every urgent call buy is “smart money.”
|
||||
|
||||
The last limitation is regime drift. The SEC’s recent options market-structure work and the newer 0DTE literature both show that short-dated and expiration-day activity has become a much larger share of the market, especially in index products and select equities. Thresholds that worked before widespread 0DTE activity can age badly. Refit by liquidity regime, by DTE bucket, and by event context, or the model will quietly rot.
|
||||
|
|
@ -1,170 +0,0 @@
|
|||
# Server-Backed Persistent History
|
||||
|
||||
## Summary
|
||||
|
||||
Make live mode server-authoritative across refreshes, sessions, and devices. The browser will not own data persistence. On load, the app will hydrate from ClickHouse-backed server history, then layer live WebSocket updates on top. Users will immediately see a substantial recent persisted window, with older records available through history pagination.
|
||||
|
||||
## Chosen Defaults
|
||||
|
||||
- Source of truth: ClickHouse on the server.
|
||||
- Browser persistence: UI preferences only, no market-data cache.
|
||||
- Initial load: recent persisted window per active channel.
|
||||
- Older data: fetched on demand using cursor pagination.
|
||||
- Scope: every channel the server handles, including options, NBBO, equities, equity quotes, equity joins, flow packets, classifier hits, alerts, inferred dark events, candles, and chart overlays.
|
||||
- Freshness: freshness affects status labels only; it must not hide persisted history from a refreshed browser.
|
||||
|
||||
## Current State To Change
|
||||
|
||||
- `LiveStateManager` hydrates from Redis or ClickHouse, but freshness gates currently suppress stale options, NBBO, equities, and flow snapshots.
|
||||
- The unified `/ws/live` protocol supports snapshots and `next_before`, but the frontend does not retain/use per-channel history cursors for live-mode pagination.
|
||||
- Some channels have REST history endpoints, but `equity-quotes` is not fully represented in the unified live protocol/history API.
|
||||
- Charts already query ClickHouse for candle and overlay ranges, but should be treated as part of the same server-history model.
|
||||
|
||||
## Public Interfaces And Types
|
||||
|
||||
Update `packages/types/src/live.ts`:
|
||||
|
||||
- Add `"equity-quotes"` to:
|
||||
- `LiveGenericChannelSchema`
|
||||
- `LiveChannelSchema`
|
||||
- `LiveSubscriptionSchema`
|
||||
- `livePayloadSchemas`
|
||||
- Preserve existing `FeedSnapshot` shape:
|
||||
- `items`
|
||||
- `watermark`
|
||||
- `next_before`
|
||||
|
||||
Update API routes in `services/api/src/index.ts`:
|
||||
|
||||
- Add `GET /history/equity-quotes?before_ts=&before_seq=&limit=`.
|
||||
- Include `equity-quotes` in `/ws/live` subscriptions and fanout.
|
||||
- Keep existing recent/replay endpoints compatible.
|
||||
|
||||
Update storage in `packages/storage/src/clickhouse.ts`:
|
||||
|
||||
- Add `fetchEquityQuotesBefore`.
|
||||
- Reuse existing `(ts, seq)` cursor ordering.
|
||||
- Keep limits clamped consistently with other history endpoints.
|
||||
|
||||
## Server Implementation
|
||||
|
||||
In `services/api/src/live.ts`:
|
||||
|
||||
1. Add generic config for `equity-quotes`:
|
||||
- Redis key: `live:equity-quotes`
|
||||
- cursor field: `equity-quotes`
|
||||
- parser: `EquityQuoteSchema`
|
||||
- cursor: `{ ts, seq }`
|
||||
- fetchRecent: `fetchRecentEquityQuotes`
|
||||
2. Stop filtering historical snapshots by freshness:
|
||||
- Remove `filterFreshGenericItems` from snapshot construction.
|
||||
- Keep `isLiveItemFresh` available for UI status/fanout behavior if needed.
|
||||
- Do not reject persisted ClickHouse rows just because market timestamps are older than 15s/30s.
|
||||
3. Stop rejecting stale ingests inside `LiveStateManager.ingest`.
|
||||
- The manager should store valid events it receives.
|
||||
- Event fanout can still choose how to label status, but should not silently lose durable cache state.
|
||||
4. Preserve Redis as a hot cache:
|
||||
- Redis remains an optimization.
|
||||
- ClickHouse remains the fallback and source of truth.
|
||||
- API startup should hydrate from Redis if present, otherwise from ClickHouse.
|
||||
|
||||
In `services/api/src/index.ts`:
|
||||
|
||||
1. Include `equity-quotes` in `consumerBindings`.
|
||||
2. Pump `EquityQuoteSchema` payloads into:
|
||||
- legacy `/ws/equity-quotes`
|
||||
- unified `/ws/live`
|
||||
- `LiveStateManager`
|
||||
3. Add `/history/equity-quotes`.
|
||||
4. Keep durable consumer defaults unchanged unless a test proves old events are skipped in a live-running API scenario. ClickHouse hydration handles restart and refresh persistence.
|
||||
|
||||
## Frontend Implementation
|
||||
|
||||
In `apps/web/app/terminal.tsx`:
|
||||
|
||||
1. Extend `LiveSessionState` with:
|
||||
- per-subscription `next_before` cursors
|
||||
- per-subscription loading/error state for older history
|
||||
- equity quotes if exposed in UI state
|
||||
2. When handling `snapshot` messages:
|
||||
- Replace the channel's current items with snapshot items when non-empty.
|
||||
- Store `snapshot.next_before`.
|
||||
- Do not discard stale-but-persisted rows.
|
||||
- Continue deduping by `trace_id/seq` or `id`.
|
||||
3. Add a generic live-history loader:
|
||||
- Map subscription channel to history endpoint:
|
||||
- `options` -> `/history/options`
|
||||
- `nbbo` -> `/history/nbbo`
|
||||
- `equities` -> `/history/equities`
|
||||
- `equity-quotes` -> `/history/equity-quotes`
|
||||
- `equity-joins` -> `/history/equity-joins`
|
||||
- `flow` -> `/history/flow`
|
||||
- `classifier-hits` -> `/history/classifier-hits`
|
||||
- `alerts` -> `/history/alerts`
|
||||
- `inferred-dark` -> `/history/inferred-dark`
|
||||
- Carry option/flow filters into options history queries.
|
||||
- Merge older results into existing channel state.
|
||||
- Advance `next_before` from the response.
|
||||
- Stop when `next_before` is null or the response is empty.
|
||||
4. UI behavior:
|
||||
- Add a compact "Load older" control at the bottom of each applicable tape/list.
|
||||
- Disable it while loading.
|
||||
- Hide it when no more history exists.
|
||||
- Keep existing pause/jump controls unchanged.
|
||||
- Do not add browser market-data storage.
|
||||
5. Chart behavior:
|
||||
- Keep candles loading from `/candles/equities`.
|
||||
- Keep overlay loading from `/prints/equities/range`.
|
||||
- Ensure refresh and device changes show the same server data for the same ticker/window.
|
||||
|
||||
## Config And Deployment
|
||||
|
||||
Update `.env.example`:
|
||||
|
||||
- Add `LIVE_LIMIT_EQUITY_QUOTES=10000`.
|
||||
- Document that `LIVE_LIMIT_*` controls initial server snapshot/hot-cache depth, not total persisted history.
|
||||
|
||||
Update README if needed:
|
||||
|
||||
- Clarify persistence model:
|
||||
- ClickHouse is durable history.
|
||||
- Redis is hot cache.
|
||||
- Browser is not a market-data database.
|
||||
- All devices connected to the same API see the same server-seen data.
|
||||
|
||||
Docker volumes already persist ClickHouse/Redis/NATS data locally and in deployment compose, so no migration is needed for volume-backed persistence.
|
||||
|
||||
## Tests
|
||||
|
||||
API tests in `services/api/tests/live.test.ts`:
|
||||
|
||||
- Snapshot hydration returns stale historical options/NBBO/equities/flow instead of filtering them out.
|
||||
- `LiveStateManager.ingest` stores older valid events.
|
||||
- `equity-quotes` hydrates from Redis.
|
||||
- `equity-quotes` hydrates from ClickHouse when Redis is empty.
|
||||
- `next_before` is set from the oldest item in returned snapshot.
|
||||
- Redis hot cache persists hydrated ClickHouse data.
|
||||
|
||||
Storage tests:
|
||||
|
||||
- Add `fetchEquityQuotesBefore` coverage using the existing storage test style.
|
||||
|
||||
Frontend tests in `apps/web/app/terminal.test.ts`:
|
||||
|
||||
- Live snapshot with older persisted rows populates visible rows.
|
||||
- Empty snapshot does not wipe existing visible rows only when preserving an already visible channel during reconnect.
|
||||
- Older-history merge dedupes existing items.
|
||||
- History cursor advances after loading older rows.
|
||||
- "No more history" state is reached when `next_before` is null.
|
||||
- Live status can be stale while items remain visible.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- Refreshing the app shows persisted data immediately, even when no new live events arrive after page load.
|
||||
- Opening the app on another device connected to the same API shows the same server-backed recent history.
|
||||
- Stale market timestamps do not cause persisted history to disappear.
|
||||
- Users can load older data beyond the initial recent window.
|
||||
- Live WebSocket updates still appear without requiring refresh.
|
||||
- Redis loss does not lose history; API falls back to ClickHouse.
|
||||
- Browser cache deletion does not lose market data.
|
||||
- `bun test services/api/tests/live.test.ts apps/web/app/terminal.test.ts packages/storage/tests/*.test.ts` passes, or any unavailable test target is documented.
|
||||
|
|
@ -1,320 +0,0 @@
|
|||
# Options Overhaul Phase 1: Snapshot Tape Table
|
||||
|
||||
## Summary
|
||||
|
||||
Convert the Options tape into a dense table where every row is an individual option print with preserved execution context. The print itself becomes the authoritative record for what was known around that trade at the moment it printed: option NBBO, underlying spot, IV, notional, side/classification metadata, and classifier-derived row coloring.
|
||||
|
||||
This phase includes backend enrichment, storage/type changes, synthetic IV behavior, and the frontend table redesign together.
|
||||
|
||||
## Core Principle
|
||||
|
||||
Do not treat NBBO, spot, or IV as live lookups in the table once the print has been recorded.
|
||||
|
||||
Each option print should carry a snapshot of its execution context. The UI should prefer those preserved fields and only fall back to current side maps for legacy rows that predate the migration.
|
||||
|
||||
## Public Type Changes
|
||||
|
||||
Extend `OptionPrintSchema` / `OptionPrint` in `packages/types/src/events.ts`.
|
||||
|
||||
Add optional flat fields:
|
||||
|
||||
```ts
|
||||
execution_nbbo_bid?: number;
|
||||
execution_nbbo_ask?: number;
|
||||
execution_nbbo_mid?: number;
|
||||
execution_nbbo_spread?: number;
|
||||
execution_nbbo_bid_size?: number;
|
||||
execution_nbbo_ask_size?: number;
|
||||
execution_nbbo_ts?: number;
|
||||
execution_nbbo_age_ms?: number;
|
||||
execution_nbbo_side?: OptionNbboSide;
|
||||
|
||||
execution_underlying_spot?: number;
|
||||
execution_underlying_bid?: number;
|
||||
execution_underlying_ask?: number;
|
||||
execution_underlying_mid?: number;
|
||||
execution_underlying_spread?: number;
|
||||
execution_underlying_ts?: number;
|
||||
execution_underlying_age_ms?: number;
|
||||
execution_underlying_source?: "equity_quote_mid";
|
||||
|
||||
execution_iv?: number;
|
||||
execution_iv_source?: "provider" | "synthetic_pressure_model";
|
||||
```
|
||||
|
||||
Keep existing fields for compatibility:
|
||||
|
||||
- `nbbo_side`
|
||||
- `notional`
|
||||
- `underlying_id`
|
||||
- `option_type`
|
||||
- `signal_*`
|
||||
|
||||
Set `nbbo_side` to match `execution_nbbo_side` for new prints so existing filters continue working.
|
||||
|
||||
## Storage Changes
|
||||
|
||||
Update `packages/storage/src/option-prints.ts`.
|
||||
|
||||
Add ClickHouse columns:
|
||||
|
||||
```sql
|
||||
execution_nbbo_bid Nullable(Float64),
|
||||
execution_nbbo_ask Nullable(Float64),
|
||||
execution_nbbo_mid Nullable(Float64),
|
||||
execution_nbbo_spread Nullable(Float64),
|
||||
execution_nbbo_bid_size Nullable(UInt32),
|
||||
execution_nbbo_ask_size Nullable(UInt32),
|
||||
execution_nbbo_ts Nullable(UInt64),
|
||||
execution_nbbo_age_ms Nullable(Float64),
|
||||
execution_nbbo_side Nullable(String),
|
||||
|
||||
execution_underlying_spot Nullable(Float64),
|
||||
execution_underlying_bid Nullable(Float64),
|
||||
execution_underlying_ask Nullable(Float64),
|
||||
execution_underlying_mid Nullable(Float64),
|
||||
execution_underlying_spread Nullable(Float64),
|
||||
execution_underlying_ts Nullable(UInt64),
|
||||
execution_underlying_age_ms Nullable(Float64),
|
||||
execution_underlying_source Nullable(String),
|
||||
|
||||
execution_iv Nullable(Float64),
|
||||
execution_iv_source Nullable(String)
|
||||
```
|
||||
|
||||
Add `ALTER TABLE ... ADD COLUMN IF NOT EXISTS` migrations for all fields.
|
||||
|
||||
Update row normalization so missing legacy values parse as `undefined`.
|
||||
|
||||
## Ingest Enrichment
|
||||
|
||||
Update `services/ingest-options/src/index.ts`.
|
||||
|
||||
Maintain caches:
|
||||
|
||||
- latest option NBBO by contract
|
||||
- latest equity quote by underlying
|
||||
- synthetic/adapter-provided IV by contract when available
|
||||
|
||||
When an option trade arrives:
|
||||
|
||||
1. Parse raw print.
|
||||
2. Derive underlying, option type, notional, ETF flag as today.
|
||||
3. Select latest option NBBO for the contract at or before `print.ts`.
|
||||
4. Attach preserved NBBO fields:
|
||||
- bid, ask, mid, spread
|
||||
- bid/ask sizes
|
||||
- quote timestamp
|
||||
- quote age
|
||||
- execution NBBO side
|
||||
5. Select latest equity quote for the underlying at or before `print.ts`.
|
||||
6. Attach preserved underlying fields:
|
||||
- bid, ask, mid
|
||||
- spread
|
||||
- quote timestamp
|
||||
- quote age
|
||||
- `execution_underlying_spot = mid`
|
||||
- `execution_underlying_source = "equity_quote_mid"`
|
||||
7. Attach IV if available.
|
||||
8. Evaluate signal filters using preserved execution fields.
|
||||
9. Persist and publish the enriched print.
|
||||
|
||||
Important behavior:
|
||||
|
||||
- Do not mark these preserved fields stale in the UI.
|
||||
- Age fields are still stored for auditability.
|
||||
- If no at-or-before quote exists, leave that context unset.
|
||||
- Never use a quote after the option print timestamp for preserved execution context.
|
||||
|
||||
## Synthetic IV Model
|
||||
|
||||
Update `services/ingest-options/src/adapters/synthetic.ts`.
|
||||
|
||||
Add persistent contract-level IV state:
|
||||
|
||||
```ts
|
||||
type SyntheticContractIvState = {
|
||||
iv: number;
|
||||
pressure: number;
|
||||
lastTs: number;
|
||||
};
|
||||
```
|
||||
|
||||
Behavior:
|
||||
|
||||
- Initialize IV from a plausible baseline based on DTE and moneyness.
|
||||
- Maintain IV per contract across bursts.
|
||||
- Repeated aggressive buying of the same contract raises pressure and IV.
|
||||
- Aggressive buying means synthetic placement `A` or `AA`.
|
||||
- `MID` has small/no pressure.
|
||||
- `B` or `BB` reduces pressure slightly.
|
||||
- Pressure decays over time after inactivity.
|
||||
- IV is clamped to a plausible range.
|
||||
|
||||
Recommended defaults:
|
||||
|
||||
- Baseline IV: `0.18` to `0.65`
|
||||
- 0DTE contracts start higher than far-dated contracts.
|
||||
- Out-of-the-money contracts start slightly higher than near-the-money contracts.
|
||||
- Ask/above-ask print pressure increment: proportional to size and notional.
|
||||
- Decay half-life: roughly 30-90 seconds in synthetic time.
|
||||
- Clamp IV to `0.05..2.5`.
|
||||
|
||||
Each synthetic `OptionPrint` should include:
|
||||
|
||||
```ts
|
||||
execution_iv
|
||||
execution_iv_source: "synthetic_pressure_model"
|
||||
```
|
||||
|
||||
Synthetic NBBO and trade price generation should remain coherent:
|
||||
|
||||
- As IV rises, option mid/ask should drift higher for that contract.
|
||||
- Rapid same-contract buying should visibly increase both print price and IV over subsequent prints.
|
||||
- Bid/ask spread may widen mildly with higher IV.
|
||||
|
||||
## Real Adapter IV Behavior
|
||||
|
||||
For Alpaca, Databento, and IBKR in Phase 1:
|
||||
|
||||
- Preserve NBBO and underlying spot context through ingest enrichment.
|
||||
- Leave `execution_iv` unset unless the adapter already provides a reliable IV value.
|
||||
- Do not invent IV for real feeds in Phase 1.
|
||||
|
||||
Synthetic is the only source that must generate IV in this phase.
|
||||
|
||||
## Frontend Table Redesign
|
||||
|
||||
Update `apps/web/app/terminal.tsx` and `apps/web/app/globals.css`.
|
||||
|
||||
Each Options row remains an `OptionPrint`.
|
||||
|
||||
Default columns:
|
||||
|
||||
- `TIME`
|
||||
- `SYM`
|
||||
- `EXP`
|
||||
- `STRIKE`
|
||||
- `C/P`
|
||||
- `SPOT`
|
||||
- `DETAILS`
|
||||
- `TYPE`
|
||||
- `VALUE`
|
||||
- `SIDE`
|
||||
- `IV`
|
||||
- `CLASSIFIER`
|
||||
|
||||
Column sources:
|
||||
|
||||
- `SPOT`: `execution_underlying_spot`, fallback `--`
|
||||
- `SIDE`: `execution_nbbo_side ?? nbbo_side`
|
||||
- `IV`: `execution_iv`, formatted as percent, fallback `--`
|
||||
- `DETAILS`: `{size}@{price}_{side}`
|
||||
- `VALUE`: `notional ?? price * size * 100`
|
||||
|
||||
For legacy rows only:
|
||||
|
||||
- If preserved NBBO is missing, fallback to existing frontend NBBO map.
|
||||
- If preserved spot/IV is missing, render `--`.
|
||||
|
||||
## Classifier Row Coloring
|
||||
|
||||
Add derived indexes in `TerminalProvider`:
|
||||
|
||||
- `classifierHitsByPacketId`
|
||||
- `packetIdByOptionTraceId`
|
||||
- `classifierDecorByOptionTraceId`
|
||||
|
||||
A print inherits classifier color if its trace ID belongs to a flow packet that produced classifier hits.
|
||||
|
||||
Primary hit selection:
|
||||
|
||||
1. Highest confidence
|
||||
2. Newest `source_ts`
|
||||
3. Highest `seq`
|
||||
|
||||
Classifier families:
|
||||
|
||||
- `large_bullish_call_sweep`: green
|
||||
- `large_bearish_put_sweep`: red
|
||||
- `unusual_contract_spike`: amber
|
||||
- `large_call_sell_overwrite`: copper
|
||||
- `large_put_sell_write`: copper
|
||||
- `straddle` / `strangle`: blue
|
||||
- `vertical_spread`: teal
|
||||
- `ladder_accumulation`: yellow-green
|
||||
- `roll_up_down_out`: violet
|
||||
- `far_dated_conviction`: cyan
|
||||
- `zero_dte_gamma_punch`: magenta
|
||||
- unknown: neutral
|
||||
|
||||
Confidence controls row intensity.
|
||||
|
||||
## Interaction
|
||||
|
||||
Classified rows:
|
||||
|
||||
- Click opens existing classifier/alert drawer behavior through `state.openFromClassifierHit(primaryHit)`.
|
||||
- Keyboard Enter/Space does the same.
|
||||
- Row remains compact and table-like.
|
||||
|
||||
Unclassified rows:
|
||||
|
||||
- Hover only.
|
||||
- No drawer action.
|
||||
|
||||
## Live Manifest
|
||||
|
||||
Update `/tape` live subscriptions to include classifier hits:
|
||||
|
||||
```ts
|
||||
[
|
||||
{ channel: "options", filters: flowFilters },
|
||||
{ channel: "nbbo" },
|
||||
{ channel: "equities" },
|
||||
{ channel: "flow", filters: flowFilters },
|
||||
{ channel: "classifier-hits" }
|
||||
]
|
||||
```
|
||||
|
||||
The table uses preserved execution context from options first, not these side feeds.
|
||||
|
||||
## Tests
|
||||
|
||||
Add/update tests for:
|
||||
|
||||
- `OptionPrintSchema` accepts preserved execution context fields.
|
||||
- ClickHouse option print normalization handles missing legacy context fields.
|
||||
- Ingest enrichment attaches preserved NBBO context.
|
||||
- Ingest enrichment attaches preserved underlying quote mid as spot.
|
||||
- Enrichment never uses quotes after the option print timestamp.
|
||||
- `nbbo_side` mirrors `execution_nbbo_side` for new enriched prints.
|
||||
- Synthetic IV increases under repeated same-contract ask/above-ask buying.
|
||||
- Synthetic IV decays after inactivity.
|
||||
- Synthetic IV remains within clamps.
|
||||
- Options table renders SPOT from `execution_underlying_spot`.
|
||||
- Options table renders IV from `execution_iv`.
|
||||
- Legacy rows render `--` for missing SPOT/IV.
|
||||
- Classifier family mapping and primary hit selection work.
|
||||
- Classified row opens existing classifier/alert drawer path.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- The Options tape is a dense table, not card rows.
|
||||
- Every new option print stores preserved execution NBBO context.
|
||||
- Every new option print stores preserved execution underlying spot when an at-or-before equity quote exists.
|
||||
- Synthetic option prints store dynamic IV.
|
||||
- Synthetic repeated buying of the same contract visibly increases IV.
|
||||
- The table reads NBBO, SPOT, and IV from preserved print fields first.
|
||||
- Classifier-hit rows are color-coded by classifier family.
|
||||
- Existing live/replay filters and tape controls still work.
|
||||
- No context field is visually treated as stale after being attached to the print.
|
||||
- Legacy data remains readable with graceful fallbacks.
|
||||
|
||||
## Assumptions
|
||||
|
||||
- Phase 1 uses flat fields for queryability and simple table rendering.
|
||||
- Underlying spot means equity quote mid at or before the option print timestamp.
|
||||
- NBBO context means option quote at or before the option print timestamp.
|
||||
- Preserved age fields are audit metadata, not UI freshness warnings.
|
||||
- Real-feed IV can remain absent until a reliable provider value is available.
|
||||
Loading…
Add table
Add a link
Reference in a new issue