# Docker Deployment This directory contains a VPS-oriented Docker deployment for the full Islandflow stack. It is separate from the repo-root `docker-compose.yml`, which is still the lightweight local infra stack for development. ## What this stack does - Assumes Nginx Proxy Manager is the edge proxy and runs on the same Docker network. - Keeps `web` and `api` internal to the Docker network instead of publishing host ports. - Targets a two-subdomain routing model by default: - `app.` -> `web:3000` - `api.` -> `api:4000` - Runs ClickHouse, Redis, and NATS JetStream with persistent Docker volumes. - Runs the core runtime services: `ingest-options`, `ingest-equities`, `compute`, `candles`, `api`, and `web`. - Keeps `replay` opt-in through a Compose profile, because the current replay service starts immediately when the container is enabled. ## Files - `deployment/docker/docker-compose.yml`: production-style stack for a single VPS - `deployment/docker/Dockerfile.service`: shared Bun runtime image for most services - `deployment/docker/Dockerfile.ingest-options`: Bun runtime plus Python dependencies for Databento and IBKR adapters - `deployment/docker/Dockerfile.web`: multi-stage build for the Next.js web app - `deployment/docker/.env.example`: container-oriented environment template ## Prerequisites - A Linux VPS with Docker Engine and Docker Compose v2 installed - Enough RAM for ClickHouse plus the Bun services - Nginx Proxy Manager running in Docker on the same host/network path you plan to use Optional: - A DNS record pointed at the VPS - Alpaca, Databento, or IBKR credentials if you are not using the synthetic adapters ## First deployment 1. Copy the env template: ```bash cd deployment/docker cp .env.example .env ``` 2. Edit `.env`. Important defaults: - `NATS_URL`, `CLICKHOUSE_URL`, and `REDIS_URL` should stay on the internal container hostnames unless you intentionally split infra out. - `OPTIONS_INGEST_ADAPTER=synthetic` and `EQUITIES_INGEST_ADAPTER=synthetic` are the safest first boot settings. - `NEXT_PUBLIC_API_URL=https://api.example.com` is the recommended production shape when using NPM with two subdomains. 3. Build and start the stack: ```bash docker compose up -d --build ``` 4. Confirm the containers are healthy: ```bash docker compose ps docker compose logs -f api web compute candles ingest-options ingest-equities ``` 5. Make sure NPM can reach the stack network. The Compose project name is pinned to `islandflow-vps`, so the default network name will be: ```bash islandflow-vps_default ``` If your NPM container is separate, connect it once: ```bash docker network connect islandflow-vps_default ``` 6. Create these NPM proxy hosts: - `app.example.com` -> forward to `web`, port `3000` - `api.example.com` -> forward to `api`, port `4000` For the API host, enable websocket support. 7. Open the app: - `https://app.example.com/` - Health check: `https://api.example.com/health` ## Replay service Replay is disabled by default in this stack. Start it only when you want it: ```bash docker compose --profile replay up -d replay ``` Stop it again: ```bash docker compose stop replay ``` ## Adapter notes ### Synthetic mode This is the easiest way to smoke-test the deployment: - `OPTIONS_INGEST_ADAPTER=synthetic` - `EQUITIES_INGEST_ADAPTER=synthetic` ### Alpaca mode Set the adapter values and credentials in `.env`: - `OPTIONS_INGEST_ADAPTER=alpaca` - `EQUITIES_INGEST_ADAPTER=alpaca` - `ALPACA_KEY_ID=...` - `ALPACA_SECRET_KEY=...` ### Databento mode The `ingest-options` image in this deployment includes Python plus the repo’s sidecar dependencies, so Databento can run without a custom image. Set the Databento env vars in `.env`, especially: - `OPTIONS_INGEST_ADAPTER=databento` - `DATABENTO_API_KEY=...` - `DATABENTO_START=...` ### IBKR mode If TWS or IB Gateway is running on the VPS host, the default `.env.example` already points `IBKR_HOST` at `host.docker.internal`, and the Compose stack adds the required host gateway mapping. If IBKR is running somewhere else, change: - `IBKR_HOST` - `IBKR_PORT` ## NPM routing Recommended proxy hosts: - `app.` -> `web:3000` - `api.` -> `api:4000` The web app should be built with `NEXT_PUBLIC_API_URL=https://api.` so browser REST and websocket traffic goes straight to the API host through NPM. The API host needs websocket support enabled because the app uses `/ws/*` endpoints for live streams. ## Updating the deployment When you pull new code: ```bash cd deployment/docker docker compose up -d --build ``` If you changed only env values for the Bun services: ```bash docker compose up -d ``` If you changed `NEXT_PUBLIC_API_URL` or `NEXT_PUBLIC_NBBO_MAX_AGE_MS`, rebuild the web image because those are public Next.js build-time values: ```bash docker compose build web docker compose up -d web ``` ## Backups and persistence Persistent data lives in Docker volumes: - `clickhouse-data` - `redis-data` - `nats-data` Before destructive maintenance, back up those volumes or the underlying Docker data directory for the host. ## Shutdown Stop everything while keeping data: ```bash docker compose down ``` Stop everything and remove volumes too: ```bash docker compose down -v ``` Only use `-v` if you intentionally want to wipe ClickHouse, Redis, and JetStream state. ## Known caveats - The root `.env.example` still contains a `REPLAY_ENABLED` comment, but the current replay service does not read that variable. Use the Compose replay profile instead. - This stack does not publish `web` or `api` to host ports. NPM must be able to resolve `web` and `api` over the shared Docker network. - The stack assumes a single-node VPS deployment. If you later split infra or add external managed services, update the three core connection URLs in `.env`. ## Smoke checks After NPM is wired up: - `https://app./` should load the UI. - Browser network requests from the UI should target `https://api./...`. - Live feeds should connect over `wss://api./ws/...`. - `docker compose ps` should show no service publishing host port `80`.