islandflow/deployment/docker/README.md

Docker Deployment

This directory contains a VPS-oriented Docker deployment for the full Islandflow stack.

It is separate from the repo-root docker-compose.yml, which is still the lightweight local infra stack for development.

What this stack does

• Assumes Nginx Proxy Manager is the edge proxy and runs on the shared Docker network named bridge.
• Keeps web and api internal to the Docker network instead of publishing host ports.
• Targets a two-subdomain routing model by default:
  • app.<domain> -> web:3000
  • api.<domain> -> api:4000
• Runs ClickHouse, Redis, and NATS JetStream with persistent Docker volumes.
• Runs the core runtime services: ingest-options, ingest-equities, compute, candles, api, and web.
• Keeps replay opt-in through a Compose profile, because the current replay service starts immediately when the container is enabled.

Files

• deployment/docker/docker-compose.yml: production-style stack for a single VPS
• deployment/docker/Dockerfile.service: shared Bun runtime image for most services
• deployment/docker/Dockerfile.ingest-options: Bun runtime plus Python dependencies for Databento and IBKR adapters
• deployment/docker/Dockerfile.web: multi-stage build for the Next.js web app
• deployment/docker/.env.example: container-oriented environment template

Prerequisites

• A Linux VPS with Docker Engine and Docker Compose v2 installed
• Enough RAM for ClickHouse plus the Bun services
• Nginx Proxy Manager running in Docker on the same host
• A shared Docker network named bridge

Optional:

• A DNS record pointed at the VPS
• Alpaca, Databento, or IBKR credentials if you are not using the synthetic adapters

First deployment

1. Copy the env template:

cd deployment/docker
cp .env.example .env

2. Edit .env.

Important defaults:

• NATS_URL, CLICKHOUSE_URL, and REDIS_URL should stay on the internal container hostnames unless you intentionally split infra out.
• OPTIONS_INGEST_ADAPTER=synthetic and EQUITIES_INGEST_ADAPTER=synthetic are the safest first boot settings.
• NEXT_PUBLIC_API_URL=https://api.example.com is the recommended production shape when using NPM with two subdomains.

3. Build and start the stack:

docker compose up -d --build

4. Confirm the containers are healthy:

docker compose ps
docker compose logs -f api web compute candles ingest-options ingest-equities

5. Make sure NPM can reach the stack network.

This deployment attaches web and api to the external Docker network named bridge, in addition to the stack-local network.

If your NPM container is not already attached to bridge, connect it once:

docker network connect bridge <npm-container-name>

If your NPM stack uses a different shared user-defined network, update the bridge network block in deployment/docker/docker-compose.yml to point at that external network name, then redeploy. The important part is that NPM, web, and api all share the same external Docker network.

6. Create these NPM proxy hosts:

• app.example.com -> forward to web, port 3000
• api.example.com -> forward to api, port 4000

For the API host, enable websocket support.

7. Open the app:

• https://app.example.com/
• Health check: https://api.example.com/health

Replay service

Replay is disabled by default in this stack.

Start it only when you want it:

docker compose --profile replay up -d replay

Stop it again:

docker compose stop replay

Adapter notes

Synthetic mode

This is the easiest way to smoke-test the deployment:

• OPTIONS_INGEST_ADAPTER=synthetic
• EQUITIES_INGEST_ADAPTER=synthetic

Alpaca mode

Set the adapter values and credentials in .env:

• OPTIONS_INGEST_ADAPTER=alpaca
• EQUITIES_INGEST_ADAPTER=alpaca
• ALPACA_KEY_ID=...
• ALPACA_SECRET_KEY=...

Databento mode

The ingest-options image in this deployment includes Python plus the repo’s sidecar dependencies, so Databento can run without a custom image. Set the Databento env vars in .env, especially:

• OPTIONS_INGEST_ADAPTER=databento
• DATABENTO_API_KEY=...
• DATABENTO_START=...

IBKR mode

If TWS or IB Gateway is running on the VPS host, the default .env.example already points IBKR_HOST at host.docker.internal, and the Compose stack adds the required host gateway mapping.

If IBKR is running somewhere else, change:

• IBKR_HOST
• IBKR_PORT

NPM routing

Recommended proxy hosts:

• app.<domain> -> web:3000
• api.<domain> -> api:4000

The web app should be built with NEXT_PUBLIC_API_URL=https://api.<domain> so browser REST and websocket traffic goes straight to the API host through NPM.

The API host needs websocket support enabled because the app uses /ws/* endpoints for live streams.

Because web and api are both attached to bridge, NPM can target them directly by container DNS name:

• web
• api

Updating the deployment

When you pull new code:

cd deployment/docker
docker compose up -d --build

If you changed only env values for the Bun services:

docker compose up -d

If you changed NEXT_PUBLIC_API_URL or NEXT_PUBLIC_NBBO_MAX_AGE_MS, rebuild the web image because those are public Next.js build-time values:

docker compose build web
docker compose up -d web

Backups and persistence

Persistent data lives in Docker volumes:

• clickhouse-data
• redis-data
• nats-data

Before destructive maintenance, back up those volumes or the underlying Docker data directory for the host.

Shutdown

Stop everything while keeping data:

docker compose down

Stop everything and remove volumes too:

docker compose down -v

Only use -v if you intentionally want to wipe ClickHouse, Redis, and JetStream state.

Known caveats

• The root .env.example still contains a REPLAY_ENABLED comment, but the current replay service does not read that variable. Use the Compose replay profile instead.
• This stack does not publish web or api to host ports. NPM must be able to resolve web and api over the shared bridge network.
• The stack assumes a single-node VPS deployment. If you later split infra or add external managed services, update the three core connection URLs in .env.

Smoke checks

After NPM is wired up:

• https://app.<domain>/ should load the UI.
• Browser network requests from the UI should target https://api.<domain>/....
• Live feeds should connect over wss://api.<domain>/ws/....
• docker compose ps should show no service publishing host port 80.