diff --git a/deployment/docker/.env.example b/deployment/docker/.env.example index 986968c..0e1df23 100644 --- a/deployment/docker/.env.example +++ b/deployment/docker/.env.example @@ -6,17 +6,19 @@ CLICKHOUSE_DATABASE=default REDIS_URL=redis://redis:6379 API_PORT=4000 +API_BIND_IP=127.0.0.1 +API_HOST_PORT=4000 +WEB_BIND_IP=127.0.0.1 +WEB_HOST_PORT=3000 REST_DEFAULT_LIMIT=200 API_DELIVER_POLICY=new API_CONSUMER_RESET=false -NPM_SHARED_NETWORK=npm-shared - -# Recommended with NPM on the same Docker network: -# app. -> web:3000 -# api. -> api:4000 -# Leave NEXT_PUBLIC_API_URL empty to use same-origin mode -# (app. serves UI and proxies API paths to api:4000). +# Public web build target: +# - Set NEXT_PUBLIC_API_URL=https://api.example.com when an external proxy +# or load balancer serves the API on a distinct origin. +# - Leave NEXT_PUBLIC_API_URL empty to use same-origin mode and proxy API +# paths to the published API host port yourself. NEXT_PUBLIC_API_URL=https://api.example.com NEXT_PUBLIC_NBBO_MAX_AGE_MS=1000 diff --git a/deployment/docker/README.md b/deployment/docker/README.md index 13b619a..7822dbd 100644 --- a/deployment/docker/README.md +++ b/deployment/docker/README.md @@ -1,16 +1,15 @@ # Docker Deployment -This directory contains a VPS-oriented Docker deployment for the full Islandflow stack. +This directory is the supported VPS deployment path for Islandflow. -It is separate from the repo-root `docker-compose.yml`, which is still the lightweight local infra stack for development. +The repo no longer ships or supports a separate `deployment/npm` stack. Docker Compose is the deployment surface; if you want a reverse proxy, point it at the host ports published by this stack. + +It is separate from the repo-root `docker-compose.yml`, which remains the lightweight local infra stack for development. ## What this stack does -- Assumes Nginx Proxy Manager is the edge proxy and runs on a shared user-defined Docker network. -- Keeps `web` and `api` internal to the Docker network instead of publishing host ports. -- Targets a two-subdomain routing model by default: - - `app.` -> `web:3000` - - `api.` -> `api:4000` +- Builds and runs the full Islandflow stack with Docker Compose. +- Publishes `web` and `api` to host ports, bound to loopback by default. - Runs ClickHouse, Redis, and NATS JetStream with persistent Docker volumes. - Runs the core runtime services: `ingest-options`, `ingest-equities`, `compute`, `candles`, `api`, and `web`. - Keeps `replay` opt-in through a Compose profile, because the current replay service starts immediately when the container is enabled. @@ -29,12 +28,11 @@ It is separate from the repo-root `docker-compose.yml`, which is still the light - A Linux VPS with Docker Engine and Docker Compose v2 installed - Enough RAM for ClickHouse plus the Bun services -- Nginx Proxy Manager running in Docker on the same host -- A shared user-defined Docker network for NPM and this stack Optional: - A DNS record pointed at the VPS +- Any reverse proxy or load balancer you prefer - Alpaca, Databento, or IBKR credentials if you are not using the synthetic adapters ## First deployment @@ -51,23 +49,14 @@ cp .env.example .env Important defaults: - `NATS_URL`, `CLICKHOUSE_URL`, and `REDIS_URL` should stay on the internal container hostnames unless you intentionally split infra out. -- `OPTIONS_INGEST_ADAPTER=synthetic` and `EQUITIES_INGEST_ADAPTER=synthetic` are the safest first boot settings. -- `NPM_SHARED_NETWORK=npm-shared` is the recommended external Docker network name for NPM and this stack. -- `NEXT_PUBLIC_API_URL=https://api.example.com` uses a two-subdomain setup (`app` + `api`). -- `NEXT_PUBLIC_API_URL=` (empty) uses same-origin mode where the app host also proxies API paths to `api:4000`. +- `OPTIONS_INGEST_ADAPTER=synthetic` and `EQUITIES_INGEST_ADAPTER=synthetic` are the safest first-boot settings. +- `WEB_BIND_IP=127.0.0.1` and `API_BIND_IP=127.0.0.1` keep the published ports local to the host by default. +- `WEB_HOST_PORT=3000` and `API_HOST_PORT=4000` control the host-side published ports. +- `NEXT_PUBLIC_API_URL=https://api.example.com` fits a two-origin setup where the browser reaches the API on a separate public origin. +- `NEXT_PUBLIC_API_URL=` (empty) fits same-origin mode where your edge layer proxies API paths from the app origin to the API host port. 3. Build and start the stack: -If you have not created the shared Docker network yet, do that once first: - -```bash -docker network create npm-shared -``` - -Then make sure `.env` keeps `NPM_SHARED_NETWORK=npm-shared`, or set it to whatever user-defined network you want to share with NPM. - -Now build and start the stack: - ```bash docker compose up -d --build ``` @@ -86,31 +75,44 @@ docker compose ps docker compose logs -f api web compute candles ingest-options ingest-equities ``` -5. Make sure NPM can reach the stack network. +5. Open the app. -This deployment attaches `web` and `api` to the external Docker network named by `NPM_SHARED_NETWORK`, in addition to the stack-local network. +With the default loopback binding: -If your NPM container is not already attached to that network, connect it once: +- UI: `http://127.0.0.1:3000/` +- Health check: `http://127.0.0.1:4000/health` -```bash -docker network connect npm-shared -``` +If you want direct remote access without a reverse proxy, change `WEB_BIND_IP` and `API_BIND_IP` to `0.0.0.0` and restrict exposure with your firewall. -If you want to use a different network name, set `NPM_SHARED_NETWORK` in `.env` and make sure that external Docker network already exists. The important part is that NPM, `web`, and `api` all share the same user-defined Docker network. +## Access patterns -6. Create these NPM proxy hosts: +### Direct host-port mode -- `app.example.com` -> forward to `web` (or `islandflow-vps-web-1`), port `3000` -- `api.example.com` -> forward to `api` (or `islandflow-vps-api-1`), port `4000` +Use this when you want Docker alone to serve the app: -For the API host, enable websocket support. +- set `WEB_BIND_IP=0.0.0.0` +- set `API_BIND_IP=0.0.0.0` +- optionally change `WEB_HOST_PORT` / `API_HOST_PORT` +- point DNS or clients at the host directly -If NPM is attached to multiple Docker networks and another stack also has an `api` container alias, prefer the explicit container name (`islandflow-vps-api-1`) to avoid DNS collisions. +### Reverse proxy mode -7. Open the app: +If you use Caddy, Nginx, Traefik, a cloud load balancer, or another edge layer, proxy to the published host ports from this stack. The repo does not require or manage any specific proxy anymore. -- `https://app.example.com/` -- Health check: `https://api.example.com/health` +Supported routing modes: + +1. Two-origin mode + - `app.` -> host `WEB_HOST_PORT` + - `api.` -> host `API_HOST_PORT` + - Build web with `NEXT_PUBLIC_API_URL=https://api.`. + +2. Same-origin mode + - Build web with `NEXT_PUBLIC_API_URL=` (empty). + - Point `app.` at the web host port. + - Proxy these API routes from the app origin to the API host port: + - `/ws/*`, `/replay/*`, `/prints/*`, `/joins/*`, `/nbbo/*`, `/dark/*`, `/flow/*`, `/candles/*` + +Enable websocket support on whichever host serves `/ws/*`. ## Replay service @@ -163,30 +165,9 @@ If IBKR is running somewhere else, change: - `IBKR_HOST` - `IBKR_PORT` -## NPM routing - -The Islandflow stack expects an external NPM instance on the shared Docker network. The dedicated NPM stack now lives in `../npm`. - -Supported routing modes: - -1. Two-subdomain mode - - `app.` -> `web:3000` - - `api.` -> `api:4000` - - Build web with `NEXT_PUBLIC_API_URL=https://api.`. - -2. Same-origin fallback mode - - Build web with `NEXT_PUBLIC_API_URL=` (empty). - - Keep `app.` -> web. - - Add path-based proxy rules on `app.` for API routes to `api:4000`: - - `/ws/*`, `/replay/*`, `/prints/*`, `/joins/*`, `/nbbo/*`, `/dark/*`, `/flow/*`, `/candles/*` - -Use websocket support on whichever host serves `/ws/*`. - -If NPM is on multiple networks and names collide (for example another stack also exposes `api`), target explicit container names (`islandflow-vps-api-1`, `islandflow-vps-web-1`) instead of generic aliases. - ## Updating the deployment -This deployment installs dependencies from `deployment/docker/workspace-root/bun.lock` (not the repo-root lockfile). +This deployment installs dependencies from `deployment/docker/workspace-root/bun.lock` rather than the repo-root lockfile. When dependencies change in any workspace used by Docker builds, refresh and validate the deployment snapshot first: @@ -210,9 +191,8 @@ The checked-in deploy helper is meant to run from your local repo checkout, not - SSH key: `~/.ssh/delta_ed25519` - Live repo checkout: `/home/delta/islandflow` - Live compose directory: `/home/delta/islandflow/deployment/docker` -- Shared proxy network: `npm-shared` -It preserves the current proxy topology, reuses the existing Docker Compose project, and avoids destructive cleanup on the server. +It preserves the current Docker Compose project and avoids destructive cleanup on the server. ### Deploy `origin/main` @@ -271,11 +251,11 @@ The helper always does the final public verification against: - `https://flow.deltaisland.io` - `https://api.flow.deltaisland.io/health` -It also uses container-local health checks inside `islandflow-vps-api-1` and `islandflow-vps-web-1`, because host loopback `127.0.0.1:4000` is not the right primary check for this topology. +Those checks assume your current edge routing already forwards those domains to the host ports published by this stack. ## Manual server fallback -If you need to run the rollout steps manually over SSH, use the same live checkout and compose directory. Avoid `git clean -fd`, `git reset --hard`, proxy changes, or Docker network recreation during normal app rollouts. +If you need to run the rollout steps manually over SSH, use the same live checkout and compose directory. Avoid `git clean -fd`, `git reset --hard`, or other destructive cleanup during normal app rollouts. Deploy `main` manually: @@ -349,16 +329,16 @@ Only use `-v` if you intentionally want to wipe ClickHouse, Redis, and JetStream ## Known caveats - The root `.env.example` still contains a `REPLAY_ENABLED` comment, but the current replay service does not read that variable. Use the Compose replay profile instead. -- This stack does not publish `web` or `api` to host ports. NPM must be able to resolve `web` and `api` over the shared user-defined network from `NPM_SHARED_NETWORK`. -- If NPM is attached to more than one application network, generic upstream aliases like `api` can resolve to the wrong stack. Prefer explicit container names in NPM upstream settings. +- `web` and `api` bind to loopback by default. External access requires changing the bind IPs or placing a reverse proxy in front of the published host ports. - Some hosts disable IPv6 inside containers; the bundled ClickHouse config pins `listen_host` to `0.0.0.0` so the API can reach ClickHouse reliably over Docker networking. - The stack assumes a single-node VPS deployment. If you later split infra or add external managed services, update the three core connection URLs in `.env`. ## Smoke checks -After NPM is wired up: +After the stack is up: -- `https://app./` should load the UI. -- In two-subdomain mode, browser requests should target `https://api./...` and live feeds should use `wss://api./ws/...`. +- `docker compose ps` should show healthy `api`, `web`, `clickhouse`, and `redis` services. +- `curl http://127.0.0.1:4000/health` should return a healthy response on the server. +- `curl -I http://127.0.0.1:3000/` should return a successful HTTP status on the server. +- In two-origin mode, browser requests should target `https://api./...` and live feeds should use `wss://api./ws/...`. - In same-origin mode, browser requests should target `https://app./...` for API paths and live feeds should use `wss://app./ws/...`. -- `docker compose ps` should show no service publishing host port `80`. diff --git a/deployment/docker/docker-compose.yml b/deployment/docker/docker-compose.yml index 96598ba..c9eb610 100644 --- a/deployment/docker/docker-compose.yml +++ b/deployment/docker/docker-compose.yml @@ -42,9 +42,8 @@ services: init: true expose: - "3000" - networks: - - default - - shared + ports: + - "${WEB_BIND_IP:-127.0.0.1}:${WEB_HOST_PORT:-3000}:3000" depends_on: api: condition: service_healthy @@ -66,9 +65,8 @@ services: command: ["services/api/src/index.ts"] expose: - "4000" - networks: - - default - - shared + ports: + - "${API_BIND_IP:-127.0.0.1}:${API_HOST_PORT:-4000}:4000" healthcheck: test: [ @@ -166,11 +164,6 @@ services: volumes: - nats-data:/data -networks: - shared: - external: true - name: ${NPM_SHARED_NETWORK:-npm-shared} - volumes: clickhouse-data: redis-data: diff --git a/deployment/npm/.env.example b/deployment/npm/.env.example deleted file mode 100644 index f8123eb..0000000 --- a/deployment/npm/.env.example +++ /dev/null @@ -1,13 +0,0 @@ -TZ=Etc/UTC -NPM_ADMIN_BIND_IP=100.87.130.79 -NPM_EDGE_NETWORK=nextcloud_edge -NPM_SHARED_NETWORK=npm-shared - -# Smart money refdata -SMART_MONEY_EVENT_CALENDAR_PATH=data/event-calendar.json -REFDATA_EVENT_CALENDAR_PATH= -REFDATA_EVENT_CALENDAR_PROVIDER= -ALPHA_VANTAGE_API_KEY= -ALPHA_VANTAGE_EARNINGS_HORIZON=3month -ALPHA_VANTAGE_EARNINGS_SYMBOL= -REFDATA_EVENT_CALENDAR_REFRESH_MS=86400000 diff --git a/deployment/npm/.gitignore b/deployment/npm/.gitignore deleted file mode 100644 index 383dbe5..0000000 --- a/deployment/npm/.gitignore +++ /dev/null @@ -1,3 +0,0 @@ -data/ -letsencrypt/ -.env diff --git a/deployment/npm/README.md b/deployment/npm/README.md deleted file mode 100644 index 38d4aa6..0000000 --- a/deployment/npm/README.md +++ /dev/null @@ -1,65 +0,0 @@ -# Nginx Proxy Manager - -This stack runs Nginx Proxy Manager separately from the Nextcloud stack while preserving the existing proxy host database and certificates. - -## Layout - -- `docker-compose.yml` defines the standalone NPM service. -- `.env` holds only stack-local settings like `TZ` and the admin bind IP. -- Runtime state lives in: - - `./data` - - `./letsencrypt` - -## Networks - -This stack joins the same external Docker networks that the current proxy hosts depend on: - -- `nextcloud_edge` for `nextcloud-app` and `portainer` -- `npm-shared` for Islandflow services like `web` and `api` - -Because the container name stays `nginx-proxy-manager`, the existing `proxy.deltaisland.io -> nginx-proxy-manager:81` host continues to work after migration. - -### Upstream alias collisions - -This NPM instance is attached to multiple Docker networks. If two stacks both expose a generic alias like `api` or `web`, Nginx can resolve the wrong upstream. - -For Islandflow hosts, prefer explicit upstream hostnames in NPM: - -- `islandflow-vps-web-1` on port `3000` -- `islandflow-vps-api-1` on port `4000` - -This avoids routing Islandflow traffic to similarly named containers from other stacks. - -## Migration - -1. Copy `.env.example` to `.env` and adjust values if needed. -2. Stop the old NPM service from `/home/delta/nextcloud`. -3. Copy the existing state directories into this stack: - -```bash -cp -rf /home/delta/nextcloud/npm/data /home/delta/islandflow/deployment/npm/ -cp -rf /home/delta/nextcloud/npm/letsencrypt /home/delta/islandflow/deployment/npm/ -``` - -4. Start the new stack: - -```bash -docker compose up -d -``` - -5. Verify the expected hosts still load: - -- `https://proxy.deltaisland.io` -- `https://portainer.deltaisland.io` -- `https://cloud.dpdrm.com` - -## Current Live Proxy Hosts - -- `cloud.dpdrm.com` -> `nextcloud-app:80` -- `portainer.deltaisland.io` -> `portainer:9000` -- `proxy.deltaisland.io` -> `nginx-proxy-manager:81` - -Islandflow-specific host mapping should use explicit upstream container names whenever possible: - -- `flow.deltaisland.io` -> `islandflow-vps-web-1:3000` -- `api.flow.deltaisland.io` -> `islandflow-vps-api-1:4000` diff --git a/deployment/npm/docker-compose.yml b/deployment/npm/docker-compose.yml deleted file mode 100644 index 4b7372d..0000000 --- a/deployment/npm/docker-compose.yml +++ /dev/null @@ -1,29 +0,0 @@ -name: nginx-proxy-manager - -services: - npm: - image: jc21/nginx-proxy-manager:2 - container_name: nginx-proxy-manager - restart: unless-stopped - ports: - - "80:80" - - "${NPM_ADMIN_BIND_IP:-100.87.130.79}:81:81" - - "443:443" - env_file: - - ./.env - environment: - TZ: ${TZ} - volumes: - - ./data:/data - - ./letsencrypt:/etc/letsencrypt - networks: - - edge - - shared - -networks: - edge: - external: true - name: ${NPM_EDGE_NETWORK:-nextcloud_edge} - shared: - external: true - name: ${NPM_SHARED_NETWORK:-npm-shared}