dirtydishes/islandflow
1
0
Fork 0
Code Issues 2 Pull requests 4 Projects Releases Packages Wiki Activity Actions 1

fix alpaca news auth and native worker wiring

Browse source
This commit is contained in:
dirtydishes 2026-05-19 19:57:56 -04:00
parent e9739f5dc9
commit 7d25608b35
21 changed files with 285 additions and 80 deletions
Download patch file Download diff file Expand all files Collapse all files

4
deployment/docker/.env.example
View file

@ -27,6 +27,10 @@ NEXT_PUBLIC_NBBO_MAX_AGE_MS=1000
# Options ingest
OPTIONS_INGEST_ADAPTER=synthetic
ALPACA_API_KEY=
ALPACA_API_KEY_ID=
ALPACA_KEY_ID=
ALPACA_API_SECRET_KEY=
ALPACA_SECRET_KEY=
ALPACA_REST_URL=https://data.alpaca.markets
ALPACA_WS_BASE_URL=wss://stream.data.alpaca.markets/v1beta1
ALPACA_FEED=indicative

8
deployment/docker/README.md
View file

@ -161,8 +161,10 @@ Set the adapter values and credentials in `.env`:
- `OPTIONS_INGEST_ADAPTER=alpaca`
- `EQUITIES_INGEST_ADAPTER=alpaca`
- `ALPACA_KEY_ID=...`
- `ALPACA_SECRET_KEY=...`
- `ALPACA_API_KEY_ID=...`
- `ALPACA_API_SECRET_KEY=...`
The older single-variable `ALPACA_API_KEY` fallback is still accepted for legacy setups, but Alpaca's current market-data auth expects a key ID plus secret key pair.
### Databento mode
@ -284,7 +286,7 @@ Scoped Docker deploys now build only the selected image set and then restart onl
- `--web-only`: `docker compose build web`, then `docker compose up -d web`
- `--api-only`: `docker compose build api`, then `docker compose up -d api`
- `--services-only`: builds and restarts `api`, `compute`, `candles`, `ingest-options`, and `ingest-equities`
- `--workers-only`: builds and restarts `compute`, `candles`, `ingest-options`, and `ingest-equities` without touching `web` or `api`
- `--workers-only`: builds and restarts `compute`, `candles`, `ingest-options`, `ingest-equities`, and `ingest-news` without touching `web` or `api`
- `--fast`: when no explicit scope flag is given, treats the deploy as `--services-only` and skips the public API route suite for quicker completion. It still runs remote service health checks.
Use `--no-build` only when the image is already correct and you need Compose to recreate or restart containers, such as after changing server-side environment values that do not affect a Next.js build-time variable. Do not use `--no-build` for dependency changes, application source changes, or `NEXT_PUBLIC_*` changes.

5
deployment/native/README.md
View file

@ -91,6 +91,7 @@ Checked-in unit files live under:
- `deployment/native/systemd/user/islandflow-candles.service`
- `deployment/native/systemd/user/islandflow-ingest-options.service`
- `deployment/native/systemd/user/islandflow-ingest-equities.service`
- `deployment/native/systemd/user/islandflow-ingest-news.service`
These are written for the current VPS layout:
@ -175,6 +176,7 @@ Default unit names used by `scripts/deploy.ts`:
- `islandflow-candles`
- `islandflow-ingest-options`
- `islandflow-ingest-equities`
- `islandflow-ingest-news`
Override them from your local shell before running `./deploy` if the server uses different names:
@ -191,6 +193,7 @@ Available overrides:
- `DEPLOY_NATIVE_CANDLES_UNIT`
- `DEPLOY_NATIVE_INGEST_OPTIONS_UNIT`
- `DEPLOY_NATIVE_INGEST_EQUITIES_UNIT`
- `DEPLOY_NATIVE_INGEST_NEWS_UNIT`
## systemctl invocation
@ -220,7 +223,7 @@ Scope behavior:
- `--web-only`: rebuild/restart only the web unit
- `--api-only`: restart only the API unit
- `--services-only`: restart API + worker units without touching the web unit
- `--workers-only`: restart only `compute`, `candles`, `ingest-options`, and `ingest-equities`
- `--workers-only`: restart only `compute`, `candles`, `ingest-options`, `ingest-equities`, and `ingest-news`
- `--fast`: when no explicit scope flag is provided, native deploys now default to `--workers-only`
- `--no-build`: skip `bun install --frozen-lockfile` and skip the web build step

6
deployment/native/check-native-health.sh
View file

@ -7,7 +7,7 @@ units=()
case "$scope" in
full)
units=(islandflow-web.service islandflow-api.service islandflow-compute.service islandflow-candles.service islandflow-ingest-options.service islandflow-ingest-equities.service)
units=(islandflow-web.service islandflow-api.service islandflow-compute.service islandflow-candles.service islandflow-ingest-options.service islandflow-ingest-equities.service islandflow-ingest-news.service)
;;
web)
units=(islandflow-web.service)
@ -16,10 +16,10 @@ case "$scope" in
units=(islandflow-api.service)
;;
services)
units=(islandflow-api.service islandflow-compute.service islandflow-candles.service islandflow-ingest-options.service islandflow-ingest-equities.service)
units=(islandflow-api.service islandflow-compute.service islandflow-candles.service islandflow-ingest-options.service islandflow-ingest-equities.service islandflow-ingest-news.service)
;;
workers)
units=(islandflow-compute.service islandflow-candles.service islandflow-ingest-options.service islandflow-ingest-equities.service)
units=(islandflow-compute.service islandflow-candles.service islandflow-ingest-options.service islandflow-ingest-equities.service islandflow-ingest-news.service)
;;
*)
echo "Unknown scope: $scope" >&2

8
deployment/native/cutover.sh
View file

@ -16,7 +16,7 @@ esac
echo "Stopping Docker-owned Islandflow app services before native ownership starts."
(
cd "$repo_root/deployment/docker"
docker compose stop web api compute candles ingest-options ingest-equities
docker compose stop web api compute candles ingest-options ingest-equities ingest-news
)
if [[ "$scope" == "full" || "$scope" == "services" || "$scope" == "api" || "$scope" == "web" ]]; then
@ -24,9 +24,9 @@ if [[ "$scope" == "full" || "$scope" == "services" || "$scope" == "api" || "$sco
fi
systemctl --user restart $(case "$scope" in
full) echo islandflow-web.service islandflow-api.service islandflow-compute.service islandflow-candles.service islandflow-ingest-options.service islandflow-ingest-equities.service ;;
services) echo islandflow-api.service islandflow-compute.service islandflow-candles.service islandflow-ingest-options.service islandflow-ingest-equities.service ;;
workers) echo islandflow-compute.service islandflow-candles.service islandflow-ingest-options.service islandflow-ingest-equities.service ;;
full) echo islandflow-web.service islandflow-api.service islandflow-compute.service islandflow-candles.service islandflow-ingest-options.service islandflow-ingest-equities.service islandflow-ingest-news.service ;;
services) echo islandflow-api.service islandflow-compute.service islandflow-candles.service islandflow-ingest-options.service islandflow-ingest-equities.service islandflow-ingest-news.service ;;
workers) echo islandflow-compute.service islandflow-candles.service islandflow-ingest-options.service islandflow-ingest-equities.service islandflow-ingest-news.service ;;
api) echo islandflow-api.service ;;
web) echo islandflow-web.service ;;
esac)

4
deployment/native/full-rollback.sh
View file

@ -4,7 +4,7 @@ set -euo pipefail
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
echo "Stopping native app services."
systemctl --user stop islandflow-web.service islandflow-api.service islandflow-compute.service islandflow-candles.service islandflow-ingest-options.service islandflow-ingest-equities.service || true
systemctl --user stop islandflow-web.service islandflow-api.service islandflow-compute.service islandflow-candles.service islandflow-ingest-options.service islandflow-ingest-equities.service islandflow-ingest-news.service || true
echo "Stopping native infra before Docker reopens durable data."
if [[ "${EUID}" -eq 0 ]]; then
@ -19,7 +19,7 @@ echo "Switching NPM Islandflow upstreams back to Docker service names."
echo "Restarting Docker Islandflow runtime."
(
cd "$repo_root/deployment/docker"
docker compose up -d web api compute candles ingest-options ingest-equities
docker compose up -d web api compute candles ingest-options ingest-equities ingest-news
)
curl -I -fksS "${DEPLOY_PUBLIC_APP_URL:-https://flow.deltaisland.io}" >/dev/null

8
deployment/native/install-user-units.sh
View file

@ -11,7 +11,7 @@ case "$scope" in
none)
;;
full)
units=(islandflow-web.service islandflow-api.service islandflow-compute.service islandflow-candles.service islandflow-ingest-options.service islandflow-ingest-equities.service)
units=(islandflow-web.service islandflow-api.service islandflow-compute.service islandflow-candles.service islandflow-ingest-options.service islandflow-ingest-equities.service islandflow-ingest-news.service)
;;
web)
units=(islandflow-web.service)
@ -20,10 +20,10 @@ case "$scope" in
units=(islandflow-api.service)
;;
services)
units=(islandflow-api.service islandflow-compute.service islandflow-candles.service islandflow-ingest-options.service islandflow-ingest-equities.service)
units=(islandflow-api.service islandflow-compute.service islandflow-candles.service islandflow-ingest-options.service islandflow-ingest-equities.service islandflow-ingest-news.service)
;;
workers)
units=(islandflow-compute.service islandflow-candles.service islandflow-ingest-options.service islandflow-ingest-equities.service)
units=(islandflow-compute.service islandflow-candles.service islandflow-ingest-options.service islandflow-ingest-equities.service islandflow-ingest-news.service)
;;
*)
echo "Unknown scope: $scope" >&2
@ -46,4 +46,4 @@ if [[ ${#units[@]} -gt 0 ]]; then
echo "Enabled scope: $scope"
else
echo "No units enabled yet. Pass a scope such as workers when you are ready."
fi
fi

6
deployment/native/rollback.sh
View file

@ -30,7 +30,7 @@ fi
case "$scope" in
full)
units=(islandflow-web.service islandflow-api.service islandflow-compute.service islandflow-candles.service islandflow-ingest-options.service islandflow-ingest-equities.service)
units=(islandflow-web.service islandflow-api.service islandflow-compute.service islandflow-candles.service islandflow-ingest-options.service islandflow-ingest-equities.service islandflow-ingest-news.service)
;;
web)
units=(islandflow-web.service)
@ -39,10 +39,10 @@ case "$scope" in
units=(islandflow-api.service)
;;
services)
units=(islandflow-api.service islandflow-compute.service islandflow-candles.service islandflow-ingest-options.service islandflow-ingest-equities.service)
units=(islandflow-api.service islandflow-compute.service islandflow-candles.service islandflow-ingest-options.service islandflow-ingest-equities.service islandflow-ingest-news.service)
;;
workers)
units=(islandflow-compute.service islandflow-candles.service islandflow-ingest-options.service islandflow-ingest-equities.service)
units=(islandflow-compute.service islandflow-candles.service islandflow-ingest-options.service islandflow-ingest-equities.service islandflow-ingest-news.service)
;;
*)
echo "Unknown scope: $scope" >&2

17
deployment/native/systemd/user/islandflow-ingest-news.service Normal file
View file

@ -0,0 +1,17 @@
[Unit]
Description=Islandflow ingest-news
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
WorkingDirectory=/home/delta/islandflow
EnvironmentFile=/home/delta/islandflow/.env
ExecStart=/home/delta/.bun/bin/bun services/ingest-news/src/index.ts
Restart=always
RestartSec=2
KillSignal=SIGINT
TimeoutStopSec=20
[Install]
WantedBy=default.target