From 23ed3809cc111155e5b1dba4e387f3e8b62b630b Mon Sep 17 00:00:00 2001 From: dirtydishes Date: Sat, 16 May 2026 17:54:00 -0400 Subject: [PATCH 1/3] speed up docker deploy builds --- .beads/issues.jsonl | 2 + deployment/docker/Dockerfile.ingest-options | 38 ++- deployment/docker/Dockerfile.service | 26 ++- deployment/docker/Dockerfile.web | 25 +- deployment/docker/README.md | 35 ++- ...26-05-16-1752-speed-up-docker-deploys.html | 219 ++++++++++++++++++ scripts/deploy.ts | 26 ++- 7 files changed, 349 insertions(+), 22 deletions(-) create mode 100644 docs/turns/2026-05-16-1752-speed-up-docker-deploys.html diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index 605077e..1ac2304 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -11,6 +11,7 @@ {"_type":"issue","id":"islandflow-ayo","title":"Drop stale backlog events from live fanout","description":"Follow-up to live freshness rollout: /ws/live was still fanning out stale backlog events for freshness-gated channels, which kept tape panes in Live feed behind despite active synthetic ingest. Gate fanout and cache ingest by freshness for options/nbbo/equities/flow.","status":"closed","priority":1,"issue_type":"bug","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T21:26:39Z","created_by":"dirtydishes","updated_at":"2026-04-28T21:26:44Z","started_at":"2026-04-28T21:26:44Z","closed_at":"2026-04-28T21:26:44Z","close_reason":"Completed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-0v6","title":"Fix tape freshness, NBBO coverage, pause controls, and filter popup","description":"Implement the tape fixes requested for synthetic options notional sizing, strict live freshness, live-mode pause/resume behavior, stronger NBBO snapshot coverage, and moving flow filters behind a popup. Includes server-side live cache changes, web terminal state/UI changes, and tests for synthetic pricing, live snapshot freshness/NBBO retention, and live pause/filter interactions.","status":"closed","priority":1,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T21:02:52Z","created_by":"dirtydishes","updated_at":"2026-04-28T21:13:38Z","started_at":"2026-04-28T21:02:57Z","closed_at":"2026-04-28T21:13:38Z","close_reason":"Completed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-e4r","title":"Implement smart-money flow filtering and synthetic firehose modes","description":"Implement the approved multi-surface plan for named synthetic market profiles, options raw-vs-signal filtering, live/API filter contracts, Tape page client-side flow filters, firehose-readiness improvements, tests, and README updates.","status":"closed","priority":1,"issue_type":"feature","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T20:10:49Z","created_by":"dirtydishes","updated_at":"2026-04-28T20:29:29Z","started_at":"2026-04-28T20:10:53Z","closed_at":"2026-04-28T20:29:29Z","close_reason":"Implemented synthetic market profiles, options signal-path filtering, signal-aware API/replay contracts, Tape page filters, tests, and README updates. Follow-up tracked in islandflow-biq.","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"islandflow-09a","title":"Speed up Docker deployment builds","description":"Implement the Docker deployment optimization plan from /Users/kell/Desktop/speed-up-docker.md: split dependency installation from source copy, add BuildKit caches, make scoped deploys build only their target services, update Docker deployment docs, validate, document the turn, commit, and push.","status":"closed","priority":2,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-16T21:50:24Z","created_by":"dirtydishes","updated_at":"2026-05-16T21:53:48Z","started_at":"2026-05-16T21:50:37Z","closed_at":"2026-05-16T21:53:48Z","close_reason":"Implemented Docker dependency-layer caching, scoped deploy build/up flow, Docker docs updates, validation, and turn documentation. Follow-up islandflow-cnk tracks daemon-backed image build verification.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-0sa","title":"Fix live tape auto-hold, history seam, and remove manual pause control","description":"The live tape should automatically hold when the user scrolls away from the top, resume when they return to the top or use Jump to top, and keep older prints available seamlessly beyond the hot window. Manual Pause/Resume control is now redundant and should be removed from live tape panes. This work should also fix the current regression where paused/held tapes still mutate, and align the options tape with a strict 100-row hot head backed by ClickHouse history.","notes":"Implemented live scroll-hold with no live pause button, demand-loaded ClickHouse history, a 100-row options hot head, and cache-first scoped snapshots. Validated with bun test apps/web/app/terminal.test.ts services/api/tests/live.test.ts and bun --cwd=apps/web run build.","status":"closed","priority":2,"issue_type":"bug","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-16T18:12:51Z","created_by":"dirtydishes","updated_at":"2026-05-16T18:23:43Z","started_at":"2026-05-16T18:12:54Z","closed_at":"2026-05-16T18:23:43Z","close_reason":"Closed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-2db","title":"Manually remove stale islandflow local-infra containers from VPS","description":"The live VPS still has an older compose project named islandflow created from the repo-root docker-compose.yml. Inspection shows it is separate from the supported islandflow-vps deployment stack and exposes NATS, ClickHouse, and Redis on host ports. Container removal commands currently hang when run as the delta user through Docker, so cleanup likely needs a focused maintenance window and possibly host-level intervention or a Docker daemon restart.","notes":"The duplicate islandflow compose project on the VPS was confirmed live during inspection. Nginx Proxy Manager routes public traffic only to islandflow-vps web/api by Docker name, so the stale islandflow project appears to be stray local-infra state rather than part of the supported production path. Attempts to remove the stale containers with docker compose down and docker rm -f as the delta user hung and timed out, so manual cleanup likely needs a maintenance window and possibly Docker daemon intervention.","status":"open","priority":2,"issue_type":"task","owner":"dishes@dpdrm.com","created_at":"2026-05-16T01:27:27Z","created_by":"dirtydishes","updated_at":"2026-05-16T01:28:59Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-c87","title":"Clean up duplicate Islandflow Docker infra on VPS","description":"The live VPS is currently running both the production-style islandflow-vps Docker stack and an older root-level islandflow infra stack that publishes NATS, ClickHouse, and Redis on host ports. Investigate whether the older stack is unused, remove it safely if so, and update docs/deploy guidance so the server topology is clearer.","notes":"Inspected the live VPS and confirmed the duplicate compose project: islandflow-vps is the supported deployment stack, while a separate islandflow project from the repo-root docker-compose.yml still runs exposed NATS/ClickHouse/Redis containers. Verified Nginx Proxy Manager routes only to islandflow-vps web/api by Docker name. Attempted cleanup via docker compose down and docker rm -f on the stale islandflow containers, but those commands hung for the delta user and timed out. Added repo guardrails and docs so deploy warns when the duplicate project exists, and opened islandflow-2db for manual host-level cleanup during a maintenance window.","status":"closed","priority":2,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-16T01:16:05Z","created_by":"dirtydishes","updated_at":"2026-05-16T01:28:07Z","started_at":"2026-05-16T01:16:09Z","closed_at":"2026-05-16T01:28:07Z","close_reason":"Completed the repo-side investigation and guardrails. Actual server-side container removal is blocked by hanging Docker operations and is tracked separately in islandflow-2db for a maintenance window.","dependency_count":0,"dependent_count":0,"comment_count":0} @@ -38,5 +39,6 @@ {"_type":"issue","id":"islandflow-zs0","title":"Migrate terminal UI to smart-money profiles","description":"Migrate apps/web terminal rendering to consume SmartMoneyEvent directly: primary profile, probability ladder, reason codes, and suppression/abstention state, while preserving legacy alert/classifier displays during the bridge.","status":"closed","priority":2,"issue_type":"task","owner":"dishes@dpdrm.com","created_at":"2026-05-04T21:35:23Z","created_by":"dirtydishes","updated_at":"2026-05-05T05:39:58Z","closed_at":"2026-05-05T05:39:58Z","close_reason":"Completed terminal smart-money profile migration","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-igk","title":"Add plan mode","description":"Implement a user-facing plan mode in the application so users can switch into planning before taking action. Scope to be clarified from existing app patterns.","status":"closed","priority":2,"issue_type":"feature","owner":"dishes@dpdrm.com","created_at":"2026-05-04T04:22:37Z","created_by":"dirtydishes","updated_at":"2026-05-04T04:26:18Z","started_at":"2026-05-04T04:22:40Z","closed_at":"2026-05-04T04:26:18Z","close_reason":"Implemented as a global pi extension toggled with Shift+P","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-biq","title":"Finish raw live options delivery and filter/backpressure observability","description":"The smart-money signal path and Tape filters are in place, but the next firehose pass should finish server-side selective raw live delivery for options subscriptions and add explicit filtered-out/backpressure observability for API/web counters. This was discovered while landing islandflow-e4r.\n","status":"in_progress","priority":2,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T20:28:58Z","created_by":"dirtydishes","updated_at":"2026-04-29T03:54:12Z","started_at":"2026-04-29T03:54:12Z","dependencies":[{"issue_id":"islandflow-biq","depends_on_id":"islandflow-e4r","type":"discovered-from","created_at":"2026-04-28T16:28:58Z","created_by":"dirtydishes","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"islandflow-cnk","title":"Run Docker image build verification with active Docker daemon","description":"Targeted image builds could not run in the implementation session because the local Docker daemon was unavailable at unix:///Users/kell/.orbstack/run/docker.sock. When Docker or OrbStack is running, validate the refactored deployment Dockerfiles with: docker compose -f deployment/docker/docker-compose.yml build api; docker compose -f deployment/docker/docker-compose.yml build web; docker compose -f deployment/docker/docker-compose.yml build ingest-options.","status":"open","priority":3,"issue_type":"task","owner":"dishes@dpdrm.com","created_at":"2026-05-16T21:53:41Z","created_by":"dirtydishes","updated_at":"2026-05-16T21:53:41Z","dependencies":[{"issue_id":"islandflow-cnk","depends_on_id":"islandflow-09a","type":"discovered-from","created_at":"2026-05-16T17:53:40Z","created_by":"dirtydishes","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-38p","title":"Add native deployment unit templates and rollback helpers","description":"The deploy helper now supports --runtime native, but the repo still relies on operator-managed systemd units and manual rollback. Add checked-in native deployment templates or provisioning guidance for the expected units, and consider lightweight rollback/smoke-test helpers once the host-native path is exercised on the real VPS.","status":"open","priority":3,"issue_type":"task","owner":"dishes@dpdrm.com","created_at":"2026-05-15T23:46:42Z","created_by":"dirtydishes","updated_at":"2026-05-15T23:46:42Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-575","title":"Document smart-money event calendar env","description":"Document smart-money event-calendar environment configuration in env examples and README.\n","status":"closed","priority":3,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-05T06:57:14Z","created_by":"dirtydishes","updated_at":"2026-05-05T06:57:57Z","started_at":"2026-05-05T06:57:17Z","closed_at":"2026-05-05T06:57:57Z","close_reason":"Documented event-calendar env variables","dependency_count":0,"dependent_count":0,"comment_count":0} diff --git a/deployment/docker/Dockerfile.ingest-options b/deployment/docker/Dockerfile.ingest-options index 156dc1d..52cba59 100644 --- a/deployment/docker/Dockerfile.ingest-options +++ b/deployment/docker/Dockerfile.ingest-options @@ -1,3 +1,5 @@ +# syntax=docker/dockerfile:1.7 + FROM oven/bun:1.3.11 WORKDIR /app @@ -9,15 +11,39 @@ ENV PATH="${VIRTUAL_ENV}/bin:${PATH}" COPY --from=workspace package.json ./package.json COPY --from=workspace bun.lock ./bun.lock COPY --from=workspace tsconfig.base.json ./tsconfig.base.json -COPY --from=services . ./services -COPY --from=packages . ./packages -COPY --from=apps . ./apps RUN apt-get update \ && apt-get install -y --no-install-recommends python3 python3-pip python3-venv \ && rm -rf /var/lib/apt/lists/* \ - && python3 -m venv "${VIRTUAL_ENV}" \ - && "${VIRTUAL_ENV}/bin/pip" install --no-cache-dir -r services/ingest-options/py/requirements.txt \ - && bun install --frozen-lockfile + && python3 -m venv "${VIRTUAL_ENV}" + +COPY --from=apps desktop/package.json ./apps/desktop/package.json +COPY --from=apps web/package.json ./apps/web/package.json + +COPY --from=packages bus/package.json ./packages/bus/package.json +COPY --from=packages config/package.json ./packages/config/package.json +COPY --from=packages observability/package.json ./packages/observability/package.json +COPY --from=packages storage/package.json ./packages/storage/package.json +COPY --from=packages types/package.json ./packages/types/package.json + +COPY --from=services api/package.json ./services/api/package.json +COPY --from=services candles/package.json ./services/candles/package.json +COPY --from=services compute/package.json ./services/compute/package.json +COPY --from=services eod-enricher/package.json ./services/eod-enricher/package.json +COPY --from=services ingest-equities/package.json ./services/ingest-equities/package.json +COPY --from=services ingest-options/package.json ./services/ingest-options/package.json +COPY --from=services ingest-options/py/requirements.txt ./services/ingest-options/py/requirements.txt +COPY --from=services refdata/package.json ./services/refdata/package.json +COPY --from=services replay/package.json ./services/replay/package.json + +RUN --mount=type=cache,target=/root/.cache/pip \ + "${VIRTUAL_ENV}/bin/pip" install -r services/ingest-options/py/requirements.txt + +RUN --mount=type=cache,target=/root/.bun/install/cache \ + bun install --frozen-lockfile + +COPY --from=services . ./services +COPY --from=packages . ./packages +COPY --from=apps . ./apps ENTRYPOINT ["bun"] diff --git a/deployment/docker/Dockerfile.service b/deployment/docker/Dockerfile.service index bc48d2d..e0fcf72 100644 --- a/deployment/docker/Dockerfile.service +++ b/deployment/docker/Dockerfile.service @@ -1,3 +1,5 @@ +# syntax=docker/dockerfile:1.7 + FROM oven/bun:1.3.11 WORKDIR /app @@ -7,10 +9,30 @@ ENV NODE_ENV=production COPY --from=workspace package.json ./package.json COPY --from=workspace bun.lock ./bun.lock COPY --from=workspace tsconfig.base.json ./tsconfig.base.json + +COPY --from=apps desktop/package.json ./apps/desktop/package.json +COPY --from=apps web/package.json ./apps/web/package.json + +COPY --from=packages bus/package.json ./packages/bus/package.json +COPY --from=packages config/package.json ./packages/config/package.json +COPY --from=packages observability/package.json ./packages/observability/package.json +COPY --from=packages storage/package.json ./packages/storage/package.json +COPY --from=packages types/package.json ./packages/types/package.json + +COPY --from=services api/package.json ./services/api/package.json +COPY --from=services candles/package.json ./services/candles/package.json +COPY --from=services compute/package.json ./services/compute/package.json +COPY --from=services eod-enricher/package.json ./services/eod-enricher/package.json +COPY --from=services ingest-equities/package.json ./services/ingest-equities/package.json +COPY --from=services ingest-options/package.json ./services/ingest-options/package.json +COPY --from=services refdata/package.json ./services/refdata/package.json +COPY --from=services replay/package.json ./services/replay/package.json + +RUN --mount=type=cache,target=/root/.bun/install/cache \ + bun install --frozen-lockfile + COPY --from=services . ./services COPY --from=packages . ./packages COPY --from=apps . ./apps -RUN bun install --frozen-lockfile - ENTRYPOINT ["bun"] diff --git a/deployment/docker/Dockerfile.web b/deployment/docker/Dockerfile.web index 6956335..33723ae 100644 --- a/deployment/docker/Dockerfile.web +++ b/deployment/docker/Dockerfile.web @@ -1,3 +1,5 @@ +# syntax=docker/dockerfile:1.7 + FROM oven/bun:1.3.11 AS build WORKDIR /app @@ -13,11 +15,32 @@ ENV NEXT_PUBLIC_NBBO_MAX_AGE_MS=${NEXT_PUBLIC_NBBO_MAX_AGE_MS} COPY --from=workspace package.json ./package.json COPY --from=workspace bun.lock ./bun.lock COPY --from=workspace tsconfig.base.json ./tsconfig.base.json + +COPY --from=apps desktop/package.json ./apps/desktop/package.json +COPY --from=apps web/package.json ./apps/web/package.json + +COPY --from=packages bus/package.json ./packages/bus/package.json +COPY --from=packages config/package.json ./packages/config/package.json +COPY --from=packages observability/package.json ./packages/observability/package.json +COPY --from=packages storage/package.json ./packages/storage/package.json +COPY --from=packages types/package.json ./packages/types/package.json + +COPY --from=services api/package.json ./services/api/package.json +COPY --from=services candles/package.json ./services/candles/package.json +COPY --from=services compute/package.json ./services/compute/package.json +COPY --from=services eod-enricher/package.json ./services/eod-enricher/package.json +COPY --from=services ingest-equities/package.json ./services/ingest-equities/package.json +COPY --from=services ingest-options/package.json ./services/ingest-options/package.json +COPY --from=services refdata/package.json ./services/refdata/package.json +COPY --from=services replay/package.json ./services/replay/package.json + +RUN --mount=type=cache,target=/root/.bun/install/cache \ + bun install --frozen-lockfile + COPY --from=services . ./services COPY --from=packages . ./packages COPY --from=apps . ./apps -RUN bun install --frozen-lockfile RUN bun run --cwd apps/web build FROM oven/bun:1.3.11 AS runtime diff --git a/deployment/docker/README.md b/deployment/docker/README.md index 7c4f03b..4a5019f 100644 --- a/deployment/docker/README.md +++ b/deployment/docker/README.md @@ -65,14 +65,16 @@ Important defaults: 3. Build and start the stack: ```bash -docker compose up -d --build +docker compose build api web compute candles ingest-options ingest-equities +docker compose up -d ``` If you are updating an existing deployment that already has failing `api` restart loops, do a full recreate so the ClickHouse config mount and dependency changes are applied cleanly: ```bash docker compose down -docker compose up -d --build --force-recreate +docker compose build api web compute candles ingest-options ingest-equities +docker compose up -d --force-recreate ``` 4. Confirm the containers are healthy: @@ -190,6 +192,19 @@ cd deployment/docker docker compose build web ``` +### Faster Docker builds + +The app images are structured so dependency installation is isolated from source code changes: + +- Docker first copies `package.json`, `bun.lock`, `tsconfig.base.json`, and workspace `package.json` files. +- `bun install --frozen-lockfile` runs in a cacheable layer with a BuildKit Bun cache mount. +- Source from `apps`, `services`, and `packages` is copied only after dependencies are installed. +- `ingest-options` also installs its Python sidecar dependencies from `services/ingest-options/py/requirements.txt` before source copy, using a BuildKit pip cache mount. + +That means normal TypeScript edits should reuse dependency layers. The first build after a fresh server checkout, Docker cache cleanup, dependency change, or Python requirement change can still be slow; later deploys should spend their time on changed source and the specific service images being rolled out. + +BuildKit cache mounts require a modern Docker Engine with Dockerfile frontend support. Docker Compose v2 on the VPS path enables this by default. + ## Safe rollouts on `152.53.80.229` The current live VPS uses Nginx Proxy Manager on the shared Docker network and routes public traffic to the Docker `web` and `api` containers by container name. Because of that, this Docker path remains the operationally correct default for the live server today. @@ -218,7 +233,7 @@ This flow: - checks the server checkout before switching anything - stops if the server has tracked local modifications - allows the known untracked tarball at `deployment/docker/signal-cli-0.14.3-Linux-native.tar.gz` -- runs `git switch main`, `git pull --ff-only origin main`, and `docker compose up -d --build` +- runs `git switch main`, `git pull --ff-only origin main`, `docker compose build api web compute candles ingest-options ingest-equities`, and `docker compose up -d` - verifies the stack with `docker compose ps`, recent service logs, container-local health checks, and public HTTPS checks ### Deploy the current local branch @@ -253,6 +268,14 @@ Examples: ./deploy main --runtime docker --web-only --no-build ``` +Scoped Docker deploys now build only the selected image set and then restart only those services: + +- `--web-only`: `docker compose build web`, then `docker compose up -d web` +- `--api-only`: `docker compose build api`, then `docker compose up -d api` +- `--services-only`: builds and restarts `api`, `compute`, `candles`, `ingest-options`, and `ingest-equities` + +Use `--no-build` only when the image is already correct and you need Compose to recreate or restart containers, such as after changing server-side environment values that do not affect a Next.js build-time variable. Do not use `--no-build` for dependency changes, application source changes, or `NEXT_PUBLIC_*` changes. + ### Escalation path Use force recreate only when a normal refresh does not update the services cleanly: @@ -299,7 +322,8 @@ git switch main git pull --ff-only origin main cd /home/delta/islandflow/deployment/docker -docker compose up -d --build +docker compose build api web compute candles ingest-options ingest-equities +docker compose up -d ``` Deploy the current branch manually: @@ -314,7 +338,8 @@ git switch || git switch -c --track origin/ cd /home/delta/islandflow/deployment/docker -docker compose up -d --build +docker compose build api web compute candles ingest-options ingest-equities +docker compose up -d ``` If you changed only env values for the Bun services on the server: diff --git a/docs/turns/2026-05-16-1752-speed-up-docker-deploys.html b/docs/turns/2026-05-16-1752-speed-up-docker-deploys.html new file mode 100644 index 0000000..df16d62 --- /dev/null +++ b/docs/turns/2026-05-16-1752-speed-up-docker-deploys.html @@ -0,0 +1,219 @@ + + + + + + Speed Up Docker Deploys + + + +
+
+
2026-05-16 17:52 America/New_York
+

Speed Up Docker Deploys

+

+ Summary + Docker app images now cache dependency installation separately from source changes, and Docker rollouts now build only the images required by the selected deploy scope before restarting containers. +

+
+ +
+

Summary

+

+ Implemented the Docker deployment speed-up plan from /Users/kell/Desktop/speed-up-docker.md. The first build after this change may still be slow, but source-only changes should no longer invalidate the expensive Bun and Python dependency layers. +

+
+ +
+

Changes Made

+
    +
  • Refactored deployment/docker/Dockerfile.service to copy workspace manifests, run cached bun install --frozen-lockfile, then copy source.
    • +
  • Applied the same dependency-first build model to deployment/docker/Dockerfile.web, keeping the Next.js build after source copy.
    • +
  • Updated deployment/docker/Dockerfile.ingest-options with separate cached pip and Bun install layers before copying source.
    • +
  • Changed scripts/deploy.ts so Docker rollouts run explicit docker compose build <services> followed by docker compose up -d <services>.
    • +
  • Documented the faster-build model, scoped rollouts, and appropriate --no-build usage in deployment/docker/README.md.
    • +
+
+ +
+

Context

+

+ The previous Dockerfiles copied all app, service, and package source before dependency installation. That made nearly every code change invalidate bun install, increasing VPS deploy time. The deployment helper also used broad up -d --build behavior rather than a clean build phase scoped to the selected service set. +

+
+ +
+

Important Implementation Details

+

+ Each app image now copies root deployment manifests plus every workspace package.json before installing dependencies. The source tree is copied only after the install layer is complete. +

+ 
RUN --mount=type=cache,target=/root/.bun/install/cache \
+  bun install --frozen-lockfile
+

+ The ingest-options image also copies services/ingest-options/py/requirements.txt before source and uses a pip cache mount: +

+ 
RUN --mount=type=cache,target=/root/.cache/pip \
+  "${VIRTUAL_ENV}/bin/pip" install -r services/ingest-options/py/requirements.txt
+

+ For full Docker deploys, the helper builds the six core app services explicitly. For scoped deploys, it builds and restarts only the requested services. +

+
+ +
+

Expected Impact for End-Users

+

+ Users should see faster deployment turnaround after ordinary source edits because dependency installation is reused when manifests and locks have not changed. Scoped deploys should also disturb fewer containers, reducing restart surface for web-only, API-only, and backend-only updates. +

+
+ +
+

Validation

+
    +
  • Passed: bun run check:docker-workspace
    • +
  • Passed: ./deploy --help
    • +
  • Passed: docker compose -f deployment/docker/docker-compose.yml config --quiet with a temporary copy of .env.example
    • +
  • Passed: bun --cwd=apps/web run build
    • +
  • Passed: bun test with 222 passing tests
    • +
  • Not run: targeted Docker image builds because this session could not connect to the Docker daemon at unix:///Users/kell/.orbstack/run/docker.sock.
    • +
+
+ +
+

Issues, Limitations, and Mitigations

+

+ Docker daemon access was unavailable locally, so image builds still need to be exercised on a machine with a running Docker daemon or during the next VPS rollout. Static Compose validation and repo test coverage passed, and the Dockerfiles use standard BuildKit cache mounts supported by modern Docker Compose v2. +

+
+ +
+

Follow-up Work

+

+ No separate follow-up issue was created. The remaining verification is operational: run the targeted image builds once Docker or OrbStack is available. +

+
+
+ + diff --git a/scripts/deploy.ts b/scripts/deploy.ts index 1ec3e6c..d6adcb1 100644 --- a/scripts/deploy.ts +++ b/scripts/deploy.ts @@ -324,6 +324,15 @@ function dockerServicesForScope(scope: DeployScope): string[] { } } +function dockerBuildServicesForScope(scope: DeployScope): string[] { + switch (scope) { + case "full": + return [...DOCKER_CORE_SERVICES]; + default: + return dockerServicesForScope(scope); + } +} + function dockerLogServicesForScope(scope: DeployScope): string[] { switch (scope) { case "web": @@ -565,15 +574,16 @@ function remoteDockerRollout( forceRecreate: boolean, noBuild: boolean ): void { - const services = dockerServicesForScope(scope); - const args = ["up", "-d"]; - if (!noBuild) { - args.push("--build"); - } + const rolloutServices = dockerServicesForScope(scope); + const upArgs = ["up", "-d"]; if (forceRecreate) { - args.push("--force-recreate"); + upArgs.push("--force-recreate"); } - const command = `docker compose ${[...args, ...services].join(" ")}`; + const buildServices = dockerBuildServicesForScope(scope); + const buildCommand = noBuild + ? null + : `docker compose build ${buildServices.join(" ")}`; + const upCommand = `docker compose ${[...upArgs, ...rolloutServices].join(" ")}`; runRemoteScript( "Remote Rollout", @@ -583,7 +593,7 @@ set -euo pipefail ${remoteGitUpdateScript(mode, branch)} cd ${shellEscape(REMOTE_DOCKER_DEPLOYMENT)} -${command} +${buildCommand ? `${buildCommand}\n` : ""}${upCommand} ` ); } From 1424a2716fc7d53863bf3df36428464301406bac Mon Sep 17 00:00:00 2001 From: dirtydishes Date: Sat, 16 May 2026 22:00:21 -0400 Subject: [PATCH 2/3] fix durable options history routing --- .beads/issues.jsonl | 2 + apps/web/app/globals.css | 11 + apps/web/app/terminal.tsx | 12 ++ deployment/docker/README.md | 9 +- ...9-fix-durable-options-history-routing.html | 195 ++++++++++++++++++ package.json | 1 + scripts/check-public-api-routes.ts | 41 ++++ scripts/deploy.ts | 4 +- 8 files changed, 271 insertions(+), 4 deletions(-) create mode 100644 docs/turns/2026-05-16-2159-fix-durable-options-history-routing.html create mode 100644 scripts/check-public-api-routes.ts diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index 1ac2304..2bf9d72 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -1,3 +1,4 @@ +{"_type":"issue","id":"islandflow-qso","title":"Fix durable options tape history routing","description":"Implement the fix-tape plan: make same-origin history routing durable, add deployment/public smoke checks for required API routes, expose tape history loading failures in the UI, document the work, and track api.flow.deltaisland.io migration separately.","status":"closed","priority":1,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-17T01:53:22Z","created_by":"dirtydishes","updated_at":"2026-05-17T02:00:04Z","started_at":"2026-05-17T01:53:25Z","closed_at":"2026-05-17T02:00:04Z","close_reason":"Implemented durable same-origin history routing, public route smoke checks, tape history diagnostics, docs, validation, and follow-up tracking for api.flow.deltaisland.io.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-200","title":"Implement durable options tape history","description":"Implement the plan from docs/plans/2026-05-16-1711-durable-options-tape-history.html: durable ClickHouse-backed options history, signal/all prints view selection, preserved execution context, stale semantics limited to live health, reset runbook, tests, and turn documentation.","status":"closed","priority":1,"issue_type":"feature","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-16T21:21:30Z","created_by":"dirtydishes","updated_at":"2026-05-16T21:26:51Z","started_at":"2026-05-16T21:21:33Z","closed_at":"2026-05-16T21:26:51Z","close_reason":"Implemented durable options tape history, signal/raw view selection, reset runbook, tests, and turn documentation.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-k4f","title":"Gate deploy script on docker workspace snapshot sync","description":"Prevent frozen-lockfile build failures during deploy by adding a local preflight in scripts/deploy.ts that runs bun run check:docker-workspace and aborts with a clear sync+commit remediation message when stale.","status":"closed","priority":1,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-15T23:01:44Z","created_by":"dirtydishes","updated_at":"2026-05-15T23:04:11Z","started_at":"2026-05-15T23:01:48Z","closed_at":"2026-05-15T23:04:11Z","close_reason":"Closed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-xll","title":"Fix bun.lock drift causing frozen-lockfile Docker build failures","description":"Docker image builds fail in multiple targets (candles, web, ingest services) because bun install --frozen-lockfile detects lockfile changes. Update workspace lockfile to match manifests and verify frozen install succeeds.","status":"closed","priority":1,"issue_type":"bug","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-15T22:52:38Z","created_by":"dirtydishes","updated_at":"2026-05-15T22:55:23Z","started_at":"2026-05-15T22:52:40Z","closed_at":"2026-05-15T22:55:23Z","close_reason":"Closed","dependency_count":0,"dependent_count":0,"comment_count":0} @@ -11,6 +12,7 @@ {"_type":"issue","id":"islandflow-ayo","title":"Drop stale backlog events from live fanout","description":"Follow-up to live freshness rollout: /ws/live was still fanning out stale backlog events for freshness-gated channels, which kept tape panes in Live feed behind despite active synthetic ingest. Gate fanout and cache ingest by freshness for options/nbbo/equities/flow.","status":"closed","priority":1,"issue_type":"bug","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T21:26:39Z","created_by":"dirtydishes","updated_at":"2026-04-28T21:26:44Z","started_at":"2026-04-28T21:26:44Z","closed_at":"2026-04-28T21:26:44Z","close_reason":"Completed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-0v6","title":"Fix tape freshness, NBBO coverage, pause controls, and filter popup","description":"Implement the tape fixes requested for synthetic options notional sizing, strict live freshness, live-mode pause/resume behavior, stronger NBBO snapshot coverage, and moving flow filters behind a popup. Includes server-side live cache changes, web terminal state/UI changes, and tests for synthetic pricing, live snapshot freshness/NBBO retention, and live pause/filter interactions.","status":"closed","priority":1,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T21:02:52Z","created_by":"dirtydishes","updated_at":"2026-04-28T21:13:38Z","started_at":"2026-04-28T21:02:57Z","closed_at":"2026-04-28T21:13:38Z","close_reason":"Completed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-e4r","title":"Implement smart-money flow filtering and synthetic firehose modes","description":"Implement the approved multi-surface plan for named synthetic market profiles, options raw-vs-signal filtering, live/API filter contracts, Tape page client-side flow filters, firehose-readiness improvements, tests, and README updates.","status":"closed","priority":1,"issue_type":"feature","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-04-28T20:10:49Z","created_by":"dirtydishes","updated_at":"2026-04-28T20:29:29Z","started_at":"2026-04-28T20:10:53Z","closed_at":"2026-04-28T20:29:29Z","close_reason":"Implemented synthetic market profiles, options signal-path filtering, signal-aware API/replay contracts, Tape page filters, tests, and README updates. Follow-up tracked in islandflow-biq.","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"islandflow-qd7","title":"Migrate production web to api.flow.deltaisland.io","description":"Follow-up from the durable options tape history fix. Plan and migrate production from same-origin API path proxying on flow.deltaisland.io to a dedicated api.flow.deltaisland.io origin, including DNS, proxy config, CORS/websocket behavior, deployment docs, and public smoke checks.","status":"open","priority":2,"issue_type":"task","owner":"dishes@dpdrm.com","created_at":"2026-05-17T01:55:02Z","created_by":"dirtydishes","updated_at":"2026-05-17T01:55:02Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-09a","title":"Speed up Docker deployment builds","description":"Implement the Docker deployment optimization plan from /Users/kell/Desktop/speed-up-docker.md: split dependency installation from source copy, add BuildKit caches, make scoped deploys build only their target services, update Docker deployment docs, validate, document the turn, commit, and push.","status":"closed","priority":2,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-16T21:50:24Z","created_by":"dirtydishes","updated_at":"2026-05-16T21:53:48Z","started_at":"2026-05-16T21:50:37Z","closed_at":"2026-05-16T21:53:48Z","close_reason":"Implemented Docker dependency-layer caching, scoped deploy build/up flow, Docker docs updates, validation, and turn documentation. Follow-up islandflow-cnk tracks daemon-backed image build verification.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-0sa","title":"Fix live tape auto-hold, history seam, and remove manual pause control","description":"The live tape should automatically hold when the user scrolls away from the top, resume when they return to the top or use Jump to top, and keep older prints available seamlessly beyond the hot window. Manual Pause/Resume control is now redundant and should be removed from live tape panes. This work should also fix the current regression where paused/held tapes still mutate, and align the options tape with a strict 100-row hot head backed by ClickHouse history.","notes":"Implemented live scroll-hold with no live pause button, demand-loaded ClickHouse history, a 100-row options hot head, and cache-first scoped snapshots. Validated with bun test apps/web/app/terminal.test.ts services/api/tests/live.test.ts and bun --cwd=apps/web run build.","status":"closed","priority":2,"issue_type":"bug","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-16T18:12:51Z","created_by":"dirtydishes","updated_at":"2026-05-16T18:23:43Z","started_at":"2026-05-16T18:12:54Z","closed_at":"2026-05-16T18:23:43Z","close_reason":"Closed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-2db","title":"Manually remove stale islandflow local-infra containers from VPS","description":"The live VPS still has an older compose project named islandflow created from the repo-root docker-compose.yml. Inspection shows it is separate from the supported islandflow-vps deployment stack and exposes NATS, ClickHouse, and Redis on host ports. Container removal commands currently hang when run as the delta user through Docker, so cleanup likely needs a focused maintenance window and possibly host-level intervention or a Docker daemon restart.","notes":"The duplicate islandflow compose project on the VPS was confirmed live during inspection. Nginx Proxy Manager routes public traffic only to islandflow-vps web/api by Docker name, so the stale islandflow project appears to be stray local-infra state rather than part of the supported production path. Attempts to remove the stale containers with docker compose down and docker rm -f as the delta user hung and timed out, so manual cleanup likely needs a maintenance window and possibly Docker daemon intervention.","status":"open","priority":2,"issue_type":"task","owner":"dishes@dpdrm.com","created_at":"2026-05-16T01:27:27Z","created_by":"dirtydishes","updated_at":"2026-05-16T01:28:59Z","dependency_count":0,"dependent_count":0,"comment_count":0} diff --git a/apps/web/app/globals.css b/apps/web/app/globals.css index 1b2205c..a0e1822 100644 --- a/apps/web/app/globals.css +++ b/apps/web/app/globals.css @@ -1003,6 +1003,17 @@ h3 { overflow: hidden; } +.history-load-warning { + flex: 0 0 auto; + padding: 8px 12px; + border-top: 1px solid oklch(0.72 0.13 58 / 0.45); + border-bottom: 1px solid oklch(0.72 0.13 58 / 0.45); + background: oklch(0.24 0.05 58 / 0.72); + color: oklch(0.91 0.08 72); + font-size: 0.78rem; + line-height: 1.35; +} + .data-table-wrap { display: flex; flex: 1 1 auto; diff --git a/apps/web/app/terminal.tsx b/apps/web/app/terminal.tsx index 2135a75..1cd6f42 100644 --- a/apps/web/app/terminal.tsx +++ b/apps/web/app/terminal.tsx @@ -7109,6 +7109,13 @@ type OptionsPaneProps = { const OptionsPane = memo(({ state, limit }: OptionsPaneProps) => { const items = limit ? state.filteredOptions.slice(0, limit) : state.filteredOptions; const virtual = useTapeVirtualList(items, state.optionsScroll.listRef, getTapeVirtualConfig("options")); + const optionHistorySubscription = state.liveSession.manifest.find( + (subscription) => subscription.channel === "options" + ); + const optionHistoryKey = optionHistorySubscription ? getLiveSubscriptionKey(optionHistorySubscription) : null; + const optionHistoryError = optionHistoryKey + ? state.liveSession.historyErrors[optionHistoryKey] + : null; useVirtualHistoryGate(state.mode === "live" && !limit, items.length, virtual.virtualItems.at(-1)?.index ?? -1, () => void state.liveSession.loadOlder("options") ); @@ -7139,6 +7146,11 @@ const OptionsPane = memo(({ state, limit }: OptionsPaneProps) => { } >
+ {state.mode === "live" && optionHistoryError ? ( +
+ Older option history failed to load: {optionHistoryError} +
+ ) : null} {items.length === 0 ? (
{state.mode === "live" diff --git a/deployment/docker/README.md b/deployment/docker/README.md index 4a5019f..0f5c886 100644 --- a/deployment/docker/README.md +++ b/deployment/docker/README.md @@ -119,10 +119,16 @@ Supported routing modes: - Build web with `NEXT_PUBLIC_API_URL=` (empty). - Point `app.` at the web host port. - Proxy these API routes from the app origin to the API host port: - - `/ws/*`, `/replay/*`, `/prints/*`, `/joins/*`, `/nbbo/*`, `/dark/*`, `/flow/*`, `/candles/*` + - `/ws/*`, `/replay/*`, `/prints/*`, `/joins/*`, `/nbbo/*`, `/dark/*`, `/flow/*`, `/candles/*`, `/history/*` Enable websocket support on whichever host serves `/ws/*`. +For the current live Nginx Proxy Manager setup behind `flow.deltaisland.io`, keep the API location regex durable in the proxy host advanced config or API, not by hand-editing generated files under `/data/nginx/proxy_host/`. The route matcher should include history: + +```nginx +^/(ws|replay|prints|joins|nbbo|dark|flow|candles|history)/ +``` + ## Replay service Replay is disabled by default in this stack. @@ -441,3 +447,4 @@ After the stack is up: - `curl -I http://127.0.0.1:3000/` should return a successful HTTP status on the server. - In two-origin mode, browser requests should target `https://api./...` and live feeds should use `wss://api./ws/...`. - In same-origin mode, browser requests should target `https://app./...` for API paths and live feeds should use `wss://app./ws/...`. +- In same-origin mode, `bun run check:public-api-routes` should pass for `/prints/options`, `/history/options`, `/replay/options`, `/nbbo/options`, and `/ws/live`. diff --git a/docs/turns/2026-05-16-2159-fix-durable-options-history-routing.html b/docs/turns/2026-05-16-2159-fix-durable-options-history-routing.html new file mode 100644 index 0000000..62be8b7 --- /dev/null +++ b/docs/turns/2026-05-16-2159-fix-durable-options-history-routing.html @@ -0,0 +1,195 @@ + + + + + + Fix Durable Options History Routing + + + +
+
+ Validated +

Fix Durable Options History Routing

+

Turn completed on 2026-05-16 21:59 America/New_York.

+
+ +
+

Summary

+

+ Options tape history now has a durable public route through same-origin deployments. The live Nginx Proxy Manager route was updated to include /history/*, deployment checks now fail when required API paths reach the web app, and the tape UI surfaces older-history load failures instead of leaving the user to infer that only the hot window exists. +

+
+ +
+

Changes Made

+
    +
  • Added scripts/check-public-api-routes.ts and the check:public-api-routes package script.
    • +
  • Updated scripts/deploy.ts so same-origin API deploy verification probes required public routes.
    • +
  • Updated deployment/docker/README.md to include /history/* in same-origin proxy routing and document the Nginx Proxy Manager regex.
    • +
  • Added an options tape warning banner for live /history/options load errors.
    • +
  • Updated live Nginx Proxy Manager config for flow.deltaisland.io so the public route regex includes history.
    • +
  • Created follow-up Beads issue islandflow-qd7 for the later api.flow.deltaisland.io migration.
    • +
+
+ +
+

Context

+

+ The API and ClickHouse path already supported older options history, but the public same-origin route sent /history/options to the Next.js app. That made the live tape feel capped at the newest hot-window rows even though durable history existed behind the API. +

+
+ +
+

Important Implementation Details

+

+ The deploy smoke check performs GET probes and verifies JSON responses for these same-origin routes: +

+ 
/prints/options
+/history/options
+/replay/options
+/nbbo/options
+/ws/live
+

+ The live proxy matcher is now: +

+ 
^/(ws|replay|prints|joins|nbbo|dark|flow|candles|history)/
+
+ +
+

Expected Impact for End-Users

+

+ Users on /tape can scroll beyond the initial options hot window and receive older ClickHouse-backed rows through the same cursor path for Signal and All prints. If public routing regresses, the tape now shows a visible history loading failure. +

+
+ +
+

Validation

+
    +
  • Passed: bun test apps/web/app/terminal.test.ts
    • +
  • Passed: bun test
    • +
  • Passed: bun --cwd=apps/web run build
    • +
  • Passed: bun run check:public-api-routes
    • +
  • Passed: remote Nginx syntax check after updating the route.
    • +
+
+ +
+

Issues, Limitations, and Mitigations

+
    +
  • The long-term API subdomain migration remains separate work. Mitigation: tracked as islandflow-qd7.
    • +
  • The Nginx Proxy Manager database and generated proxy host file were both updated because the existing live file had prior generated-file edits. Mitigation: deployment docs now call out the durable advanced-config/API path.
    • +
+
+ +
+

Follow-up Work

+

+ Complete islandflow-qd7 to move production API traffic to api.flow.deltaisland.io deliberately, including DNS, proxy behavior, CORS/websocket checks, docs, and deployment verification. +

+
+
+ + diff --git a/package.json b/package.json index e02d218..7a9a509 100644 --- a/package.json +++ b/package.json @@ -20,6 +20,7 @@ "deploy": "bun run scripts/deploy.ts", "deploy:main": "./deploy main", "deploy:current-branch": "./deploy current-branch", + "check:public-api-routes": "bun run scripts/check-public-api-routes.ts", "sync:docker-workspace": "bun run scripts/sync-docker-workspace.ts", "check:docker-workspace": "bun run scripts/check-docker-workspace.ts" }, diff --git a/scripts/check-public-api-routes.ts b/scripts/check-public-api-routes.ts new file mode 100644 index 0000000..d1f0a18 --- /dev/null +++ b/scripts/check-public-api-routes.ts @@ -0,0 +1,41 @@ +#!/usr/bin/env bun + +type RouteCheck = { + path: string; + expectJson: boolean; +}; + +const routeChecks: RouteCheck[] = [ + { path: "/prints/options?view=signal&limit=1", expectJson: true }, + { path: "/history/options?view=signal&before_ts=4102444800000&before_seq=999999999&limit=1", expectJson: true }, + { path: "/replay/options?view=signal&after_ts=0&after_seq=0&limit=1", expectJson: true }, + { path: "/nbbo/options?limit=1", expectJson: true }, + { path: "/ws/live", expectJson: true } +]; + +const appUrl = process.env.DEPLOY_PUBLIC_APP_URL?.trim() || process.argv[2]?.trim(); +const baseUrl = appUrl || "https://flow.deltaisland.io"; + +const isJsonResponse = (response: Response): boolean => { + return (response.headers.get("content-type") ?? "").toLowerCase().includes("application/json"); +}; + +const assertPublicApiRoute = async ({ path, expectJson }: RouteCheck): Promise => { + const url = new URL(path, baseUrl); + const response = await fetch(url); + const responseText = await response.text(); + + if (response.status === 404) { + throw new Error(`${url.pathname} returned 404; route is likely reaching the web app`); + } + + if (expectJson && !isJsonResponse(response)) { + const sample = responseText.replace(/\s+/g, " ").slice(0, 120); + throw new Error(`${url.pathname} returned non-JSON content (${response.headers.get("content-type") ?? "none"}): ${sample}`); + } +}; + +for (const check of routeChecks) { + await assertPublicApiRoute(check); + console.log(`ok ${check.path}`); +} diff --git a/scripts/deploy.ts b/scripts/deploy.ts index d6adcb1..cb30de9 100644 --- a/scripts/deploy.ts +++ b/scripts/deploy.ts @@ -732,9 +732,7 @@ function publicVerification(scope: DeployScope): void { } if (scopeIncludesApi(scope)) { - console.log( - "Skipping separate public API health check; same-origin mode relies on the public app check plus runtime-local API verification." - ); + runChecked("bun", ["run", "scripts/check-public-api-routes.ts", PUBLIC_APP_URL]); } } From d334e16874f6989d8a54aeb095b72ddc8a1bafbd Mon Sep 17 00:00:00 2001 From: dirtydishes Date: Sun, 17 May 2026 03:33:06 -0400 Subject: [PATCH 3/3] fix live tape scroll stability --- .beads/issues.jsonl | 1 + apps/web/app/globals.css | 24 +++ apps/web/app/terminal.test.ts | 60 ++++++- apps/web/app/terminal.tsx | 59 +++++- ...7-0331-fix-live-tape-scroll-stability.html | 168 ++++++++++++++++++ 5 files changed, 298 insertions(+), 14 deletions(-) create mode 100644 docs/turns/2026-05-17-0331-fix-live-tape-scroll-stability.html diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index 2bf9d72..eb38e91 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -1,3 +1,4 @@ +{"_type":"issue","id":"islandflow-9dg","title":"Fix live tape scroll stability","description":"Live tape rows can shift while a user is scrolled away from the hot head because newer live prints and ClickHouse history are merged into the displayed segment. Implement held-history freezing so only truly older rows append below the current tail, resync on jump-to-top, and tune virtualization/background rendering to reduce fast-scroll blank gaps.","status":"closed","priority":1,"issue_type":"bug","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-17T07:28:52Z","created_by":"dirtydishes","updated_at":"2026-05-17T07:32:53Z","started_at":"2026-05-17T07:29:00Z","closed_at":"2026-05-17T07:32:53Z","close_reason":"Implemented held live tape history freezing, older-only held history append, jump-to-top resync behavior, virtualizer overscan tuning, and stable row-lane table background. Validated with scoped Bun tests, web production build, and local /tape HTTP smoke check.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-qso","title":"Fix durable options tape history routing","description":"Implement the fix-tape plan: make same-origin history routing durable, add deployment/public smoke checks for required API routes, expose tape history loading failures in the UI, document the work, and track api.flow.deltaisland.io migration separately.","status":"closed","priority":1,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-17T01:53:22Z","created_by":"dirtydishes","updated_at":"2026-05-17T02:00:04Z","started_at":"2026-05-17T01:53:25Z","closed_at":"2026-05-17T02:00:04Z","close_reason":"Implemented durable same-origin history routing, public route smoke checks, tape history diagnostics, docs, validation, and follow-up tracking for api.flow.deltaisland.io.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-200","title":"Implement durable options tape history","description":"Implement the plan from docs/plans/2026-05-16-1711-durable-options-tape-history.html: durable ClickHouse-backed options history, signal/all prints view selection, preserved execution context, stale semantics limited to live health, reset runbook, tests, and turn documentation.","status":"closed","priority":1,"issue_type":"feature","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-16T21:21:30Z","created_by":"dirtydishes","updated_at":"2026-05-16T21:26:51Z","started_at":"2026-05-16T21:21:33Z","closed_at":"2026-05-16T21:26:51Z","close_reason":"Implemented durable options tape history, signal/raw view selection, reset runbook, tests, and turn documentation.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"islandflow-k4f","title":"Gate deploy script on docker workspace snapshot sync","description":"Prevent frozen-lockfile build failures during deploy by adding a local preflight in scripts/deploy.ts that runs bun run check:docker-workspace and aborts with a clear sync+commit remediation message when stale.","status":"closed","priority":1,"issue_type":"task","assignee":"dirtydishes","owner":"dishes@dpdrm.com","created_at":"2026-05-15T23:01:44Z","created_by":"dirtydishes","updated_at":"2026-05-15T23:04:11Z","started_at":"2026-05-15T23:01:48Z","closed_at":"2026-05-15T23:04:11Z","close_reason":"Closed","dependency_count":0,"dependent_count":0,"comment_count":0} diff --git a/apps/web/app/globals.css b/apps/web/app/globals.css index a0e1822..46f20bb 100644 --- a/apps/web/app/globals.css +++ b/apps/web/app/globals.css @@ -1039,11 +1039,27 @@ h3 { min-height: 0; overflow-y: auto; overflow-x: hidden; + background-color: oklch(0.12 0.01 250); } .data-table-body { position: relative; min-width: 100%; + --tape-row-height: 36px; + --tape-row-double-height: 72px; + background: + repeating-linear-gradient( + to bottom, + oklch(0.98 0.008 250 / 0.01) 0, + oklch(0.98 0.008 250 / 0.01) calc(var(--tape-row-height) - 1px), + oklch(0.72 0.012 250 / 0.08) calc(var(--tape-row-height) - 1px), + oklch(0.72 0.012 250 / 0.08) var(--tape-row-height), + oklch(0.98 0.008 250 / 0.018) var(--tape-row-height), + oklch(0.98 0.008 250 / 0.018) calc(var(--tape-row-double-height) - 1px), + oklch(0.72 0.012 250 / 0.08) calc(var(--tape-row-double-height) - 1px), + oklch(0.72 0.012 250 / 0.08) var(--tape-row-double-height) + ), + oklch(0.12 0.01 250); } .data-table-options { @@ -1137,6 +1153,14 @@ h3 { height: 44px; } +.data-table-flow .data-table-body, +.data-table-alerts .data-table-body, +.data-table-classifier .data-table-body, +.data-table-dark .data-table-body { + --tape-row-height: 44px; + --tape-row-double-height: 88px; +} + .data-table-row-classified { background: linear-gradient(90deg, rgba(var(--classifier-rgb, 192, 200, 210), calc(0.012 + var(--classifier-intensity, 0) * 0.06)), transparent 62%), diff --git a/apps/web/app/terminal.test.ts b/apps/web/app/terminal.test.ts index 03114c4..b6214eb 100644 --- a/apps/web/app/terminal.test.ts +++ b/apps/web/app/terminal.test.ts @@ -24,6 +24,7 @@ import { getLiveManifest, getRouteFeatures, getTapeVirtualConfig, + mergeHeldTapeHistory, mergeNewestWithOverflow, normalizeAlertSeverity, normalizeTickerFilterInput, @@ -394,12 +395,12 @@ describe("route feature map", () => { describe("fixed tape virtualization config", () => { it("uses expected fixed row heights and overscan by table", () => { - expect(getTapeVirtualConfig("options")).toEqual({ rowHeight: 36, overscan: 24, debugLabel: "options" }); - expect(getTapeVirtualConfig("equities")).toEqual({ rowHeight: 36, overscan: 20, debugLabel: "equities" }); - expect(getTapeVirtualConfig("flow")).toEqual({ rowHeight: 44, overscan: 16, debugLabel: "flow" }); - expect(getTapeVirtualConfig("alerts")).toEqual({ rowHeight: 44, overscan: 16, debugLabel: "alerts" }); - expect(getTapeVirtualConfig("classifier")).toEqual({ rowHeight: 44, overscan: 16, debugLabel: "classifier" }); - expect(getTapeVirtualConfig("dark")).toEqual({ rowHeight: 44, overscan: 16, debugLabel: "dark" }); + expect(getTapeVirtualConfig("options")).toEqual({ rowHeight: 36, overscan: 44, debugLabel: "options" }); + expect(getTapeVirtualConfig("equities")).toEqual({ rowHeight: 36, overscan: 36, debugLabel: "equities" }); + expect(getTapeVirtualConfig("flow")).toEqual({ rowHeight: 44, overscan: 24, debugLabel: "flow" }); + expect(getTapeVirtualConfig("alerts")).toEqual({ rowHeight: 44, overscan: 24, debugLabel: "alerts" }); + expect(getTapeVirtualConfig("classifier")).toEqual({ rowHeight: 44, overscan: 24, debugLabel: "classifier" }); + expect(getTapeVirtualConfig("dark")).toEqual({ rowHeight: 44, overscan: 24, debugLabel: "dark" }); }); }); @@ -683,6 +684,53 @@ describe("live tape history helpers", () => { const nextKeys = ["anchor", "after-1", "after-2", "older-1", "older-2"]; expect(findAnchorRestoreIndex(nextKeys, "anchor", ["anchor", "after-1", "after-2"])).toBe(0); }); + + it("keeps held ClickHouse history stable when newer live overflow arrives", () => { + const frozenLive = [makeItem("hot-5", 5, 500), makeItem("hot-4", 4, 400)]; + const displayed = [makeItem("hist-3", 3, 300), makeItem("hist-2", 2, 200)]; + const incoming = [ + makeItem("overflow-newer", 6, 600), + makeItem("hot-4", 4, 400), + makeItem("hist-3", 3, 300), + makeItem("hist-2", 2, 200) + ]; + + expect(mergeHeldTapeHistory(displayed, incoming, frozenLive).map((item) => item.trace_id)).toEqual([ + "hist-3", + "hist-2" + ]); + }); + + it("appends truly older lazy-loaded rows to the held history tail", () => { + const frozenLive = [makeItem("hot-5", 5, 500), makeItem("hot-4", 4, 400)]; + const displayed = [makeItem("hist-3", 3, 300), makeItem("hist-2", 2, 200)]; + const incoming = [ + makeItem("hist-3", 3, 300), + makeItem("hist-2", 2, 200), + makeItem("older-1", 1, 100), + makeItem("older-0", 0, 50) + ]; + + expect(mergeHeldTapeHistory(displayed, incoming, frozenLive).map((item) => item.trace_id)).toEqual([ + "hist-3", + "hist-2", + "older-1", + "older-0" + ]); + }); + + it("resyncs buffered live history by replacing the held segment after resume", () => { + const frozenLive = [makeItem("hot-5", 5, 500), makeItem("hot-4", 4, 400)]; + const held = mergeHeldTapeHistory( + [makeItem("hist-3", 3, 300), makeItem("hist-2", 2, 200)], + [makeItem("overflow-newer", 6, 600), makeItem("hist-3", 3, 300), makeItem("older-1", 1, 100)], + frozenLive + ); + const resynced = appendHistoryTail([], [makeItem("overflow-newer", 6, 600), ...held], [], 0); + + expect(held.map((item) => item.trace_id)).toEqual(["hist-3", "hist-2", "older-1"]); + expect(resynced.map((item) => item.trace_id)).toEqual(["overflow-newer", "hist-3", "hist-2", "older-1"]); + }); }); describe("options display formatters", () => { diff --git a/apps/web/app/terminal.tsx b/apps/web/app/terminal.tsx index 1cd6f42..0dfc199 100644 --- a/apps/web/app/terminal.tsx +++ b/apps/web/app/terminal.tsx @@ -142,12 +142,12 @@ type TapeVirtualListConfig = { }; const TAPE_VIRTUAL_CONFIG: Record = { - options: { rowHeight: 36, overscan: 24, debugLabel: "options" }, - equities: { rowHeight: 36, overscan: 20, debugLabel: "equities" }, - flow: { rowHeight: 44, overscan: 16, debugLabel: "flow" }, - alerts: { rowHeight: 44, overscan: 16, debugLabel: "alerts" }, - classifier: { rowHeight: 44, overscan: 16, debugLabel: "classifier" }, - dark: { rowHeight: 44, overscan: 16, debugLabel: "dark" } + options: { rowHeight: 36, overscan: 44, debugLabel: "options" }, + equities: { rowHeight: 36, overscan: 36, debugLabel: "equities" }, + flow: { rowHeight: 44, overscan: 24, debugLabel: "flow" }, + alerts: { rowHeight: 44, overscan: 24, debugLabel: "alerts" }, + classifier: { rowHeight: 44, overscan: 24, debugLabel: "classifier" }, + dark: { rowHeight: 44, overscan: 24, debugLabel: "dark" } }; export const getTapeVirtualConfig = (pane: TapeVirtualPane): TapeVirtualListConfig => @@ -844,6 +844,30 @@ export const appendHistoryTail = ( return cap > 0 ? combined.slice(0, cap) : combined; }; +export const mergeHeldTapeHistory = ( + displayedHistory: T[], + incomingHistory: T[], + frozenLiveHead: T[] +): T[] => { + if (displayedHistory.length === 0) { + return appendHistoryTail([], incomingHistory, frozenLiveHead, 0); + } + + const sortedDisplayed = appendHistoryTail([], displayedHistory, frozenLiveHead, 0); + const tail = sortedDisplayed.at(-1); + const tailTs = tail ? extractSortTs(tail) : Number.POSITIVE_INFINITY; + const tailSeq = tail ? extractSortSeq(tail) : Number.POSITIVE_INFINITY; + const olderIncoming = incomingHistory.filter((item) => { + const itemTs = extractSortTs(item); + if (itemTs < tailTs) { + return true; + } + return itemTs === tailTs && extractSortSeq(item) < tailSeq; + }); + + return appendHistoryTail(sortedDisplayed, olderIncoming, frozenLiveHead, 0); +}; + export const getLiveHistoryRetentionCap = (subscription: LiveSubscription): number => { switch (subscription.channel) { case "options": @@ -2491,6 +2515,7 @@ const usePausableTapeView = ( config: PausableTapeViewConfig ): TapeState => { const [data, setData] = useState>(EMPTY_PAUSABLE_TAPE); + const displayedHistoryRef = useRef([]); const holdForScroll = config.enabled ? (config.shouldHold ? config.shouldHold() : false) : false; useEffect(() => { @@ -2557,13 +2582,31 @@ const usePausableTapeView = ( const status = config.enabled ? config.sourceStatus : "disconnected"; const projected = projectPausableTapeState(data.visible, status, config.lastUpdate); const historyItems = config.historyTail ?? []; - const items = useMemo(() => composeTapeItems([], projected.items, historyItems), [projected.items, historyItems]); + const displayedHistoryItems = useMemo(() => { + if (!config.enabled) { + displayedHistoryRef.current = []; + return []; + } + + if (!holdForScroll) { + displayedHistoryRef.current = historyItems; + return historyItems; + } + + const next = mergeHeldTapeHistory(displayedHistoryRef.current, historyItems, projected.items); + displayedHistoryRef.current = next; + return next; + }, [config.enabled, historyItems, holdForScroll, projected.items]); + const items = useMemo( + () => composeTapeItems([], projected.items, displayedHistoryItems), + [projected.items, displayedHistoryItems] + ); return { status, items, liveItems: projected.items, - historyItems, + historyItems: displayedHistoryItems, lastUpdate: projected.lastUpdate, replayTime: null, replayComplete: false, diff --git a/docs/turns/2026-05-17-0331-fix-live-tape-scroll-stability.html b/docs/turns/2026-05-17-0331-fix-live-tape-scroll-stability.html new file mode 100644 index 0000000..81b1576 --- /dev/null +++ b/docs/turns/2026-05-17-0331-fix-live-tape-scroll-stability.html @@ -0,0 +1,168 @@ + + + + + + Fix Live Tape Scroll Stability + + + +
+
+

Fix Live Tape Scroll Stability

+

+ Completed on 2026-05-17 at 03:31 America/New_York for Beads issue + islandflow-9dg. +

+
+ +
+

Summary

+

+ The live tape now keeps the visible scrolled segment stable while new prints arrive. When + the user is away from the top, the view freezes both the hot live head and the displayed + history segment, only allowing genuinely older history to append below the current tail. +

+
+ +
+

Changes Made

+
    +
  • Added mergeHeldTapeHistory to filter held history updates by the visible tail.
    • +
  • Updated usePausableTapeView to keep a displayed history ref while scroll-held.
    • +
  • Resynced displayed history automatically when the user jumps back to the top or otherwise resumes.
    • +
  • Increased tape virtualizer overscan for options, equities, flow, alerts, classifier, and dark panes.
    • +
  • Added a fixed row-lane table background so fast scrolling shows a stable substrate instead of blank holes.
    • +
+
+ +
+

Context

+

+ Live session history receives both ClickHouse history and hot-window overflow from new live + prints. Before this change, the pausable view froze live rows during scroll hold but still + composed against the mutating history array, so newer overflow rows could insert above the + user's current viewport. +

+
+ +
+

Important Implementation Details

+

+ The stable merge compares incoming history with the current displayed history tail. Rows + newer than that tail are withheld during hold, duplicates from the frozen live head are + removed, and older lazy-loaded rows remain eligible to append. +

+ 
const next = mergeHeldTapeHistory(displayedHistoryRef.current, historyItems, projected.items);
+

+ When hold ends, displayedHistoryRef is replaced with the latest live session + history, so buffered overflow catches up cleanly on jump-to-top. +

+
+ +
+

Expected Impact for End-Users

+

+ Users can scroll into older options or equities prints without the rows shifting under them + as new live prints arrive. The +N new counter can continue accumulating until + they jump back to the top, where the tape catches up. +

+
+ +
+

Validation

+
    +
  • bun test apps/web/app/terminal.test.ts services/api/tests/live.test.ts: passed, 90 tests.
    • +
  • bun --cwd=apps/web run build: passed.
    • +
  • curl -I http://localhost:3000/tape against the local dev server: returned 200 OK.
    • +
+
+ +
+

Issues, Limitations, and Mitigations

+

+ This change preserves row stability in the frontend view model. It does not alter backend + history pagination or wire protocols. The fixed table substrate mitigates visual blanking + during fast scrolls, while actual row rendering remains virtualized. Browser automation was + attempted, but the local Node automation runtime did not have Playwright installed, so the + handoff relies on unit tests, production build, and the local HTTP smoke check. +

+
+ +
+

Follow-up Work

+

No follow-up Beads issues were needed for this turn.

+
+
+ +