dirtydishes/islandflow
1
0
Fork 0
Code Issues 2 Pull requests 4 Projects Releases Packages Wiki Activity Actions 1

stabilize live api memory and add an options pipeline explainer #8

Open
dirtydishes wants to merge 6 commits from stabilize-live-api-memory into main
pull from: stabilize-live-api-memory
merge into: dirtydishes:main
Conversation 0 Commits 6 Files changed 12 +2950 -92
dirtydishes commented 2026-05-23 02:49:06 +00:00
Owner
Copy link

this pr stabilizes the live api after the vps memory blow-up and adds a proper explainer page for the options-print to smart-money pipeline.

what changed

  • capped live cache windows in the api so unsafe env values cannot push bun back into multi-gb rss
  • changed the hot generic redis persistence path from full rewrites to append-and-trim behavior, with safer shutdown flushing
  • added minute-level api visibility for memory, live subscriptions, snapshot sizes, and redis flush volume
  • tightened the web-side live window bounds and aligned env examples with the safer defaults
  • added docs/anatomy.html, a standalone reference page that explains how prints move from ingest to tape, flow packets, smart-money events, classifier hits, and alerts
  • regenerated docs/index.html so the new explainer is easy to find

why this changed

the immediate problem was that the live api could retain huge hot caches and churn redis hard enough to get oom-killed on the vps. the follow-on problem was that the options and smart-money pipeline still lived mostly in code and chat context, which made it harder to reason about what the system is actually doing when a print shows up on tape but never becomes a signal.

root cause

the api was trusting oversized live cache settings, serving large reconnect snapshots, and rewriting hot redis lists too aggressively. that combination amplified loopback traffic and memory growth inside the box. separately, the event pipeline had a documentation gap even though the implementation already had a clear staged model.

validation

  • bun test services/api/tests/live.test.ts
  • bun --cwd=apps/web run build
  • manual vps rollout and restart verification for the api and web services
  • regenerated docs index with node scripts/generate-docs-index.mjs
  • content checks for the new anatomy page and its turn document

intended effect for end users

end users should see a much more stable live product under load, especially during reconnects and busy tape conditions. the hosted app should stop falling over because of runaway live-cache behavior, and the team now has a durable reference page that makes the options-flow and smart-money pipeline much easier to understand and debug.

this pr stabilizes the live api after the vps memory blow-up and adds a proper explainer page for the options-print to smart-money pipeline. what changed - capped live cache windows in the api so unsafe env values cannot push bun back into multi-gb rss - changed the hot generic redis persistence path from full rewrites to append-and-trim behavior, with safer shutdown flushing - added minute-level api visibility for memory, live subscriptions, snapshot sizes, and redis flush volume - tightened the web-side live window bounds and aligned env examples with the safer defaults - added `docs/anatomy.html`, a standalone reference page that explains how prints move from ingest to tape, flow packets, smart-money events, classifier hits, and alerts - regenerated `docs/index.html` so the new explainer is easy to find why this changed the immediate problem was that the live api could retain huge hot caches and churn redis hard enough to get oom-killed on the vps. the follow-on problem was that the options and smart-money pipeline still lived mostly in code and chat context, which made it harder to reason about what the system is actually doing when a print shows up on tape but never becomes a signal. root cause the api was trusting oversized live cache settings, serving large reconnect snapshots, and rewriting hot redis lists too aggressively. that combination amplified loopback traffic and memory growth inside the box. separately, the event pipeline had a documentation gap even though the implementation already had a clear staged model. validation - `bun test services/api/tests/live.test.ts` - `bun --cwd=apps/web run build` - manual vps rollout and restart verification for the api and web services - regenerated docs index with `node scripts/generate-docs-index.mjs` - content checks for the new anatomy page and its turn document intended effect for end users end users should see a much more stable live product under load, especially during reconnects and busy tape conditions. the hosted app should stop falling over because of runaway live-cache behavior, and the team now has a durable reference page that makes the options-flow and smart-money pipeline much easier to understand and debug.
dirtydishes added 4 commits 2026-05-23 02:49:06 +00:00
dirtydishes added 1 commit 2026-05-23 20:24:43 +00:00
dirtydishes added 1 commit 2026-05-23 20:26:06 +00:00
This pull request has changes conflicting with the target branch.
  • .beads/issues.jsonl
  • services/api/src/live.ts
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin stabilize-live-api-memory:stabilize-live-api-memory
git checkout stabilize-live-api-memory

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git checkout main
git merge --no-ff stabilize-live-api-memory
git checkout stabilize-live-api-memory
git rebase main
git checkout main
git merge --ff-only stabilize-live-api-memory
git checkout stabilize-live-api-memory
git rebase main
git checkout main
git merge --no-ff stabilize-live-api-memory
git checkout main
git merge --squash stabilize-live-api-memory
git checkout main
git merge --ff-only stabilize-live-api-memory
git checkout main
git merge stabilize-live-api-memory
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: dirtydishes/islandflow#8
No description provided.
Delete branch "stabilize-live-api-memory"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?