Created 2026-05-19 20:05 EDT · Branch: `alpaca-news` · Issue: `islandflow-laq`

# Fix Native Alpaca News

Restored the native Alpaca news pipeline on the VPS by correcting Alpaca auth to use key ID + secret, adding the missing native `islandflow-ingest-news` unit and worker-scope wiring, fixing the Alpaca news backfill defaults to match the current API contract, requesting article content explicitly, and repairing API-side news persistence so the feed is both live and queryable.

VPS unit installed and enabled Alpaca auth aligned to current docs Live news confirmed ClickHouse news history confirmed

## Summary

The original native news rollout failed for two separate reasons: the repo never fully wired `ingest-news` into the native worker templates, and the service was still using bearer-style Alpaca auth plus an oversized backfill limit that Alpaca's current News API rejects. After the service started flowing again, one more pipeline gap appeared: the API fanned news out live but never persisted it to ClickHouse, so `/news` stayed empty even when headlines showed up in the UI.

## Changes Made

- Added shared Alpaca credential helpers in `packages/config` with support for official key ID + secret auth and a legacy bearer fallback.
- Rewired the Alpaca news, options, and equities adapters to use the shared auth model instead of hardcoded bearer headers and empty websocket secrets.
- Added the checked-in native user unit `deployment/native/systemd/user/islandflow-ingest-news.service`.
- Updated native install, health, cutover, rollback, and deploy-scope scripts so worker/native rollouts include `ingest-news`.
- Corrected the native and Docker env/docs story to advertise current Alpaca credential names.
- Lowered the default Alpaca news backfill limit from `100` to `50` to match the current endpoint contract.
- Requested `include_content=true` for Alpaca news backfill and added a safe summary fallback when article content is missing.
- Fixed API-side persistence by inserting each consumed news story into ClickHouse before live fanout.
- On the VPS, created a fresh `.env` backup, added `ALPACA_API_KEY_ID` and `ALPACA_API_SECRET_KEY`, set `ALPACA_NEWS_BACKFILL_LIMIT=50`, switched the server checkout to `alpaca-news`, installed the new user unit, and restarted `api` plus `ingest-news`.

## Context

Alpaca's current official auth docs require the `APCA-API-KEY-ID` and `APCA-API-SECRET-KEY` header pair for market-data requests, and the current News endpoint documents a `limit` range of `1..50` plus optional `include_content`. This turn aligned Islandflow's native news path with those present-day contracts instead of relying on the older single-token assumption that had drifted into the repo.

## Important Implementation Details

- The shared helper prefers `ALPACA_API_KEY_ID` + `ALPACA_API_SECRET_KEY`, also accepts `ALPACA_KEY_ID` + `ALPACA_SECRET_KEY`, and only falls back to legacy bearer auth when no secret is present.
- The news backfill now requests article bodies explicitly. When Alpaca still omits full content, the service emits an escaped summary paragraph instead of a blank story body.
- The native worker scope now treats `ingest-news` as a first-class worker everywhere the repo previously only handled options and equities.
- The API now persists each consumed news story into ClickHouse before live fanout, which restores `/news` and history behavior without removing the live websocket path.

## Relevant Diff Snippets

    diff --git a/packages/config/src/alpaca.ts b/packages/config/src/alpaca.ts
    +export const buildAlpacaAuthHeaders = (credentials) => ({
    +  "APCA-API-KEY-ID": credentials.keyId,
    +  "APCA-API-SECRET-KEY": credentials.secret
    +})
    +export const buildAlpacaWebSocketAuthMessage = (credentials) => ({
    +  action: "auth",
    +  key: credentials.keyId,
    +  secret: credentials.secret
    +})

    diff --git a/services/ingest-news/src/index.ts b/services/ingest-news/src/index.ts
    -  ALPACA_NEWS_BACKFILL_LIMIT: z.coerce.number().int().positive().max(200).default(100),
    +  ALPACA_NEWS_BACKFILL_LIMIT: z.coerce.number().int().positive().max(50).default(50),
    +  url.searchParams.set("include_content", "true");
    +  const contentHtml = item.content?.trim() || (summary ? `<p>${escapeHtml(summary)}</p>` : "");

    diff --git a/services/api/src/index.ts b/services/api/src/index.ts
       const payload = NewsStorySchema.parse(newsSubscription.decode(msg));
    +  await insertNewsStory(clickhouse, payload);
       await fanoutLive({ channel: "news" }, payload, "news");
       msg.ack();

These snippets are included in a diff-style rendering format for fast review.

## Expected Impact for End-Users

Native Islandflow deployments on the VPS now have a real Alpaca-backed news worker instead of a missing unit and a crash loop. News stories populate with actual article body content in the feed more reliably, and the API's `/news` path can serve persisted recent stories instead of only depending on live websocket state.

## Validation

- Ran local targeted tests: `bun test packages/config/tests packages/storage/tests/news.test.ts services/ingest-news/tests services/ingest-equities/tests` and all passed.
- Ran `bun run check:docker-workspace` and confirmed the Docker workspace snapshot stayed in sync.
- Verified against current Alpaca docs that market-data auth uses key ID + secret and that the news endpoint limit is capped at 50.
- On the VPS, confirmed the new `islandflow-ingest-news.service` unit is installed, enabled, and active under `systemd --user`.
- Queried Alpaca directly from the VPS with the configured credentials and confirmed `GET https://data.alpaca.markets/v1beta1/news?limit=1&sort=desc` returned HTTP 200.
- Restarted the VPS `api` and `ingest-news` services after the persistence fix so the API would store newly republished backfill stories.
- Verified VPS API output: `GET http://127.0.0.1:4000/news?limit=3` returned 3 recent real Alpaca stories with non-empty `content_html` payloads.
- Verified ClickHouse persistence: `SELECT count(), max(story_id), max(published_ts) FROM news` returned `50` rows after the republished backfill.

## Issues, Limitations, and Mitigations

- The server checkout still carries an unrelated untracked file, `deployment/docker/signal-cli-0.14.3-Linux-native.tar.gz`. It does not block the news fix, but it is repo hygiene debt on the VPS checkout.
- The shared Alpaca helper keeps a legacy bearer fallback so older setups do not fail immediately, but the repo documentation now treats key ID + secret as the supported path.
- Some Alpaca/Benzinga stories may still omit full content. The summary fallback prevents a blank drawer in those cases, but it cannot synthesize text Alpaca does not send.

## Follow-up Work

- No new follow-up Beads issue was required to ship this repair.
- If native Alpaca options or equities are re-enabled later, the shared credential changes in this turn already cover the same key ID + secret auth model.
- If the team wants historical news beyond the startup backfill, the next logical extension is a scheduled catch-up cursor instead of only restart-time republishing.