Skip to content

fix(podcast): give long scripts their own request policy; cap social publishing per platform - #277

Merged
i-xtsu-sixyou-ken-mei merged 20 commits into
mainfrom
fix/pipeline-llm-social-schedule
Aug 29, 2026
Merged

fix(podcast): give long scripts their own request policy; cap social publishing per platform#277
i-xtsu-sixyou-ken-mei merged 20 commits into
mainfrom
fix/pipeline-llm-social-schedule

Conversation

@i-xtsu-sixyou-ken-mei

@i-xtsu-sixyou-ken-mei i-xtsu-sixyou-ken-mei commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

Why

Two production behaviours were expensive rather than resilient.

  1. Long generateScript requests were killed while they were working. The prompt forbids summarizing, permits an output longer than its input, and sets no token ceiling, so a 13k-character article legitimately generates for minutes — but every OpenRouter call shared one 120s deadline, and the shared retry then replayed the identical request for another 120s. The 2026-08-28 PANews ingest (run_ref=26e50dd0) spent 248s that way before failing; the same article later succeeded. The failure left no trace anywhere: ops.pipeline_stage_runs could always express a failed attempt, but nothing produced one, and the Sentry event named neither the run nor the episode.
  2. Four release cohorts a day meant eight to eleven posts a day. Every lane of an episode shared one scheduled_at, so a multilingual article published everywhere at once, and production reach medians say the marginal posts reached nobody. publishSlotsJst lived in social_strategy_versions and was never read — the scheduler always used its own defaults.

Script generation

  • A private 600s deadline for script generation only, overriding the SDK's own timeout per request so the cached client's 120s ceiling cannot fire first. It is a constant, not an environment variable: it is a property of the prompt, not of a deployment (and an unregistered variable would fail the dead-env gate).
  • A timeout is terminal. A request that ran ten minutes had a model working on it; resubmitting the URL resumes from the persisted scrape far more cheaply than regenerating.
  • A gateway failure is re-routed exactly once, with sort dropped from provider. The throughput sort is deterministic, so an identical resend goes straight back to the endpoint that just refused it; dropping it hands selection back to OpenRouter's load balancer while require_parameters still holds.
  • The JSON-contract re-prompt is kept — it carries the rejection reason, so it is a correction, not a replay.
  • Language classroom and translation keep the shared 120s deadline and their bounded retry: their output is bounded by design.

Observability

  • Every upstream script request now writes its own ops.pipeline_stage_runs row — attempt, status, started_at/finished_at/elapsed_ms, provider, deadline, route, token counts, and on failure its category and message. A failed attempt is recorded unpriced rather than as a zero cost, which would read as a free success.
  • The language that dies mid-run pushes its own cost-sink entry, so its partial spend survives the throw — and it is the only place the run row could have got an episode id from.
  • The terminal Sentry boundary carries runRef and episodeId, and drains before the process can exit (index.ts shutdown, matching worker.ts).

Social publish budgets

Per-platform daily caps and candidate slots, code-owned in policy.ts:

Platform Posts / JST day Slots (JST)
rednote 1 14:30 (80%) / 12:00
threads 1 09:30 / 12:00
x 2 12:15 and 17:00, languages swapping daily
youtube 1 17:15

Five posts a day, down from eight to eleven.

  • The scheduling unit is (episode, platform). Language lanes of one platform still share a slot; different platforms are independent releases.
  • A cap counts cohorts per JST day across every language, so a multilingual platform cannot publish once per language and call it one post. X is the only cap above one because its language experiment assigns each episode exactly one of en/ja.
  • Slot assignment is deterministic and recorded: rednote-slot-v1 (80/20) and threads-timing-v1 (50/50) write social_experiment_assignments rows, so a reach report can attribute a post to the time it actually published at. X's crossover means neither language is permanently confounded with one time.
  • A completed row whose completed_at falls on a different JST day than its scheduled_at does not reserve a day it never used — reconciliation binds already-live posts to future slots (20 such rows exist dated 9/1–9/4).
  • A backlog longer than the 8-day horizon stays undiscovered and drains a day at a time. It is never compressed and never dropped.
  • YouTube distributes in English only.

Removed rather than adapted

  • Cohort alignment pulled every lane back to one timestamp — the opposite of a per-platform budget.
  • The partial-cohort fence held every other episode shut until a half-released one finished. Under these budgets a partial cohort is the steady state (Rednote 14:30 and YouTube 17:15 are the same article), so fencing on it would deadlock the queue against its own schedule.
  • skipOverdueSocialPublishJobs marked a missed lane completed with a skipped: overdue note — a post that never existed, recorded as published — behind SOCIAL_PUBLISH_SKIP_OVERDUE_MINUTES, which was unset in practice. A missed slot is now moved forward after a 90-minute grace, never dropped and never burst-published.
  • Publishing runs only 09:00–18:00 JST, because Rednote and X drive real browser sessions on a Mac someone has to be able to watch fail.

Strategy learning

SocialStrategyConfig carries copy guidance only. Timing in the DB would let a learner widen the very cap it is optimising inside. refreshSocialStrategies now retires any active lane the policy no longer ships, so the four stale rows (threads/zh-Hant, x/zh-Hant, youtube/zh-Hant, youtube/ja) heal themselves on the next refresh instead of needing SQL.

Account snapshots

Three-hour cadence, and the browser opens only when something that needs one is due (8 launches a day otherwise). YouTube now reports its absolute subscriber count via channels.list?mine=true — per-post subscribersGained cannot be summed into it. It needs youtube.readonly, which social:login now requests; until the operator re-consents once, that one platform's snapshot is skipped in isolation rather than failing the tick.

Database

One migration, view-only: 20260828143000_social_youtube_english_only_waiting_media.sql updates social_waiting_media's language policy so a Japanese asset is no longer reported as media the YouTube lane is waiting for. No RPC signature change, no strategy rows rewritten, no publish job touched — marking a queued row completed would record a post that never existed. A parity test keeps the view and policy.ts honest.

Rollout

  1. Merge, then apply the migration with supabase db push.
  2. Deploy podcast-pipeline to Fly (script deadline + ledger + Sentry changes).
  3. Restart the local social:daemon. First tick should deactivate the four stale strategy rows and schedule at or below each platform's cap; the 9/1–9/4 ghost rows must not consume budget.
  4. Re-run pnpm social:login once to grant youtube.readonly, after which YouTube snapshots start appearing.

🤖 Generated with Claude Code

@vercel

vercel Bot commented Aug 28, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
zap-engine-frontend Ready Ready Preview Aug 29, 2026 11:26pm
zap-engine-landing-page Ready Ready Preview Aug 29, 2026 11:26pm

i-xtsu-sixyou-ken-mei and others added 5 commits August 29, 2026 07:47
A 13k-character article legitimately generates for minutes, so the shared
120s OpenRouter deadline was killing correct work and the shared retry then
replayed the identical request for another 120s. Script generation now runs
on a private 600s deadline, treats a timeout as terminal, and re-routes a
gateway failure exactly once with `sort` dropped so OpenRouter picks a
different endpoint instead of the one that just refused.

The deadline is a constant rather than an env var: it is a property of the
prompt, not of a deployment, and an unregistered variable would also fail
the dead-env gate. The payload re-prompt is kept -- it carries the rejection
reason, so it is a correction, not a replay. Each upstream request now
reports an attempt record so the ops ledger can hold per-attempt timing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 2026-08-28 timeout left no trace: `pipeline_stage_runs` could always
express a failed attempt, but nothing produced one, so four minutes of
generation looked like a stage that never started. The language that dies
mid-run now pushes its own sink entry, which is also the only place the run
row could have got an episode id from, and script rows are written from
attempt records rather than cost lines -- a cost line has no room for a
deadline, a route, or a failure reason.

The terminal Sentry boundary now names the run and the episode, and drains
before the process can exit: the original event named neither, so it could
not be matched against the logs that explained it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four release cohorts a day meant eight to eleven posts a day, and the
production reach medians say the marginal ones reached nobody. Each platform
now has its own daily cap and candidate slots -- five posts a day in total --
and each `(episode, platform)` cohort takes one of them independently, so
Rednote at 14:30 and YouTube at 17:15 are the same article, hours apart, on
purpose. Language lanes of one platform still share a slot, and a cap counts
cohorts per JST day across every language, so a multilingual platform cannot
publish once per language and call it one post.

The policy is code, not configuration. Timing lived in
`social_strategy_versions` and was never read -- the scheduler always used its
own defaults -- so keeping it there would let a learner widen the very cap it
is optimising inside. The strategy config now carries copy guidance only, and
a refresh retires any active lane the policy no longer ships.

Three mechanisms are removed rather than adapted. Cohort alignment pulled
every lane back to one timestamp, which is the opposite of a per-platform
budget. The partial-cohort fence held every other episode shut until a
half-released one finished, and under these budgets a partial cohort is the
steady state -- fencing on it would deadlock the queue against its own
schedule. `skipOverdueSocialPublishJobs` marked a missed lane `completed`
with a `skipped: overdue` note, recording a post that never existed, behind
an environment variable that was unset in practice; a missed slot is now
moved to the next free one. Publishing runs only inside working hours,
because Rednote and X drive browser sessions someone has to watch fail.

YouTube distributes in English only, and now reports its own subscriber
count -- per-post `subscribersGained` cannot be summed into one. Account
snapshots move to a three-hour cadence and open a browser only when something
that needs one is due.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Moving lanes individually would drift the language lanes of one platform
apart, which is the exact failure the shared `(episode, platform)` slot
exists to prevent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@i-xtsu-sixyou-ken-mei i-xtsu-sixyou-ken-mei changed the title fix(podcast): make long scripts single-attempt and cap social publishing fix(podcast): give long scripts their own request policy; cap social publishing per platform Aug 28, 2026
@i-xtsu-sixyou-ken-mei
i-xtsu-sixyou-ken-mei merged commit 3822e93 into main Aug 29, 2026
9 of 11 checks passed
@i-xtsu-sixyou-ken-mei
i-xtsu-sixyou-ken-mei deleted the fix/pipeline-llm-social-schedule branch August 29, 2026 23:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant