Skip to content

Play audio and voice-note attachments - #133

Merged
wesbillman merged 9 commits into
mainfrom
zmarley/bot-1934-audio-playback
Sep 22, 2026
Merged

wesbillman merged 9 commits into
mainfrom
zmarley/bot-1934-audio-playback

Conversation

@zrmarley

@zrmarley zrmarley commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Fixes BOT-1934 (parent: BOT-1830 Files & media).

#120 has merged, so this PR is now based directly on main.

Problem

Old Buzz voice notes do not travel as audio/*. The recorder produces WAV locally, then the upload path transcodes to MP4 with a 16×16 black H.264 video track plus AAC audio (the relay's video validator requires a video track), naming the file voice-note-<ms>.mp4. They therefore classify as kind: "video" and render in the video player as a tiny black square. Genuine audio/* attachments — which the relay currently rejects on upload, so they only arrive from foreign clients — fell through to the file card with no playback.

Change

  • Contract: adds kind: "audio" (the slot BOT-1929 reserved) and duration?: number, documented alongside mime/size/name as sender-supplied presentation metadata, plus a shared MAX_ATTACHMENT_DURATION_SECONDS bound (24h).
  • Classification (fold.ts): audio/* (case-insensitive) → audio; video/mp4audio only when the detected filename matches old Buzz's voice-note-*.mp4 convention (also case-insensitive). Plain video/mp4 stays video. Parses imeta filename through the existing safeAttachmentName and imeta duration under a strict decimal regex with an upper bound, matching the "untrusted layout input" posture the neighbouring dim/size fields already document. The element's own reported duration is bounded by the same constant.
  • AudioAttachment (new): visually hidden native <audio preload="metadata"> plus play/pause, range seek, and current/total time. The seek control uses step="any", ended snaps progress to completion and restores the Play control, wire duration makes the slider usable before metadata loads, and the element corrects it on metadata/duration updates while ignoring non-finite values. Errors render an "Audio unavailable" status card, keyed to the failing source so a later source recovers. Audio↔audio playback exclusivity uses a module singleton with lifecycle cleanup owned by the element ref.
  • Routing (MessageRow): audio renders inline only when the resolved source is the same-origin media proxy; external https: and unresolved sources fall back to Render and download generic file attachments #120's FileAttachment card. This honours the navigation policy from Render and download generic file attachments #120 — we never ship an inline player the packaged CSP would block.
  • Dev broker (dev/relay-broker.mjs): accepts audio/* and routes it through the existing video streaming branch so Range/seeking works, rather than the buffering image branch. Auth, origin pinning, nosniff, MAX_MEDIA_BYTES and teardown are untouched. This also fixes the 415 audio rejection.
  • Compatibility token update (latest commit): .audioPlay now uses semantic roles (--text-inverse, --surface-inverse, --affordance-prominent) instead of neutral ramp tokens. This is a three-line CSS compatibility update intended to paint identically in both light and dark modes.

Deliberate non-goals

Waveform and playback speed (old Buzz has both; deferred as decoration — "access to existing recordings" means listenable). Recording (BOT-1935). Uploads (BOT-1927). Relay-side audio/* upload acceptance (cross-repo, gated on a sanitizer/metadata validator). CSP changes. Cross-type audio↔video exclusivity. No concurrency scheduler: old Buzz caps audio loads at 3 because waveform rendering forces a full download plus decodeAudioData per card. Without waveform there is no decode and no full-file download — preload="metadata" fetches headers only, playback streams via Range, and play-exclusivity bounds concurrent streams to one.

Test plan

  • fold.test.ts: voice-note detection from imeta filename and from the link label; uppercase mime and uppercase filename; plain video/mp4 stays video; audio/mpeg → audio; duration validation; control-character filename rejection; name-precedence ordering.
  • AudioAttachment.test.tsx (jsdom + RTL, real React in StrictMode; HTMLMediaElement.play/pause spied since jsdom has no media pipeline): accessible-name toggle, wire duration before metadata, durationchange correction, non-finite ignored, slider aria-valuetext, seek sets currentTime, correct-instance toggle, error → unavailable, source-keyed failure recovery, and singleton-lifecycle regressions.
  • MessageRow.test.tsx: proxy source → player; external https → open card; no source → unavailable card.
  • dev/relay-broker-api.test.mjs: audio/mpeg + Range → 206 with streamed body, content-range and accept-ranges; oversized non-206 audio → 413; non-media type still 415.

Current validation for the latest head (911cbd99994ea5f9de218a0f6f522ec166591c2b):

  • bin/pnpm design:check
  • bin/pnpm typecheck
  • pre-commit hook (check-staged, changed-file Biome) ✅
  • pre-push hook (check-push: TypeScript + related Vitest set, 75 files / 1,063 tests; design-system guards) ✅

CI for the new head is pending at the time of this body update. Older green validation remains historical evidence only, not the current hosted status.

jsdom cannot cover real AAC-in-MP4 decode, actual Range seeking, or currentTime progression. No browser journey added — there is no existing video-playback spec and no browser-only contract the lower layers can't hold; the decode/seek reality was verified manually during development.

Design pack consulted

AGENTS.md, DESIGN.md, tokens/{color,radius,spacing,typography}.json, components/registry.json. The registry contains no audio-player or slider component, and every entry is status: "proposed" — so this follows the existing sibling patterns in Messages.module.css rather than inventing values or importing ui/ components into an unmigrated surface. Icons are PlayIcon/PauseIcon from src/shared/design-system/icons; no new icon exports. Focus rings stay keyboard-only via the existing html[data-keyboard-navigation] pattern. Flagged for designer review: player layout, slider styling, and whether a speaker glyph is wanted.

Deferred checks

  • Live reconfirmation that the scrubber-short-of-end fix behaves correctly with a real old-Buzz voice note.
  • Light/dark and narrow/wide visual pass.
  • No Play audio and voice-note attachments #133 screenshots were backfilled; the user opted out.
  • Packaged playback is unavailable, consistent with images and video, which are also dev-broker-only today (the packaged transport returns no media URL). Packaged media retrieval wants a Rust-owned URI scheme with Range passthrough — shared platform work owed to images and video, not funded here.

Review

One adversarial round plus a general code review, then two further verification rounds. Notable catches: a dead unmount-cleanup effect for the playback singleton, the error path leaking the singleton, a residual stale-singleton path via fail→recover→unmount, unbounded sender-controlled duration collapsing the card layout, the same defect one hop upstream in the element-reported duration, and startsWith("video/mp4") matching the real IANA type video/mp4V-ES. Each lifecycle regression was confirmed to fail against the prior commit before its fix landed.

Deferred as follow-ups from review discussion: moving isProxySource out of FileAttachment into a shared media/policy module; deduplicating the MessageRow audio/file fallback blocks; replacing the module-scoped playback singleton with a stateless or lifted-state approach; audio↔video mutual exclusivity; waveform and playback speed if wanted; relay-side audio/* upload acceptance.

Container duration quirk

Manual testing surfaced a real defect: the scrubber parked short of the end even after the audio had finished. Two compounding causes, both now fixed.

step={1} meant the thumb could only land on whole seconds, so it could never reach a non-integer max — now step="any". Separately, these voice notes carry a 16×16 black video track generated at 1 fps, so the MP4 container duration quantizes to whole video frames and can exceed the real audio length by up to about a second; the audio genuinely ends before element.duration.

On ended, the component therefore corrects the displayed duration down to the observed end — but only when the gap falls within that quantization bound (0.05s < gap ≤ 1.5s). A larger gap is treated as suspicious playback rather than a container artifact: the duration is left alone, progress simply snaps to the known total, and later durationchange/loadedmetadata sync stays fully functional. Without that upper bound, a truncated or stalled stream firing ended early would silently relabel a 60-second clip as a 2-second one and latch that value for the life of the source. At most one correction can ever apply, so replay cannot ratchet the duration downward.

A possible follow-up: an early ended with a large gap is real evidence that playback failed, and currently surfaces nothing to the user.

@zrmarley zrmarley closed this Sep 22, 2026
@zrmarley zrmarley reopened this Sep 22, 2026
@zrmarley
zrmarley force-pushed the zmarley/bot-1934-audio-playback branch 2 times, most recently from c0bf779 to 239cb62 Compare September 22, 2026 16:59
@zrmarley
zrmarley marked this pull request as ready for review September 22, 2026 17:08
@zrmarley
zrmarley requested review from a team, comp615 and wesbillman as code owners September 22, 2026 17:09
Base automatically changed from zmarley/bot-1929-generic-files to main September 22, 2026 17:32
Signed-off-by: Zach Marley <zmarley@squareup.com>
Signed-off-by: Zach Marley <zmarley@squareup.com>
Signed-off-by: Zach Marley <zmarley@squareup.com>
Signed-off-by: Zach Marley <zmarley@squareup.com>
Signed-off-by: Zach Marley <zmarley@squareup.com>
Signed-off-by: Zach Marley <zmarley@squareup.com>
Signed-off-by: Zach Marley <zmarley@squareup.com>
Signed-off-by: Zach Marley <zmarley@squareup.com>
Signed-off-by: Zach Marley <zmarley@squareup.com>
@zrmarley
zrmarley force-pushed the zmarley/bot-1934-audio-playback branch from 239cb62 to 911cbd9 Compare September 22, 2026 20:02
@wesbillman
wesbillman merged commit 20d7447 into main Sep 22, 2026
12 checks passed
@wesbillman
wesbillman deleted the zmarley/bot-1934-audio-playback branch September 22, 2026 23:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants