Play audio and voice-note attachments - #133
Merged
Merged
Conversation
zrmarley
force-pushed
the
zmarley/bot-1934-audio-playback
branch
2 times, most recently
from
September 22, 2026 16:59
c0bf779 to
239cb62
Compare
zrmarley
marked this pull request as ready for review
September 22, 2026 17:08
zrmarley
requested review from
a team,
comp615 and
wesbillman
as code owners
September 22, 2026 17:09
Signed-off-by: Zach Marley <zmarley@squareup.com>
Signed-off-by: Zach Marley <zmarley@squareup.com>
Signed-off-by: Zach Marley <zmarley@squareup.com>
Signed-off-by: Zach Marley <zmarley@squareup.com>
Signed-off-by: Zach Marley <zmarley@squareup.com>
Signed-off-by: Zach Marley <zmarley@squareup.com>
Signed-off-by: Zach Marley <zmarley@squareup.com>
Signed-off-by: Zach Marley <zmarley@squareup.com>
Signed-off-by: Zach Marley <zmarley@squareup.com>
zrmarley
force-pushed
the
zmarley/bot-1934-audio-playback
branch
from
September 22, 2026 20:02
239cb62 to
911cbd9
Compare
wesbillman
approved these changes
Sep 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes BOT-1934 (parent: BOT-1830 Files & media).
#120 has merged, so this PR is now based directly on
main.Problem
Old Buzz voice notes do not travel as
audio/*. The recorder produces WAV locally, then the upload path transcodes to MP4 with a 16×16 black H.264 video track plus AAC audio (the relay's video validator requires a video track), naming the filevoice-note-<ms>.mp4. They therefore classify askind: "video"and render in the video player as a tiny black square. Genuineaudio/*attachments — which the relay currently rejects on upload, so they only arrive from foreign clients — fell through to the file card with no playback.Change
kind: "audio"(the slot BOT-1929 reserved) andduration?: number, documented alongsidemime/size/nameas sender-supplied presentation metadata, plus a sharedMAX_ATTACHMENT_DURATION_SECONDSbound (24h).fold.ts):audio/*(case-insensitive) →audio;video/mp4→audioonly when the detected filename matches old Buzz'svoice-note-*.mp4convention (also case-insensitive). Plainvideo/mp4staysvideo. Parses imetafilenamethrough the existingsafeAttachmentNameand imetadurationunder a strict decimal regex with an upper bound, matching the "untrusted layout input" posture the neighbouringdim/sizefields already document. The element's own reported duration is bounded by the same constant.AudioAttachment(new): visually hidden native<audio preload="metadata">plus play/pause, range seek, and current/total time. The seek control usesstep="any",endedsnaps progress to completion and restores the Play control, wiredurationmakes the slider usable before metadata loads, and the element corrects it on metadata/duration updates while ignoring non-finite values. Errors render an "Audio unavailable" status card, keyed to the failing source so a later source recovers. Audio↔audio playback exclusivity uses a module singleton with lifecycle cleanup owned by the element ref.MessageRow): audio renders inline only when the resolved source is the same-origin media proxy; externalhttps:and unresolved sources fall back to Render and download generic file attachments #120'sFileAttachmentcard. This honours the navigation policy from Render and download generic file attachments #120 — we never ship an inline player the packaged CSP would block.dev/relay-broker.mjs): acceptsaudio/*and routes it through the existing video streaming branch so Range/seeking works, rather than the buffering image branch. Auth, origin pinning,nosniff,MAX_MEDIA_BYTESand teardown are untouched. This also fixes the415audio rejection..audioPlaynow uses semantic roles (--text-inverse,--surface-inverse,--affordance-prominent) instead of neutral ramp tokens. This is a three-line CSS compatibility update intended to paint identically in both light and dark modes.Deliberate non-goals
Waveform and playback speed (old Buzz has both; deferred as decoration — "access to existing recordings" means listenable). Recording (BOT-1935). Uploads (BOT-1927). Relay-side
audio/*upload acceptance (cross-repo, gated on a sanitizer/metadata validator). CSP changes. Cross-type audio↔video exclusivity. No concurrency scheduler: old Buzz caps audio loads at 3 because waveform rendering forces a full download plusdecodeAudioDataper card. Without waveform there is no decode and no full-file download —preload="metadata"fetches headers only, playback streams via Range, and play-exclusivity bounds concurrent streams to one.Test plan
fold.test.ts: voice-note detection from imetafilenameand from the link label; uppercase mime and uppercase filename; plainvideo/mp4stays video;audio/mpeg→ audio; duration validation; control-character filename rejection; name-precedence ordering.AudioAttachment.test.tsx(jsdom + RTL, real React in StrictMode;HTMLMediaElement.play/pausespied since jsdom has no media pipeline): accessible-name toggle, wire duration before metadata,durationchangecorrection, non-finite ignored, slideraria-valuetext, seek setscurrentTime, correct-instance toggle, error → unavailable, source-keyed failure recovery, and singleton-lifecycle regressions.MessageRow.test.tsx: proxy source → player; external https → open card; no source → unavailable card.dev/relay-broker-api.test.mjs:audio/mpeg+ Range → 206 with streamed body,content-rangeandaccept-ranges; oversized non-206 audio → 413; non-media type still 415.Current validation for the latest head (
911cbd99994ea5f9de218a0f6f522ec166591c2b):bin/pnpm design:check✅bin/pnpm typecheck✅check-staged, changed-file Biome) ✅check-push: TypeScript + related Vitest set, 75 files / 1,063 tests; design-system guards) ✅CI for the new head is pending at the time of this body update. Older green validation remains historical evidence only, not the current hosted status.
jsdom cannot cover real AAC-in-MP4 decode, actual Range seeking, or
currentTimeprogression. No browser journey added — there is no existing video-playback spec and no browser-only contract the lower layers can't hold; the decode/seek reality was verified manually during development.Design pack consulted
AGENTS.md,DESIGN.md,tokens/{color,radius,spacing,typography}.json,components/registry.json. The registry contains no audio-player or slider component, and every entry isstatus: "proposed"— so this follows the existing sibling patterns inMessages.module.cssrather than inventing values or importingui/components into an unmigrated surface. Icons arePlayIcon/PauseIconfromsrc/shared/design-system/icons; no new icon exports. Focus rings stay keyboard-only via the existinghtml[data-keyboard-navigation]pattern. Flagged for designer review: player layout, slider styling, and whether a speaker glyph is wanted.Deferred checks
Review
One adversarial round plus a general code review, then two further verification rounds. Notable catches: a dead unmount-cleanup effect for the playback singleton, the error path leaking the singleton, a residual stale-singleton path via fail→recover→unmount, unbounded sender-controlled
durationcollapsing the card layout, the same defect one hop upstream in the element-reported duration, andstartsWith("video/mp4")matching the real IANA typevideo/mp4V-ES. Each lifecycle regression was confirmed to fail against the prior commit before its fix landed.Deferred as follow-ups from review discussion: moving
isProxySourceout ofFileAttachmentinto a shared media/policy module; deduplicating theMessageRowaudio/file fallback blocks; replacing the module-scoped playback singleton with a stateless or lifted-state approach; audio↔video mutual exclusivity; waveform and playback speed if wanted; relay-sideaudio/*upload acceptance.Container duration quirk
Manual testing surfaced a real defect: the scrubber parked short of the end even after the audio had finished. Two compounding causes, both now fixed.
step={1}meant the thumb could only land on whole seconds, so it could never reach a non-integermax— nowstep="any". Separately, these voice notes carry a 16×16 black video track generated at 1 fps, so the MP4 container duration quantizes to whole video frames and can exceed the real audio length by up to about a second; the audio genuinely ends beforeelement.duration.On
ended, the component therefore corrects the displayed duration down to the observed end — but only when the gap falls within that quantization bound (0.05s < gap ≤ 1.5s). A larger gap is treated as suspicious playback rather than a container artifact: the duration is left alone, progress simply snaps to the known total, and laterdurationchange/loadedmetadatasync stays fully functional. Without that upper bound, a truncated or stalled stream firingendedearly would silently relabel a 60-second clip as a 2-second one and latch that value for the life of the source. At most one correction can ever apply, so replay cannot ratchet the duration downward.A possible follow-up: an early
endedwith a large gap is real evidence that playback failed, and currently surfaces nothing to the user.