Skip to content

feat: add mmx speech transcribe for speech-to-text (asr-1.0) - #262

Merged
SaladDay merged 6 commits into
MiniMax-AI:mainfrom
Wzdhehe:feat/speech-transcribe
Sep 19, 2026
Merged

SaladDay merged 6 commits into
MiniMax-AI:mainfrom
Wzdhehe:feat/speech-transcribe

Conversation

@Wzdhehe

@Wzdhehe Wzdhehe commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

Summary

mmx can synthesize speech but cannot transcribe it. The platform exposes POST /v1/speech_to_text (ASR, asr-1.0), so Token Plan users have no way to reach it from the CLI — you have to drop to curl with a raw HTTP call. This adds the missing command:

mmx speech transcribe --file meeting.mp3
mmx speech transcribe --file call.mp3 --language zh
mmx speech transcribe --file talk.mp3 --response-format verbose_json --output json
mmx speech transcribe --file talk.mp3 --response-format srt --out talk.srt
mmx speech transcribe --file long.mp3 --stream

What it does

  • Uploads the audio as multipart/form-data to /v1/speech_to_text, reusing the existing region base URL and API key resolution. --model, --response-format, --language, and --timestamp-level map 1:1 to the API fields.
  • --response-format json (default) prints the transcript, verbose_json returns speakers and per-segment timestamps, and srt / vtt return subtitle documents (printed or saved verbatim).
  • --stream prints incremental text; --out <path> writes the result to a file.
  • SDK parity: sdk.speech.transcribe({ file, language }), with overloads for streaming (AsyncGenerator<SpeechToTextStreamEvent>) and subtitle formats (string).
  • Documents the command in README, README_CN, SDK.md, ERRORS.md, the agent skill, and the CLI design tree, and registers speech recognize as an alias.

Notes for reviewers

  1. language is sent as a request header. The API documents it as an in: header parameter. Verified against the live API: a form field is silently ignored (an invalid tag zz still returns HTTP 200 plus a transcript), while the header is honoured (zz → HTTP 400 invalid language "zz": must be a valid BCP-47 language tag). A test pins the header placement so a future refactor cannot regress it.

  2. 50 MB is checked locally. src/utils/stt.ts rejects oversized audio before the upload, because a 500 s 48 kHz stereo WAV (~92 MB) would otherwise be uploaded in full only to come back as HTTP 413. The server stays the authority for the 500 s duration limit and codec support — duration is not knowable without decoding the file, so that error is surfaced verbatim from the API.

  3. --stream is opt-in. mmx text chat auto-streams in a TTY; for ASR the whole transcript arrives in one response, so streaming is left to an explicit flag and the default path stays single and predictable. Behaviour while streaming follows text chat: in text mode the deltas go to stdout; when stdout is not a terminal (i.e. --output json) they accumulate into one JSON result instead.

  4. --stream with --out errors rather than silently ignoring --out, so nobody assumes a file was written. The hint points at --output text for piping, since non-TTY stdout defaults to json.

  5. --model and --timestamp-level are passed through unvalidated while --response-format is validated locally. That is deliberate: the CLI interprets the response format (it decides whether the body is a subtitle document), whereas the other two are opaque to it, and the API validates them with a clear message. It also means a new timestamp level does not need a CLI release.

Verification

Check Result
bun run typecheck clean
bun run lint clean, no new warnings
bun test 553 pass; the 7 failures in test/agent/* and test/files/download.test.ts are pre-existing Windows-only failures that reproduce unmodified on bfbb4cb
bun run build succeeds; bun build src/main.ts --compile smoke binary runs --version

The command was also exercised end-to-end against the live API on the cn region with real audio: default json, verbose_json + word timestamps, srt to file (valid UTF-8, no BOM), --stream, the recognize alias, a positional audio path, and the error paths (missing --file, nonexistent path, unsupported format, --stream + srt, --stream + --out, non-audio input, invalid language tag) — all exiting with the documented codes (2 for usage, 1 for API errors).

The change went through two rounds of independent blind review (repo standards/smells, and API-contract fidelity), plus a final single blind pass. Findings that were fixed:

  • the SDK parsed srt / vtt bodies as JSON and failed — those formats now resolve to the subtitle document;
  • the SDK did not enforce the 50 MB limit, and did not validate response_format, so the CLI and SDK disagreed;
  • the multipart field set was assembled twice in each layer — now built once by sttFormFields();
  • the stream loop ignored the API's final event: it now stops on finish and warns when a stream ends without it, rather than silently returning a possibly truncated transcript;
  • --out disk-full failures and a missing ERRORS.md row;
  • docs over-claimed that the 500 s limit is rejected locally, and did not say that n_speakers / segments require --output json.

Deliberately kept (raised in review, decided against) — all documented above: no local duration check, arrival-order delta assembly (text chat behaves the same; SSE over one connection is ordered), pass-through --model / --timestamp-level, --stream + --out mutually exclusive, and explicit --stream following the CLI-wide non-TTY → json output rule.

One review concern turned out to be a non-issue when checked against the live API: the SDK names Blob inputs audio (no extension), and the API does not depend on the multipart filename — audio, audio.bin, and probe.mp3 all return the same transcript.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

Adds a speech-to-text command on top of POST /v1/speech_to_text:

  mmx speech transcribe --file <path> [--language zh] [--response-format srt --out a.srt]

- uploads the audio as multipart/form-data, reusing the existing region base URL
  and API key resolution
- --response-format json (default) / verbose_json / srt / vtt
- --language is sent as a request header, matching the API contract; the same
  value sent as a form field is accepted but silently ignored by the API
- --stream pipes incremental text (json only); --out writes the result to a file
- rejects audio above the documented 50 MB limit locally instead of uploading it
  only to come back as HTTP 413
- registers speech recognize as an alias, and documents the command in README,
  README_CN, SDK.md, ERRORS.md, the agent skill, and the CLI design tree
Blind review found one hard contract bug and several consistency issues:

- SDK: `response_format: 'srt' | 'vtt'` returned a subtitle document, but the
  SDK parsed the body as JSON and failed. It now resolves to that document as a
  string through a dedicated overload, matching what the CLI already did.
- SDK now enforces the documented 50 MB limit for both paths and Blobs, so the
  limit no longer lives only in the command.
- The multipart text parts are built once by `sttFormFields()` and shared by the
  CLI (dry-run preview and request) and the SDK, instead of assembling the same
  field group twice in each.
- Subtitle detection moved next to the format lists in `utils/stt.ts` instead of
  string comparisons spread across the command.
- `--response-format` now reports `Invalid response format` rather than the
  audio-format wording borrowed from TTS.
- The streaming path reports the audio duration on stderr, which was previously
  discarded in text output mode.
- Documented the `speech recognize` alias, and clarified in the skill that
  piped `--stream` output needs `--output text` (non-TTY defaults to json).
- Tests reuse the shared mock-server helpers and cover the new SDK subtitle path,
  the SDK size guard, and the streamed duration.
Second review round, second axis (API contract fidelity):

- The stream loop now stops on the API's final `finish` event, and warns on
  stderr when a stream ends without it, instead of silently returning a possibly
  truncated transcript.
- `--out` write failures no longer surface as a raw filesystem error for the
  disk-full case; ERRORS.md documents both outcomes.
- The SDK validates `response_format` too, so the CLI and SDK reject the same
  values instead of only the CLI doing so.
- Documentation no longer implies the 500 s limit is checked locally (only the
  50 MB size limit is), and says `n_speakers` / `segments` need `--output json`.
- `validateSttFileSize` takes a display label rather than a path, since the SDK
  passes a Blob filename; `utils/stt.ts` no longer borrows `formatList` from the
  TTS module.
Final blind review flagged that stopping at `finish: true` leaves the response
body undrained, which can hold the connection (and the process) open. The body
is now cancelled in a `finally`, matching the idiom already used by
`agent/verify.ts`.

Also records why `--timestamp-level` and `--model` are passed through instead of
validated locally, so the omission reads as a decision rather than a gap.
…utput fidelity

Review follow-ups on speech transcribe:

- SDK validation now raises SDKError, not CLIError: the shared stt
  validators take the caller's error class, so consumers narrowing on
  SDKError no longer miss format/size/stream-combo rejections.
- The SDK stream generator ends at the API's final event (finish=true)
  and releases the SSE body, mirroring the CLI fix; breaking out early
  is safe. CLI and SDK now share one submission path (submitSttForm),
  so the SSE / subtitle / json dispatch cannot drift between layers.
- Stream deltas are assembled by index per the API contract: the CLI
  warns on out-of-order events and the json result is index-sorted.
- srt/vtt saved with --out are byte-exact; only stdout keeps the
  trailing-newline convention.
- mapApiError keys speech-to-text 422 (sensitive audio) off the
  endpoint + status instead of message text, and gives 413 a dedicated
  size-limit message with exit code 2.
- withStubbedFetch extracted to test/helpers; help table alignment;
  SDK.md and ERRORS.md document the new behaviour.
Wzdhehe added a commit to Wzdhehe/mmx-asr-cli that referenced this pull request Sep 19, 2026
@Wzdhehe

Wzdhehe commented Sep 19, 2026

Copy link
Copy Markdown
Contributor Author

Pushed 82f8d96, addressing the remaining review findings:

  • SDK error hierarchy: the shared stt validators now raise the caller's error class, so sdk.speech.transcribe rejects with SDKError (not CLIError) for bad formats, oversized audio, and stream/format combos — consumers narrowing on SDKError no longer miss them.
  • SDK stream parity with dda4f86: the streaming generator now ends at the API's final event (finish: true) and releases the SSE body; breaking out early is safe. Verified against the live API (no hang on early break).
  • One submission path: CLI and SDK share submitSttForm(), so the SSE / subtitle / json dispatch cannot drift between layers.
  • Index ordering: stream deltas are assembled by index per the API contract; the CLI warns on out-of-order events and the json result is index-sorted.
  • Byte-exact subtitles: srt/vtt saved with --out keep the API's own bytes (verified byte-for-byte against a raw curl response); stdout keeps the trailing-newline convention.
  • Deterministic error mapping: speech-to-text 422 (sensitive audio) keys off endpoint + status instead of message text (video's 422 handling unchanged), and 413 gets a dedicated size-limit message with exit code 2.

typecheck clean, lint clean (one pre-existing warning), all 79 tests in the touched files pass; full suite has only the previously documented Windows/network pre-existing failures. Live-tested against the cn region: default json, verbose_json + word timestamps, srt-to-file, --stream, --language en/zz (header honoured, 400 on invalid tag), and all documented exit codes.

@SaladDay
SaladDay merged commit 8678a7d into MiniMax-AI:main Sep 19, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants