Skip to content

feat(renderers): add Inkling and Inkling-Small - #103

Merged
hallerite merged 6 commits into
mainfrom
feat/inkling-renderer
Aug 12, 2026
Merged

feat(renderers): add Inkling and Inkling-Small#103
hallerite merged 6 commits into
mainfrom
feat/inkling-renderer

Conversation

@hallerite

@hallerite hallerite commented Jul 16, 2026

Copy link
Copy Markdown
Member

Summary

Adds one InklingRenderer shared by both Thinking Machines checkpoints:

  • thinkingmachines/Inkling
  • thinkingmachines/Inkling-Small

The two pinned checkpoints ship byte-identical chat templates, tokenizer assets, and processor configuration, so they intentionally use the same renderer and config. Each model ID is independently covered by the shared render, config, and multimodal parity matrices.

This PR is stacked on #123. Merge the Gemma 4 PR first, then this PR.

What it does

  • Text, tools, reasoning, post-tool thinking, and reasoning_effort render with strict chat-template parity.
  • Image and audio rendering matches the native InklingProcessor, including dynamic placeholder expansion and MultiModalData anchoring.
  • parse_response preserves native JSON tool arguments and separates reasoning/content channels.
  • bridge_to_next_turn carries multimodal state without mutating prior data.

Dependency

Keeps the repository-wide transformers>=4.50.0 floor so renderers remains co-installable with prime-rl's current Transformers pin. Inkling tokenizer and text-only rendering stay usable on older releases. Image/audio processing fails lazily with a clear upgrade message when native InklingProcessor support is unavailable (introduced in Transformers 5.14). The lock continues to resolve to transformers==5.14.1.

Review follow-up

  • unknown content-part types and malformed/non-object tool arguments now raise instead of silently dropping data
  • audio preprocessing uses the processor's public audio= call
  • audio has an independent audio_cache_max bound
  • both Inkling checkpoints now run in the shared bridge and round-trip matrices

Verification

  • full combined Gemma 4 + Inkling stack: 3,331 passed, 173 skipped, 1 xfailed
  • Ruff check, Ruff format check, and git diff --check pass
  • focused Inkling bridge/round-trip suite: 59 passed, 2 expected skips
  • Inkling multimodal suite: 13 passed, 4 expected skips
  • all commits in the stack are signed with hallerite <git@hallerite.com> using the configured SSH signing key

Note

Medium Risk
Large new renderer (~1k lines) on the critical render/parse/bridge path; image/audio depend on Transformers 5.14 at runtime though the declared floor stays lower.

Overview
Adds InklingRenderer for thinkingmachines/Inkling and Inkling-Small, sharing one implementation because both checkpoints use identical template/tokenizer/processor assets.

The renderer matches apply_chat_template for text, tools, reasoning_effort (injected as Thinking effort level: {N}), and assistant reasoning blocks; image/audio paths expand placeholders like native InklingProcessor (Transformers ≥5.14) and attach MultiModalData. parse_inkling decodes completions into content, reasoning, and native-JSON tool calls. bridge_to_next_turn uses thinking retention all, merges multimodal sidecars without mutating prior data, and refuses extensions that need unresolved tool names.

InklingRendererConfig adds reasoning_effort, image_cache_max, and audio_cache_max. The package keeps transformers>=4.50.0 for co-install with prime-rl; multimodal fails lazily with an upgrade hint when InklingProcessor is missing. Docs, registry, and shared parity/bridge/multimodal tests are wired for both model IDs; uv.lock bumps to transformers==5.14.1.

Reviewed by Cursor Bugbot for commit de48898. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Add Inkling and Inkling-Small renderers with image/audio multimodal support

  • Introduces InklingRenderer in renderers/inkling.py with full rendering, parsing, bridging, and multimodal (image + audio) support for thinkingmachines/Inkling and thinkingmachines/Inkling-Small.
  • Adds Gemma4Renderer in renderers/gemma4.py with image support and model-specific empty-thought prefill handling for google/gemma-4-* checkpoints.
  • Registers both renderers in renderers/base.py and renderers/init.py with auto-resolution from model name and multimodal capability declarations.
  • Adds InklingRendererConfig (with reasoning_effort, image_cache_max, audio_cache_max) and Gemma4RendererConfig (with enable_thinking, preserve_thinking, image_cache_max) to renderers/configs.py.
  • Adds parse_inkling to renderers/parsing.py for decoding Inkling model outputs into structured ParsedResponse with tool call extraction.
  • Extends renderers/client.py with _build_gemma4_features to serialize Gemma 4 image inputs into vLLM-compatible kwargs; raises RuntimeError if torch or required vLLM modules are absent.
  • Risk: Inkling image/audio processing and Gemma 4 image preprocessing require transformers>=5.14; text-only rendering works on older releases.

Macroscope summarized e22d00f.

@macroscopeapp

macroscopeapp Bot commented Jul 16, 2026

Copy link
Copy Markdown

Approvability

Verdict: Needs human review

This PR introduces a substantial new renderer (~1000 lines of code) supporting Inkling/Inkling-Small models with multimodal capabilities (image and audio), new configuration options, and new parsing logic. The scope and complexity of this new feature warrants human review.

No code changes detected at de48898. Prior analysis still applies.

You can customize Macroscope's approvability policy. Learn more.

Comment thread renderers/inkling.py
Comment thread renderers/inkling.py
@hallerite
hallerite requested review from eligotts and snimu July 22, 2026 21:58
@hallerite
hallerite force-pushed the feat/inkling-renderer branch from f235f2a to 7a17ef0 Compare August 11, 2026 21:35
@hallerite hallerite changed the title feat(renderers): add Inkling renderer (text + image + audio) feat(renderers): add Inkling and Inkling-Small Aug 11, 2026
@hallerite
hallerite changed the base branch from main to codex/add-gemma4-renderer August 11, 2026 21:35
@hallerite
hallerite requested a review from eexwhyzee August 11, 2026 21:42

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 7b2a55b. Configure here.

Comment thread renderers/inkling.py
Base automatically changed from codex/add-gemma4-renderer to main August 12, 2026 18:46
hallerite and others added 6 commits August 12, 2026 11:46
Implement InklingRenderer for thinkingmachines/Inkling — a token-delimited
vision+audio model whose chat template is a flat stream of role-tagged,
<|end_message|>-terminated blocks (no BOS) with a one-off "Thinking effort
level: {N}" system line and <|content_model_end_sampling|> as the assistant
close/eos.

- Text / tools / reasoning / reasoning_effort: byte-parity with
  apply_chat_template across all shapes (effort labels + raw floats, tool
  cycles, multi-tool-call, content=None, tool-name resolution via
  tool_call_id).
- Image + audio: byte-parity with the native InklingProcessor, including the
  Qwen-VL-style placeholder expansion (image -> num_patches image pads, audio
  -> one audio pad per mel frame); ships pixel_values / audio_input_ids in
  MultiModalData with exact placeholder anchoring.
- Full parse_response (native-JSON tool args, types preserved) and
  bridge_to_next_turn (mm carry-forward; refuses tool messages needing
  cross-turn name resolution).

The InklingProcessor is native in transformers >= 5.14 (merged upstream
2026-07-15), so it loads without trust_remote_code. Bump the transformers
floor to >=5.14.0 and fast-track it past the 7-day uv exclude-newer cooldown
(drop the fast-track once 5.14.x clears the window). The full existing suite
was re-validated on 5.14.1 with no regressions.

Wiring: InklingRendererConfig (reasoning_effort str|float, validated),
MODEL_RENDERER_MAP, MULTIMODAL_MODELS={"image","audio"}, registry, __init__
exports. Tests: tests/test_inkling.py plus Inkling rows in the shared
render/config-parity/multimodal matrices (extended with an "inkling" family
and an audio modality kit). Docs updated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Condense the verbose rationale comments added with the Inkling renderer
(MODEL_RENDERER_MAP / MULTIMODAL_MODELS entries, INKLING_EFFORT_MAP, the
transformers pin, and the shared-test rows) to terse one-liners matching
the surrounding style. Comments only — no behavior change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- bridge_to_next_turn copied only the outer mm_data dicts, so appending a
  carried-forward modality's items mutated the caller's
  previous_multi_modal_data lists in place. Copy the per-modality lists too.
  Adds a regression test (test_inkling_bridge_does_not_mutate_prior_mm_data).
- _audio_hash now folds in the sampling rate, so mm_hashes["audio"] uniquely
  identifies the clip (matching the processor cache key); identical samples
  at different rates no longer collide as media IDs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@hallerite
hallerite force-pushed the feat/inkling-renderer branch from e22d00f to de48898 Compare August 12, 2026 18:46
@hallerite
hallerite merged commit f9b37b6 into main Aug 12, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants