Inline raw image refs for multimodal rendering

[eligotts]

Summary

Intermediate multimodal storage mode — the inline bundle. Keeps this repo's raw multimodal contract (renderer computes token layout via pure geometry math, no image processor at render time; refs + descriptors instead of processed tensors) but images travel inline as data:image/...;base64 URLs instead of offloaded file:// run assets.

• mm_store: the mmraw ref envelope carries raw_image_data (the inline data URL) instead of raw_image_uri. offload_image_to_run_assets, run_image_dir, and VF_RENDERER_IMAGE_OFFLOAD_DIR are gone. New decode_data_image_url() is the single supported source decode.
• Qwen3-VL / Qwen3.5 / Kimi-K2.5 raw layout describes decode the inline source once for dimensions + sha256 hash; descriptors embed the source.
• client.generate() serializes the inline source into every image slot's ref — every request to /inference/v1/generate carries the full inline payload for every image in the prompt (current and prior turns), and the prime-rl serving layer dedups processor work by content hash.

Relationship to the offload bundle

This PR is cut from the offload branch (#89) and is the storage-layer intermediate: same processor-ownership architecture, no shared image directory. The offload bundle (renderers #89 / verifiers #1746 / prime-rl #2836) layers content-addressed file:// offload on top of this contract.

Companion inline PRs: verifiers feat/v1-inline-raw-images, prime-rl feat/v1-inline-mm.

Validation

• Full suite: 2153 passed, 121 skipped, 1 xfailed (processor-parity tests included, run against the real Qwen3-VL-4B and pinned Kimi-K2.5 processors).

Note

Inline raw image refs as default multimodal output for Qwen and Kimi renderers

• Changes the default multimodal_output for all multimodal renderers from processed pixel-value tensors to raw image references with layout metadata (grid shape, token count, content hash); processed payloads are still available via config.multimodal_output="processed".
• Adds a new mm_store.py module providing standardized helpers to build, serialize, and validate raw multimodal descriptors (raw_mm_item, raw_mm_ref, split_raw_mm_ref).
• Introduces layout math helpers (describe_qwen_image_layout, describe_kimi_image_layout) that compute token counts and grid dimensions from raw image bytes without running an HF image processor.
• _build_vllm_mm_features in client.py is rewritten to be renderer-agnostic, expecting raw mm_item descriptors; Qwen-specific tensor paths are removed.
• Removes image_cache_max from all renderer configs; processor loading is now lazy and internal cache size is fixed.
• Adds a vision optional-dependency group (pillow, torch, torchvision) to pyproject.toml.
• Risk: renderer constructors no longer accept a processor keyword argument; any caller passing processor= will get a TypeError at runtime.

^{Macroscope summarized 20b3f2d.}

Update: unwrapped the ref payload (603118b)

raw_mm_ref serializes its payload as compact JSON instead of base64-wrapping it. The ref travels as a string inside a JSON request body, so the only encoding cost is escaping the payload's own quotes (~200 bytes) — while the base64 wrapper inflated the entire payload, image included, by a third.

Measured on a 75 KiB JPEG: 130.3 KiB → 97.9 KiB per image slot (the 97.7 KiB data URL plus ~200 bytes), a 25% cut on every image slot of every request. Refs parse with a single partition(":") now that the payload contains colons; the base64-alphabet guard retired with the wrapper.

Update: layout knobs sourced from the checkpoint config (9208b40)

Deleted the baked QWEN_VL_IMAGE_LAYOUT / KIMI_K25_IMAGE_LAYOUT constants. The layout spec dataclass is now the canonical knob list per family:

• Renderers fill it from the checkpoint's preprocessor_config.json — a cached raw JSON read (hub_image_processor_config), never a processor instantiation, so render hosts still avoid torch and (for Kimi) trust-remote-code execution. No baked fallback: a missing knob fails loudly.
• One dual-source extractor per family (qwen_layout_from / kimi_layout_from) fills the spec from either the config dict or a live image processor object (prime-rl's adapters use the latter, deleting their hand-duplicated knob lists). Qwen honors both config shapes (top-level min_pixels/max_pixels, or nested size.shortest_edge/longest_edge) and live-processor SizeDict attributes.
• fingerprint() on the spec hashes exactly its own fields, so the knob list and the hash can't drift. Fingerprint values are byte-identical to before.
• The parity test now also asserts our grid math against HF's metadata-only oracle get_number_of_image_patches (Qwen), and asserts config-sourced spec == live-processor spec for both families (this caught a SizeDict extraction bug in review).

New runtime requirement: the tokenizer must carry name_or_path and the checkpoint's preprocessor_config.json must be resolvable (local path, HF cache, or hub); failures surface at first image render.

Full suite: 2153 passed, 121 skipped, 1 xfailed.

Update: synced with main (27fe3c6)

Merged renderers main (5 commits: vLLM token-serving import fix #115, GLM tool-name validation #114, setuptools bump, ruff pins). One conflict, in client.py: main's #115 updated imports inside the processed-tensor encoder this branch deletes by design — kept the raw-ref _build_vllm_mm_features (same resolution as the offload branch #89). Full suite: 2169 passed, 121 skipped, 1 xfailed.

Update: fingerprints deleted; layout resolved once per renderer (ce7078a)

Companion to prime-rl's "trust the train-time checkpoint contract" change: materialize no longer re-checks layout knobs (it keeps output-level grid asserts, which need no knob extraction), so the fingerprint machinery is deleted end to end — no image_layout_fingerprint, no fingerprint() on the layout specs, no fingerprint/layout_fingerprint fields in mmraw: refs or descriptor envelopes (a ref format change; nothing is merged, both bundles move together).

Render-side resolution is also simplified to the cleanest shape:

• hub_image_processor_config is a local-path check plus one cache-aware hf_hub_download (the previous local_files_only two-step re-implemented hub internals).
• Extractors (qwen_layout_from / kimi_layout_from) are Mapping-only — config JSON is the single knob source; the live-processor/SizeDict duck-typing is gone with its only caller.
• Each VL renderer resolves its layout spec lazily once on first image render and stashes it (_raw_image_layout), mirroring the existing lazy _get_processor idiom; describe_*_layout / *_image_item_for_render take the spec directly instead of a model name.

Full suite: 2169 passed, 121 skipped, 1 xfailed.

Update: pruned unread layout knobs and envelope fields (20b3f2d)

Review follow-up to the fingerprint deletion — removed everything only fingerprints consumed:

• Kimi spec drops image_mean/image_std (+ the float_triple validator); Qwen spec drops temporal_patch_size (grid_t is 1 for images). Specs now carry exactly what the geometry math reads.
• Kimi's ref payload drops num_media_tokens (no consumer; adapters read grid_thws only). The descriptor field stays — the parity test asserts it against the processor's media_tokens_calculator.
• The descriptor envelope drops its modality key: an item's modality is its key in MultiModalData.mm_items; only the standalone mmraw: ref carries its own (consumed by the vLLM-front routing check).

Full suite: 2169 passed, 121 skipped, 1 xfailed.

---

Note

High Risk
Changes the multimodal wire contract and vLLM feature serialization; breaks callers that passed processor= to VL renderers or expected processed tensors by default. Raw mode requires inline data URLs and resolvable checkpoint preprocessor configs at first image render.

Overview
Multimodal sidecars default to lightweight raw descriptors instead of shipping pixel_values from the render host. Qwen3-VL, Qwen3.5, and Kimi K2.5 compute placeholder counts from checkpoint preprocessor_config.json (via new mm_store helpers) and embed inline data:image/...;base64 sources in prime_raw_mm_item envelopes; multimodal_output="processed" keeps the old processor/tensor path for SFT.

Inference client wiring changes: generate() builds vLLM features from generic mmraw: refs (no vLLM/torch tensor encoding), carries prior-turn images in every slot, and falls back to prompt_attribution.multi_modal_data when only prebuilt prompt_ids are passed. Bridge turns use shared merge_multi_modal_data.

Config / API: BaseRendererConfig.multimodal_output (raw | processed) flows through AutoRendererConfig; per-renderer image_cache_max and constructor processor= injection are removed. Optional renderers[vision] extra added for processed-mode deps.

^{Reviewed by Cursor Bugbot for commit 20b3f2d. Bugbot is set up for automated code reviews on this repo. Configure here.}

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR introduces a new multimodal architecture that changes how images are processed and transmitted to inference endpoints, defaulting to raw refs instead of processed tensors. The scope spans multiple renderers with substantial new logic. An unresolved comment identifies a potential bug in hub config loading that could affect network-free CI environments.

^{You can customize Macroscope's approvability policy. Learn more.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit ce7078a. Configure here.}

[cursor]

Cache-first hub config load removed

Medium Severity

hub_image_processor_config no longer tries local_files_only=True before a normal hub fetch. For unpinned models this forces an etag network check even when the file is already cached, which breaks network-free CI and can pull a newer preprocessor_config.json than the rest of the local checkpoint — drifting from the knobs the docstring says must stay checkpoint-aligned.

 

^{Reviewed by Cursor Bugbot for commit ce7078a. Configure here.}