fix(bridge): expand grouped K/V heads in QK/OV and composition circuits on GQA models - #1593
Draft
TravisHaa wants to merge 1 commit into
Draft
fix(bridge): expand grouped K/V heads in QK/OV and composition circuits on GQA models#1593TravisHaa wants to merge 1 commit into
TravisHaa wants to merge 1 commit into
Conversation
…ts on GQA models On GQA models, weight circuits built FactoredMatrix objects from per-query-head W_Q/W_O ([n_layers, n_heads, ...]) and grouped W_K/W_V ([n_layers, n_kv_heads, ...]); the leading head axes cannot broadcast, so bridge.QK/OV and all_composition_scores raised on every GQA model. Add _expand_kv_heads(), which repeat_interleaves a stacked grouped K/V head axis up to n_heads (query head h reads kv head h // (n_heads // n_kv_heads), matching HookedTransformer's GroupedQueryAttention layout), and apply it at all seven grouped FactoredMatrix sites: QK, OV, QK_for_attn_layers, OV_for_attn_layers, and the three sites inside all_composition_scores. Raw W_K/W_V stacks and per-block weights stay grouped. Indivisible head counts raise ValueError instead of silently mis-expanding. Remove the inert ungroup_grouped_query_attention kwarg from TransformerBridgeConfig; no bridge code reads it and the only tests exercising the flag build HookedTransformer objects (whose flag and setter are untouched). Tests: synthetic unit tests for the helper; GQA circuit-assembly tests on the tiny random Llama fixture (no Hub access) including composition scores and an MHA bit-identity control; and a slow Qwen2-0.5B parity suite asserting QK/OV match HookedTransformer.from_pretrained_no_processing (exact locally). Fixes TransformerLensOrg#1553
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
bridge.QK,bridge.OV, andbridge.all_composition_scores()crash on every GQA model (Llama, Qwen, Gemma, Mistral, most MoE). The bridge stacksW_Q/W_Oper query head ([n_layers, n_heads, ...]) but keepsW_K/W_Vgrouped ([n_layers, n_kv_heads, ...]), soFactoredMatrixhits a broadcast error on the leading head axes before any analysis can run.HookedTransformeralready expands grouped K/V in its weight properties; the bridge never did.Fixes #1553.
Changes
_expand_kv_heads()intransformer_bridge.py:repeat_interleaves a stacked grouped K/V head axis up ton_heads. Query headhreads kv headh // (n_heads // n_kv_heads)— the same layoutGroupedQueryAttention.W_K/W_Vexpose on HT, and the same routing HF uses at runtime. No-op for MHA and non-4D tensors; raisesValueErrorifn_headsisn't a multiple ofn_kv_headsrather than silently mis-expanding.FactoredMatrixsites:QK,OV,QK_for_attn_layers(),OV_for_attn_layers(), and the three insideall_composition_scores(). The issue lists the first four; composition scores have the identical mismatch (their head labels already assumen_headsper layer), so I fixed them in the same pass rather than leaving GQA composition scores broken.bridge.W_K/W_Vstacks and per-block weights stay grouped — only the circuit views expand.ungroup_grouped_query_attentionkwarg fromTransformerBridgeConfig. No bridge code reads it, and the only tests exercising the flag buildHookedTransformerobjects; HT's flag andset_ungroup_grouped_query_attentionare untouched.Tests
tests/unit/model_bridge/test_expand_kv_heads.py(new, synthetic): the head mapping itself, MHA/non-4D pass-through as the identical tensor, indivisible-countValueError.tests/integration/model_bridge/test_attention_weight_accessors.py(extended, reuses the file's tiny random GQA Llama fixture — no Hub access): QK/OV factors align per query head against the per-block grouped weights, both*_for_attn_layersvariants, composition scores in all three modes, raw storage stays grouped, and an MHA bit-identity control on tiny GPT-2.tests/integration/model_bridge/test_bridge_qk_ov_vs_hooked_gqa.py(new,@pytest.mark.slowsince Qwen2-0.5B isn't CI-cached): bridge QK/OV factors, the layer-0ABproduct, and the*_for_attn_layersvariants matchHookedTransformer.from_pretrained_no_processingviatorch.testing.assert_close.Acceptance criteria from #1553
bridge.QK/OVmatchHookedTransformer.QK/OVwithin fp tolerance on a small GQA model — exact on Qwen2-0.5B locally (max |Δ| = 0.0 on all four factors)QK_for_attn_layers/OV_for_attn_layersreturn the same ungrouped alignmentrg ungroup transformer_lens/model_bridge/ transformer_lens/config/transformer_bridge_config.pyreturns zero hitsuv run mypy .clean (421 files);make formatapplied;patching.pyuntouchedmake unit-test: 5097 passed / 0 failed / 59 skipped / 10 xfailed locally, with one caveat belowLocal caveat: five generation-exercising unit test files SIGBUS on my macOS-arm64 machine. That's pre-existing, not from this diff — it reproduces identically on the clean
dev-4.xbase under both torch 2.7.1 and 2.11.0, and matches the "Upstream PyTorch/HF on M-series Macs" quarantines intests/QUARANTINES.md. The 5097 figure is the tier with those five files excluded; CI should be treated as authoritative for the full tier. I also rantest_analysis_methods.py(22 passed) andtest_attention_weight_accessors.py(13 passed) directly as regressions for the touched surfaces.Relation to #1584
Overlaps — I was assigned on the issue and had this in flight. Beyond the overlap, this PR also fixes the three broken sites in
all_composition_scores(), adds the HookedTransformer-parity test from the acceptance list, keeps therg ungroupcheck at zero hits, and guards indivisible head counts.