Sync native GGUF support with current upstream types

[Qubitium]

Summary

Synchronize GPTQModel's native GGUF tensor IDs, storage layouts, codecs, and runtime routing with llama.cpp 3ad1ba733 / gguf-py 0.19.0, while retaining Prism/Bonsai Q1_0_g128 as an alias for the official 128-element Q1_0 layout.

What Changed

• Match current upstream IDs and block sizes for NVFP4=40, Q1_0=41, and Q2_0=42; add reader/checkpoint coverage for the updated layouts.
• Add native NumPy quantization/dequantization for Q2_0, TQ1_0, TQ2_0, and MXFP4, plus NVFP4 dequantization; make official Q1_0 use the existing compatible 128-element sign-only codec.
• Pack Q2_0 as four consecutive weights per byte, matching llama.cpp, and preserve arbitrary leading dimensions when dequantizing all newly added formats.
• Match upstream roundf semantics at float32 half-way boundaries for Q2_0, TQ1_0, and TQ2_0 fallback quantization.
• Add public q2_0 configuration support and route current Q1/Q2 checkpoints only to native backends that implement their layouts.
• Keep accelerated C++/CUDA support unchanged; those bridges do not currently expose the new formats. NVFP4 export is intentionally out of scope.

Tests

• I added a new simple/fast unit test for this change, or documented why that is not applicable.
• I ran the new targeted test locally before opening this PR.
• I ran any other directly relevant local tests.

cd tests
python -m pytest test_internal_gguf.py -x -q
# 19 passed
python -m pytest test_internal_gguf.py test_weight_only_config.py qcfg/test_config_dispatch.py -x -q
# 64 passed before the final three threshold cases were added
python -m pytest test_local_model_paths.py -k 'native_bonsai or local_gguf or gguf_file' -x -q
# 7 passed, 16 deselected
python -m pytest test_weight_only.py -k 'gguf_dequantize_weight_accepts_requested_dtype_and_device or gguf_triton_q1_0_g128_fused_forward_matches_dense_baseline' -x -q
# 3 passed, 1 skipped, 76 deselected
python -m pytest test_gguf_qlinear_llama.py -x -q
# 2 skipped: optional model/runtime fixture unavailable
python -m pytest kernels/test_gguf_cpp.py -x -q
# 13 skipped: optional llama.cpp/CUDA bridge unavailable

cd ..
ruff check --select E4,E7,E9,F gptqmodel/utils/internal_gguf.py gptqmodel/nn_modules/qlinear/gguf.py gptqmodel/nn_modules/qlinear/gguf_triton.py gptqmodel/models/loader.py gptqmodel/quantization/config.py tests/test_internal_gguf.py tests/test_weight_only.py tests/test_weight_only_config.py
# All checks passed

Randomized TQ1_0, TQ2_0, MXFP4, and NVFP4 codec results match the cited upstream gguf-py implementation. Q2_0 encoding and decoding also match the cited llama.cpp consecutive-weight byte layout and its independent 0x64 fixture. The float32 threshold regression also matches system C roundf immediately below, at, and above both +0.5 and -0.5.

The full test_weight_only.py suite reaches a pre-existing environment failure because TorchInductor cannot compile without Python.h; the same first test fails on an untouched main worktree.

Review Requirements

AI-assisted code is welcome.

Every changed file must still be properly reviewed by a human before the PR is opened as ready for review.

We will not accept PRs that are effectively unreviewed AI output. Non-human-reviewed changes often introduce obscure structure, mismatched APIs, project-inconsistent code patterns, or unnecessary monkeypatching instead of a correct fix or clean feature expansion.

• I personally reviewed every file in this diff.
• I checked that the code matches existing project structure, APIs, and conventions.
• I avoided unnecessary monkeypatching and used the project's normal extension points where possible.

Notes

Raw tensor type 40 now follows the official NVFP4 assignment. The former private 32-element Q1_0 interpretation cannot coexist under that ID; Prism/Bonsai artifacts using type 41 remain compatible through the Q1_0_g128 alias.

Link to Devin session: https://app.devin.ai/sessions/983b34b989f842989aceb3bccefdf8c2
Open in Devin Desktop: https://app.devin.ai/desktop/session/983b34b989f842989aceb3bccefdf8c2?variant=devin
Requested by: @Qubitium

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[Qubitium]

Reviewed head ca19af627126b03f3db5e64411c325cdff5ba9db. Two actionable correctness issues:

1. [P1] Q2_0 packing and decoding disagree with the upstream byte layout

Locations: encoder, lines 213–214, decoder, lines 674–675.

The new encoder combines weights j, j+16, j+32, j+48 into byte j; the decoder reverses that same layout. However, llama.cpp at the exact upstream revision cited in this PR packs four consecutive weights per byte: byte_index = j / 4, bit_offset = (j % 4) * 2. See upstream quantize_row_q2_0_ref and dequantize_row_q2_0.

Concrete reproduction: a scale-1 block containing [-1, 0, 1, 0] repeated 16 times should contain sixteen payload bytes equal to 0x64. This PR instead produces [0x00, 0x55, 0xAA, 0x55, ...]. Feeding the correct upstream block into the PR decoder yields 40 incorrect values out of 64.

This silently permutes weights when loading upstream Q2_0 checkpoints and makes native Q2_0 output incompatible with upstream readers. The added hand-packed Q2_0 test encodes the same incorrect strided layout, so it currently enshrines the bug.

Please pack with a (n_blocks, 16, 4) grouping and unpack each byte along its four successive two-bit fields. Replace the current test with an independently generated upstream-layout fixture, and check both directions; a local encode/decode round trip alone cannot detect this.

2. [P2] New dequantizers do not preserve the tensor's leading dimensions

Locations: new decoders, lines 670–735, especially each rows = qweight.shape[0] / reshape(rows, -1) pair.

All five new decoders assume a 2D matrix, but GGUFReader preserves the tensor's rank and both internal_gguf.dequantize and dequantize_to_torch pass that data straight through. The result is an exception for a valid one-dimensional packed tensor and a flattened shape for higher-dimensional tensors, including stacked expert weights.

Reproduced directly against the PR functions:

Format	Packed input	Actual result	Expected shape
Q2_0	(18,)	ValueError	(64,)
TQ1_0	(54,)	ValueError	(256,)
TQ2_0	(66,)	ValueError	(256,)
MXFP4	(17,)	ValueError	(32,)
NVFP4	(36,)	ValueError	(64,)
MXFP4	(2, 3, 17)	(2, 96)	(2, 3, 32)
NVFP4	(2, 3, 36)	(2, 192)	(2, 3, 64)

Please restore (*qweight.shape[:-1], qweight.shape[-1] // type_size * block_size) after block decoding and validate the last-axis byte width. Add 1D and 3D coverage through the public internal runtime API; the new minimal NVFP4 reader test currently checks storage only, without dequantizing its 1D result.

Validation: reviewed all nine changed files and relevant loader/runtime call sites; executed the actual NumPy codec functions extracted from the PR via AST, without importing the GPU-heavy package. The Q2_0 and rank failures above reproduced. On 64 sampled blocks per format, TQ1_0/TQ2_0/MXFP4 quantization and dequantization, plus NVFP4 dequantization, matched the cited upstream Python block codecs exactly for 2D inputs. Full package tests and GPU inference were not run in this environment.

[devin-ai-integration]

Addressed P1 in 351c9f90: Q2_0 now packs and decodes four consecutive values per byte. The regression fixture is independently specified as sixteen 0x64 payload bytes and checks both encoding and decoding.

[devin-ai-integration]

Addressed P2 in 351c9f90: Q2_0, TQ1_0, TQ2_0, MXFP4, and NVFP4 now preserve arbitrary leading dimensions and reject invalid last-axis byte widths. Public API tests cover 1D and 3D inputs, including the reader's 1D NVFP4 tensor.

[Qubitium]

Reviewed current head 351c9f90d176f6bd0101d06d5fe15bb700166e3a.

The earlier P1 Q2_0 byte-layout and P2 dequantized-shape findings are fixed. The independent 0x64 Q2_0 fixture passes in both directions, and all five new decoders preserve 1D/2D/3D shapes and reject invalid last-axis byte widths.

One remaining low-priority correctness issue:

[P3] Preserve float32 rounding immediately below the half-way threshold

Location: _gguf_roundf, lines 204–205.

floor(abs(values) + 0.5) is not exactly equivalent to upstream's rounding for float32 inputs: the addition can round to 1.0 before floor runs. For v = np.nextafter(np.float32(0.5), np.float32(0)) (0.49999997), this helper returns 1, whereas C roundf and the cited gguf-py np_roundf return 0; the negative counterpart similarly becomes -1 instead of zero.

Reproduced with a block beginning [1.0, v, -v] and zero-filled remaining entries:

Format	Native decoded first three weights	Upstream expected
Q2_0	[1, 1, -1]	[1, 0, 0]
TQ1_0	[1, 1, -1]	[1, 0, 0]
TQ2_0	[1, 1, -1]	[1, 0, 0]

This is a narrow boundary case, but it breaks exact upstream quantization parity and can make native fallback output differ from the optional upstream codec. Please use upstream's fractional-part formulation (a = abs(x); f = floor(a); sign(x) * (f + floor(2 * (a - f)))) and add regression coverage for the floats immediately below, at, and above both half-way thresholds. References: gguf-py rounding, Q2_0 C reference.

Validation: reviewed all nine changed files and relevant codec/runtime/backend call sites. Executed the actual NumPy functions extracted from the current source via AST. Compared 256 random blocks per format for TQ1_0/TQ2_0/MXFP4 encoding and decoding, and NVFP4 decoding, against the PR's pinned upstream implementation; those comparisons passed. Confirmed the threshold discrepancy against upstream Python codecs and system C roundf. Ruff CI is successful. Full package tests, Torch execution, and GPU inference were not run locally because PyTorch is unavailable.

No additional high- or medium-priority findings in this pass.

[devin-ai-integration]

Fixed in d6b331ac. _gguf_roundf now uses the pinned gguf-py fractional-part formulation, with fallback codec regressions for Q2_0, TQ1_0, and TQ2_0 immediately below, at, and above both float32 half-way thresholds. The focused suite passes (19 tests), Ruff passes, and the helper matches system C roundf for all six boundary values.