cpufeatures: fix x86 CPUID and XSAVE state checks - #1528
Open
eslerm wants to merge 5 commits into
Open
Conversation
Closes RustCrypto#1510 CPUID(1) and CPUID(7, N) were queried unconditionally. Per the Intel SDM a leaf above CPUID(0).EAX does not read as zero: the CPU returns the highest supported basic leaf's data, which `check!` then reads as feature bits. The leaf 7 rows carrying no XSAVE requirement are where this bites, `sha` and `adx` among them, since nothing masks a stale bit once it is read. Leaf 7 subleaf 1 is gated a second time on subleaf 0's EAX, which reports the highest valid subleaf. Follows `std_detect`: https://docs.rs/crate/std_detect/latest/source/src/detect/os/x86.rs
Four rows named a state their instructions do not use. `gfni`, `vaes` and `vpclmulqdq` required `zmm`, so detection failed on parts that have them without AVX-512. None of the three is an AVX-512 feature; Tremont Atom cores, for instance, have `gfni` and no AVX at all. They now require `ymm` for the VEX-256 forms of `vaes` and `vpclmulqdq`, and no state for the legacy-SSE form of `gfni`, as with `pclmulqdq`. `sm3` required only `xmm`, which is the opposite error. Its instructions are VEX.128-encoded, and any VEX encoding faults unless XCR0 bits 1 and 2 are both set, while `"xmm"` checks only bit 1. It now requires `ymm`, the same fix the `avx` row took in RustCrypto#1511. No `"xmm"` rows remain. Each row now certifies its own CPUID bit and the XSAVE state the non-EVEX-512 encodings use, and nothing else. `zmm` previously acted as a de facto proxy for a wider set, since no part enables that state without AVX-512 and therefore without AVX2. It no longer does, so a consumer must gate separately on whatever else the code it dispatches needs: `avx512f` for an EVEX-512 form, and for `vaes` the `avx2` and `aes` that rustc's target feature closure enables inside a `vaes` region. `vpclmulqdq` likewise implies `avx` and `pclmulqdq`. Correct behaviour also requires the max-basic-leaf guard on the CPUID queries: without it, a part whose max basic leaf is below 7 aliases the leaf 7 query, and the relaxed tags no longer mask the aliased bits.
Extract leaf collection into `__collect_leaves`, parameterized over the CPUID sources so the max basic leaf guard can be driven with synthetic data. No CI runner reports a max basic leaf below 7, so the guard is otherwise unreachable under `cargo test` and a later refactor could invert it unnoticed. The XSAVE tag test expands `check!` inside the crate for the first time, which brings `__xgetbv!`'s `unsafe` block under `clippy::undocumented_unsafe_blocks`; hence the SAFETY comment. `mod x86` becomes `pub mod x86`, doc-hidden, matching `aarch64` and `loongarch64`, since the macro now reaches `__collect_leaves` through `$crate::x86::` when expanded downstream.
The `ymm` tag checks that the OS enabled YMM state, not that the CPU reports AVX. `fma` and `avx2` conjoin CPUID.1:ECX[28] alongside their `ymm` tag; do the same here, so a CPUID configuration advertising VAES while masking AVX cannot satisfy the token. No shipping part is known to report VAES without AVX, so this is a no-op on real hardware. It matters under a hypervisor or emulator that masks the two independently. `sha512` and `sm4` are VEX-encoded and still do not conjoin the AVX bit. Left alone here, since they were added as a set and are worth their own change.
The tag column was matched at runtime with a `_ => true` arm, so a mistyped tag did not fail to build. It silently dropped the extended state check for that feature, and `check!` would then report the feature present on a machine whose OS had not enabled the state its instructions need, faulting at the first one. `__xsave_state!` has no catch-all arm, and a `const _` per row runs every tag past the same list where the table is defined. The second is what puts the error here, rather than in whichever downstream crate first happens to check that feature.
eslerm
force-pushed
the
cpufeatures-vaes-xsave-state
branch
from
September 2, 2026 07:42
355fd8e to
c11948b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1510
1. Guard CPUID queries by max basic leaf.
CPUID(1)andCPUID(7, N)werequeried unconditionally. Per the SDM a leaf above
CPUID(0).EAXreturns thehighest supported leaf's data, which
check!then reads as feature bits. Theleaf 7 rows carrying no XSAVE requirement are where this bites,
shaandadxamong them, since nothing masks a stale bit once it is read.
Leaf 1 and both leaf 7 subleaves are zeroed when
CPUID(0).EAXdoes not reachthem. Subleaf 1, which the
sha512/sm3/sm4rows read, is additionally gatedon subleaf 0 reporting a max subleaf of at least 1.
std_detectgates both thesame way:
https://docs.rs/crate/std_detect/latest/source/src/detect/os/x86.rs
2. Match XSAVE tags to the encodings they gate. None of the
three is an AVX-512 feature.
ymmfor the VEX-256 forms, none for thelegacy-SSE form of
gfni, as withpclmulqdq.gfnineeds""rather than"ymm": Tremont Atom has GFNI and no AVX at all.sm3had the opposite error,"xmm"where its VEX.128 encoding needsymm; fixed the same way theavxrowwas in #1511, leaving no
"xmm"rows. Requires the max-basic-leafguard above: without it a part whose max basic leaf is below 7 aliases the leaf
7 query, and the relaxed tags no longer mask the aliased bits.
3. Make the guard and the tags unit-testable. Leaf collection moves into
__collect_leaves, parameterized over the CPUID sources. No CI runner reports amax basic leaf below 7, so the guard is otherwise unreachable in tests.
mod x86becomespub mod x86, doc-hidden, since the macro now reaches thatfunction through
$crate::x86::when expanded downstream;aarch64andloongarch64are already public on the same terms.4. Require the AVX bit for
vaes/vpclmulqdq. Theymmtag checks that theOS enabled YMM state, not that the CPU reports AVX;
fmaandavx2alreadyconjoin both.
sha512andsm4are also VEX-encoded and still do not conjoinit; left alone here, since they were added as a set and are worth their
own change.
5. Make an unknown XSAVE tag a compile error. The tag column was matched at
runtime with a
_ => truearm, so a mistyped tag did not fail to build. Itsilently removed the state check for that feature. Feature names were already
protected this way; the tag column was not.
vaes/vpclmulqdqno longer imply AVX-512 state. A consumer dispatching anEVEX-512 form must gate on
avx512fseparately.Downstream
aesgates its VAES-256 backend on the barevaestoken, so this changes whichbackend it selects on non-AVX-512 parts. That backend batches 30 blocks and does
not override
encrypt_tail_blocks, so shorter buffers fall through tocipher'sone-block-at-a-time default.
Aes128::encrypt_blocks, cycles/byte: below 480 bytes VAES-256 runs up to 34%slower than AES-NI; at and above it, 4 to 5 times faster. The crossover is
exactly the batch width, and the tail cost recurs at every
N mod 30(a45-block buffer is one full batch plus a 15-block tail, and lands at 1.8x rather
than 5x).
Two limits on those numbers. They were taken on Zen 5, which is not one of
the parts this change affects; it was the x86 hardware I had, and a Gracemont or
Zen 3 result may differ, particularly the size of the win. And they force backend
selection rather than masking CPUID, so they measure the two backends, not the
detection.
The tail path is a pre-existing
aesgap. Anencrypt_tail_blocksoverridethere would bound it at parity.
Testing
cargo test -p cpufeaturespasses onx86_64-unknown-linux-gnu; on non-x86targets the module and its tests are
cfg'd out. Three of the seven tests failagainst the pre-fix code.
Detection was checked under masked CPUID; no test in this PR produces these.
Rows 1, 3 and 4 are
qemu-x86_64(qemu-user, TCG) with the named-cpumask;row 2 is a KVM guest.
-cpu Icelake-Server,-avx512fvaes=falsevaes=true-cpu Snowridge(GFNI, no AVX)gfni=falsegfni=true-cpu Icelake-Server,level=1vaes=falsevaes=false-cpu Skylake-Clientvaes=falsevaes=falseThe
level=1row is why the guard comes first. Both shipped states are correctthere; a fourth state this PR never produces, relaxation applied with the guard
absent, reports
vaes=trueon a configuration with no leaf 7 at all.vpclmulqdqwas not measured on any configuration, because QEMU's TCG does notexpose its CPUID bit, and no test covers its tag. Its change rests on the same
encoding argument as
vaes.