Bsdkm driver cleanup

[philljj]

Description

Cleanup for bsdkm driver:

• tests should use CRYPTO_F_CBIFSYNC flag, so callback isn't randomly deferred.
• sanitize ivlen, and auth_mlen.
• handle aes session state and cleanup better.

Fixes F-5305, F-5306, F-5307, F-5930.
Fixes ZD-21992.

Testing

Tested on FreeBSD 14.3-RELEASE.

#!/bin/sh                  
BSDKM_CFLAGS="-DWOLFSSL_BSDKM_VERBOSE_DEBUG -DWOLFSSL_BSDKM_FPU_DEBUG"          
                                                                                
./configure --enable-freebsdkm --enable-freebsdkm-crypto-register \
  --enable-cryptonly --enable-crypttests \
  --enable-kernel-benchmarks --enable-all-crypto --enable-aesni \
  --enable-aesni-with-avx \
  CFLAGS="$BSDKM_CFLAGS" && make \       
  || exit 1

file bsdkm/libwolfssl.ko && sudo kldload bsdkm/libwolfssl.ko || exit 1

#!/bin/sh
./configure --enable-freebsdkm --enable-cryptonly --enable-crypttests \
  --enable-kernel-benchmarks --enable-all-crypto --enable-aesni \
  --enable-aesni-with-avx CFLAGS="-DWOLFSSL_BSDKM_VERBOSE_DEBUG" && make \
  || exit 1
file bsdkm/libwolfssl.ko && sudo kldload bsdkm/libwolfssl.ko || exit 1

[Copilot]

Pull request overview

This PR cleans up the FreeBSD kernel module (BSDKM) crypto driver by adding stricter session parameter validation, improving AES session initialization/teardown behavior, and making the in-kernel AES tests deterministic with respect to callback timing.

Changes:

• Added csp_ivlen / csp_auth_mlen validation for AES-CBC and AES-GCM in probesession and newsession.
• Reworked AES session setup to pre-set keys during session creation and adjusted per-request handling to set IVs and perform cleanup.
• Updated BSDKM AES test requests to include CRYPTO_F_CBIFSYNC to avoid deferred callbacks.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
bsdkm/wolfkmod.c	Adds parameter sanitization for AES sessions and refactors AES init/use/cleanup in CBC and GCM paths.
bsdkm/wolfkmod_aes.c	Updates kernel-mode AES tests to set CRYPTO_F_CBIFSYNC for consistent callback behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

MemBrowse Memory Report

No memory changes detected for:

• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m3
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-dtls13
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-openssl-compat
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-pq
• gcc-arm-cortex-m4-rsa-only
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls12
• gcc-arm-cortex-m4-tls13
• gcc-arm-cortex-m7
• gcc-arm-cortex-m7-pq
• gcc-arm-cortex-m7-tls13
• linuxkm-pie
• linuxkm-standard
• stm32-sim-stm32h753

[philljj]

Retest this please.

Fips harness test hung.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10695

Scan targets checked: bsdkm-bugs, bsdkm-src

No new issues found in the changed files. ✅