TLS Extensions: add more extensions and improve current

[SparkiDev]

Description

Added support for record_size_limit in TLS 1.3 and TLS 1.2. Added compress_certificate support for TLS 1.3.
Added signed_certificate_timestamp TLS 1.2 send and TLS 1.2 and 1.3 receive. Accepts server_name in CertificateRequest.
Added API for setting signature algorithms for signature_algorithms_cert.

Tests added and interop performed where possible.

Testing

Regression tested TLS.

[github-actions]

MemBrowse Memory Report

gcc-arm-cortex-m4-dtls13

• FLASH: .text +256 B (+0.1%, 186,876 B / 1,048,576 B, total: 18% used)

gcc-arm-cortex-m4-openssl-compat

• FLASH: .text +576 B (+0.1%, 783,452 B / 1,048,576 B, total: 75% used)

gcc-arm-cortex-m4-pq

• FLASH: .text +512 B (+0.2%, 302,152 B / 1,048,576 B, total: 29% used)

gcc-arm-cortex-m4-rsa-only

• FLASH: .text +512 B (+0.2%, 332,896 B / 1,048,576 B, total: 32% used)

gcc-arm-cortex-m4-tls13

• FLASH: .text +512 B (+0.2%, 242,639 B / 262,144 B, total: 93% used)

gcc-arm-cortex-m7-pq

• FLASH: .text +512 B (+0.2%, 303,048 B / 1,048,576 B, total: 29% used)

gcc-arm-cortex-m7-tls13

• FLASH: .text +512 B (+0.2%, 242,703 B / 262,144 B, total: 93% used)

linuxkm-standard

• Data: __patchable_function_entries +104 B (+0.2%, 49,888 B)
  No memory changes detected for:
• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m3
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls12
• gcc-arm-cortex-m7
• linuxkm-pie
• stm32-sim-stm32h753

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #11362

Scan targets checked: wolfcrypt-rs-bugs, wolfssl-bugs, wolfssl-src

Findings: 3
3 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #11362

Scan targets checked: wolfcrypt-rs-bugs, wolfssl-bugs, wolfssl-src

Findings: 3
3 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #11362

Scan targets checked: wolfcrypt-rs-bugs, wolfssl-bugs, wolfssl-src

Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #11362

Scan targets checked: wolfcrypt-rs-bugs, wolfssl-bugs, wolfssl-src

Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #11362

Scan targets checked: wolfcrypt-rs-bugs, wolfssl-bugs, wolfssl-src

Findings: 2
2 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

[wolfSSL-Fenrir-bot]

Server discards peer's record_size_limit when its own limit is 0 · TLS protocol issues

On client_hello the parser returns before ssl->peerRecordSizeLimit = limit when ssl->recordSizeLimit == 0. RFC 8449 limits are per-direction, so a server configured with wolfSSL_CTX_UseRecordSizeLimit(ctx, 0) keeps sending full 2^14 records to a peer that advertised a smaller limit.

Related known finding #7004 (similar but distinct): Both concern RFC 8449 record_size_limit handling, but #7004 was the absence of extension registration and parsing in TLSX_Parse, while this is an existing parser in TLSX_RecordSizeLimit_Parse failing to retain the peer's limit under a local-limit condition. The faulting operations and root causes differ, and separate patches are required.

Fix: Skip only the TLSX_Push/TLSX_SetResponse when ssl->recordSizeLimit == 0, and always assign ssl->peerRecordSizeLimit.

[wolfSSL-Fenrir-bot]

Compressed Certificate silently drops the signed_certificate_timestamp response · Cryptographic correctness

The cached ctx->certComp body is built by BuildTls13CertificateBody() with hardcoded empty per-certificate extensions. UseCompressedCertificate() opts out for status_request/status_request_v2 but not for a pending SCT response, so a server with both features enabled sends a CompressedCertificate carrying no SCT list even though it marked the extension as a response.

Fix: Add a HAVE_SIGNED_CERT_TIMESTAMP guard returning 0 when ssl->sctListSz > 0 (or ssl->sctRequested) alongside the existing status_request checks.