Skip to content

Feat/python crypto detection - #495

Open
san-zrl wants to merge 13 commits into
mainfrom
feat/python-crypto-detection
Open

Feat/python crypto detection#495
san-zrl wants to merge 13 commits into
mainfrom
feat/python-crypto-detection

Conversation

@san-zrl

@san-zrl san-zrl commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR adds detection support for the PyCrypto / PyCryptodome library (Crypto.*) and reorganises the existing Python detection rule infrastructure so that both pyca and pycrypto rules share the same translator pipeline.

The rules also work for pycryptodomex which exhibits the exact same API as pycryptodome. The only difference is the import path (Cryptography instead of Crypto). Detection rules check for both possible import paths in forObjectTypes. Rule duplication was avoided by relaxing the type checking (withMethodParameter uses ANY, parameters detected during translation). Only one duplicate rule.

Here's a result that this PR produces with cbomkit when scanning mastercard python client API (no findings before).
image


What changed

1 · New PyCrypto / PyCryptodome detection rules

Eight new detection-rule classes cover the full PyCryptodome API surface:

Rule file Algorithms covered
PythonCryptoCipher AES, 3DES, DES, Blowfish, CAST5, RC2, RC4, Salsa20, ChaCha20, ChaCha20-Poly1305, PKCS1-OAEP, PKCS1v15, HPKE
PythonCryptoHash MD2, MD5, SHA-1/2/3 family, BLAKE2b/s, RIPEMD-160, SHA-512/t, TupleHash128, cSHAKE256
PythonCryptoKDF PBKDF1, PBKDF2, scrypt, HKDF, bcrypt
PythonCryptoKeyAgreement DH, X25519, X448
PythonCryptoMac HMAC, CMAC
PythonCryptoPublicKey RSA, DSA, ECC (with ECCRules() for ECDSA dependency), ElGamal
PythonCryptoSignature DSS (sign + verify), EdDSA (sign + verify), PKCS1v15 (sign + verify), PSS (sign + verify), ECDSA (sign + verify)
PythonCryptoRandom Secure random byte generation

2 · Shared PyCA translators for PyCrypto

pycrypto detections are wired into the existing pyca context translators rather than duplicating translation logic. Three new translators were added to fill gaps:

  • PycaKeyContextTranslator — unified key material handling for both libraries
  • PycaPublicKeyContextTranslator — public-key extraction shared by pyca and pycrypto RSA/DSA/ECC
  • PycaRandomContextTranslator — random context (new, used by PythonCryptoRandom)

3 · Code structure reorganisation

All pyca detection rules and their test input files moved into a dedicated pyca/ sub-package, mirroring the new pycrypto/ layout:

rules/detection/
  pyca/          ← moved from detection/ root
    aead/  asymmetric/  fernet/  hash/  kdf/
    keyagreement/  mac/  padding/  symmetric/  wrapping/
  pycrypto/      ← new
    cipher/  hash/  kdf/  keyagreement/
    mac/  publickey/  random/  signature/

Test classes .java and test input .py files follow the same structure under src/test/files/rules/detection/pyca/ and pycrypto/.

PythonDetectionRules.java and PythonReorganizerRules.java updated to include all new rule registrations.

4 · Mapper additions

File Change
PycaCurveMapper New — maps PyCrypto/pyca curve names to canonical EllipticCurve models
PycaDigestMapper Extended with additional hash algorithms
PycaMacMapper Extended with CMAC
PycaCipherMapper Extended with PyCrypto cipher modes
SignatureReorganizer Extended to handle DSS/EdDSA/PSS/ECDSA reorganization

5· Performance

Added memoization to hot translation paths to avoid redundant re-computation during multi-rule traversals.


Tests

55 new JUnit tests under python/src/test/java/…/pycrypto/ — one per detection scenario — covering approximately 80 % of the new detection rules:

  • Cipher (13): AES, 3DES, DES, Blowfish, CAST5, RC2, RC4, Salsa20, ChaCha20, ChaCha20-Poly1305, PKCS1-OAEP, PKCS1v15, HPKE
  • Hash (16): MD2, MD5, SHA-1/224/256/384/512, SHA3-224/256/384/512, BLAKE2b, BLAKE2s, RIPEMD-160, TupleHash128, cSHAKE256
  • KDF (5): PBKDF1, PBKDF2, scrypt, HKDF, bcrypt
  • Key agreement (5): DH, X25519 private/public, X448 private/public
  • MAC (2): HMAC, CMAC
  • Public key (4): RSA, DSA, ECC, ElGamal
  • Signature (10): DSS sign/verify, EdDSA sign/verify, PKCS1v15 sign/verify, PSS sign/verify, ECDSA sign/verify

41 existing pyca tests updated to reflect the restructured package paths.


Directory structure change

Before After
detection/aead/Pyca*.java detection/pyca/aead/Pyca*.java moved
detection/asymmetric/Pyca*.java detection/pyca/asymmetric/Pyca*.java moved
detection/hash|kdf|mac|padding… detection/pyca/hash|kdf|mac|padding… moved
test/files/rules/detection/*.py test/files/rules/detection/pyca/*.py moved
detection/pycrypto/**/*.java new
test/files/rules/detection/pycrypto/**/*.py new

Translator additions & changes

Translator Status Note
PycaKeyContextTranslator new Unified key handling for pyca + pycrypto
PycaPublicKeyContextTranslator new Shared RSA/DSA/ECC public-key extraction
PycaRandomContextTranslator new Random context, used by PythonCryptoRandom
PycaCipherContextTranslator updated Extended for PyCrypto cipher modes
PycaKeyDerivationContextTranslator updated Added PBKDF1, bcrypt support
PycaSignatureContextTranslator updated Added DSS, EdDSA, PSS, ECDSA contexts
PycaPrivateKeyContextTranslator updated Simplified; shared logic moved to PycaKeyContextTranslator
PycaDigestContextTranslator updated Minor extension
PycaKeyAgreementContextTranslator updated DH / X25519 / X448 support
PycaMacContextTranslator updated CMAC added

@san-zrl
san-zrl requested a review from a team as a code owner July 22, 2026 11:25
@san-zrl
san-zrl requested a review from n1ckl0sk0rtge July 22, 2026 11:26

@n1ckl0sk0rtge n1ckl0sk0rtge left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Review: Feat/python crypto detection (#495)

Linked Issues: none — PR has no Closes #N reference
Requirements Coverage: feature largely delivered, but 3 of the 8 advertised rule groups don't reach production (see MUST FIX 1, 4, 5, 7)
Architecture: CLEAN — no layer violations; rules import no com.ibm.mapper.*, all 8 holders follow the memoization contract
Domain Patterns: 8 blocking findings (crypto mis-mapping + dead detections)
Blast Radius: MEDIUM — the engine prefix-matching change is provably inert for Java/Go, but SignatureReorganizer sits on live Java and Go paths with a semantic change and zero Java/Go tests
Security: no secrets in fixtures, no new dependencies, no analysis-time IO — clean
Tests: 55 new tests, all green (mvn test -pl python -am → BUILD SUCCESS, 106 python tests, 0 failures); assertions genuinely exhaustive
Breaking Changes: EXTERNAL — MethodMatcher public constructor replaced; existing pyca curve detection regresses

MUST FIX (Blocking)

1. [Architecture] PythonCryptoPublicKey.rules() is never registered — all public-key detection is dead in production
python/src/main/java/com/ibm/plugin/rules/detection/PythonDetectionRules.java:81-87
Only 7 of the 8 pycrypto holders are in buildRules(). RSA.generate(), DSA.generate(), ECC.generate(), ECC.import_key(), ElGamal.generate() produce zero findings in a real scan. The 4 publickey tests pass only because they instantiate PythonCryptoPublicKey.rules() directly — they give false confidence.
Fix: add PythonCryptoPublicKey.rules().stream(), to the Stream.of(...).

2. [Domain] PycaCurveMapper drops SECP521R1 and SECP256K1 — regression for existing pyca users
mapper/src/main/java/com/ibm/mapper/mapper/pyca/PycaCurveMapper.java:70
PycaPrivateKeyContextTranslator previously handled both inline; it now delegates to this mapper, which has neither. Instead it has case "SECP512R1", "PRIME512V1", "P-512", "P512", "NIST P-512" -> new Secp521r1(...)secp512r1 does not exist. So ec.SECP521R1() and ec.SECP256K1() in already-scanned Python code silently stop emitting their curve component.
Fix: rename the case set to "SECP521R1", "PRIME521V1", "P-521", "P521", "NIST P-521" and re-add case "SECP256K1" -> new Secp256k1(...).

3. [Domain] Ed25519/Ed448 conflated with X25519/X448 curves
mapper/src/main/java/com/ibm/mapper/mapper/pyca/PycaCurveMapper.java:72-75case "ED25519", "CURVE25519" -> Curve25519. EdDSA signature curves and Montgomery ECDH curves are distinct, and the repo already models them separately (Ed25519/Ed448 in PycaPrivateKeyContextTranslator:97-98). ECC.generate(curve='Ed25519') is now reported as an ECDH curve.
Fix: split the aliases. Note ECCTestFile.py:4 bakes the conflation into an assertion, so that marker needs updating too.

4. [Domain] bcrypt is detected but never translated — dead detection
python/src/main/java/com/ibm/plugin/rules/detection/pycrypto/kdf/PythonCryptoKDF.java:280,293 emit kind = "bcrypt", but PycaKeyDerivationContextTranslator has zero bcrypt cases and there is no Bcrypt model class. BcryptTestFile.py:6 admits it — # detected but not mapped — and BcryptTest.java:73 asserts nodes is empty. The test locks in the gap rather than covering the feature.
Fix: add a Bcrypt algorithm model + translator case, or drop BCRYPT/BCRYPT_CHECK until it exists.

5. [Domain] EdDSA import rules emit "ECC", which no translator handles — dead detection
python/src/main/java/com/ibm/plugin/rules/detection/pycrypto/publickey/PythonCryptoPublicKey.java:217,230 build contexts with algorithm = "ECC", but PycaPublicKeyContextTranslator handles only DH|RSA|DSA|ELGAMAL and PycaPrivateKeyContextTranslator handles EC, not ECC. Both eddsa.import_private_key / import_public_key rules never translate.
Fix: use "ED25519"/"ED448", or add an ECC case.

6. [Domain] "shuflle" typo
python/src/main/java/com/ibm/plugin/rules/detection/pycrypto/random/PythonCryptoRandom.java:58Crypto.Random.random.shuffle() is never detected. No test exists for PythonCryptoRandom at all, which is why it wasn't caught.

7. [Domain] The four common PBKDF2 forms are commented out of buildRules()
python/src/main/java/com/ibm/plugin/rules/detection/pycrypto/kdf/PythonCryptoKDF.java:322-325 — only the 5-arg PBKDF2_WITH_COUNT_AND_HASH is active, so PBKDF2(password, salt, 32) and PBKDF2(password, salt, 32, count=100000) are undetected. The private constants remain in the file (lines 149-222).
Fix: enable or delete. Same pattern at PythonCryptoCipher.java:311 (references PKCS1_OAEP_NEW_WITH_HASH, a constant that does not exist) and PythonCryptoPublicKey.java:76-88,118-130,194-206.

8. [Breaking Change] MethodMatcher public constructor replaced, not overloaded
engine/src/main/java/com/ibm/engine/detection/MethodMatcher.java:72
MethodMatcher(String[], String[], List<String>) was mutated in place into a 4-arg prefixMatch form. docs/LANGUAGE_SUPPORT.md:459 points extension authors at this class — this is a source- and binary-incompatible removal for out-of-tree consumers. Nothing in-repo breaks.
Fix: keep the 3-arg array constructor delegating this(..., false).

SHOULD FIX (Non-blocking)

9. [Domain] All 14 KDF rules use .inBundle(() -> "PyCa") instead of "PyCrypto"PythonCryptoKDF.java (14/14; every other pycrypto class correctly uses "PyCrypto"). PyCryptodome KDF findings will be attributed to the pyca library in the CBOM.

10. [Blast Radius] SignatureReorganizer mutates a live, aliasable children mapmapper/src/main/java/com/ibm/mapper/reorganizer/rules/SignatureReorganizer.java:200-201,263-264. The ConcurrentModificationException fix (values().forEach(n::put); getChildren().clear();) iterates and clears the live internal map — and Algorithm's copy constructor aliases it (Algorithm.java:39: this.children = algorithm.getChildren()), so clear() can wipe a different node's children. Both methods are on live Java (JavaReorganizerRules:58) and Go (GoReorganizerRules:64,66,68) paths, and this PR adds zero Java/Go tests.
Fix: iterate a defensive copy; add a multi-child regression test for Java and Go.

11. [Scope] mapper/ciphersuites.json + JsonCipherSuites.java are unrelated build artifactsmapper/download-cipher-suites.sh curls the file from ciphersuite.info and creatCipherSuiteClass.py regenerates the Java. The semantic diff is clean (348 → 351 entries: 3 ASCON suites added, 0 removed, 0 modified), but this has nothing to do with Python crypto detection.
Fix: git checkout main -- mapper/ciphersuites.json mapper/src/main/java/com/ibm/mapper/mapper/ssl/json/JsonCipherSuites.java; land the ASCON suites separately.

12. [Domain] DES_3 is fully subsumed by DES_2 → duplicate findingsPythonCryptoCipher.java:99-112 vs :84-97. Since prefixMatchesParameters accepts actual >= expected, DES.new(key, DES.MODE_CBC, iv) matches both rules. Fix: delete DES_3.

13. [Domain] ChaCha20 / ChaCha20-Poly1305 get no Encrypt/Decrypt nodesPythonCryptoCipher.java:186-208 end with .withoutDependingDetectionRules() while every sibling attaches List.of(ENCRYPT, DECRYPT).

14. [Domain] pkcs1_15 signatures lose their PKCS#1 v1.5 paddingPythonCryptoSignature.java:81ValueActionFactory("RSA"), indistinguishable from a raw RSA signature. Compare pss"RSA-PSS"RSAssaPSS. The padding branch at PycaSignatureContextTranslator.java:82 is still case "PKCS1V15" -> Optional.empty(); // TODO.

15. [Domain] DSS rules lack withOtherParameters()PythonCryptoSignature.java:123-152 require exactly 2 params, so the documented DSS.new(key, 'fips-186-3', encoding='der') is missed. The ECDSA twins do use it. (DSS-vs-ECDSA discrimination itself is correct — it keys off the first parameter's type, so there is no double counting.)

16. [Domain] HMAC/CMAC bind the algorithm to the 2nd positional parameter, which is msgPythonCryptoMac.java:42-64. The real signatures are HMAC.new(key, msg=b'', digestmod=None) and CMAC.new(key, msg=b'', ciphermod=None). HMAC.new(key, data, SHA256) is missed.

17. [Domain] Poly1305 / KMAC128 / KMAC256 registered as digests, not MACsPythonCryptoHash.java:41-68. This contradicts PythonCryptoMac.java:66-78, where the Poly1305 MAC rule was deliberately disabled.

18. [Domain] All hash rules use .withAnyParameters(), dropping size/variantPythonCryptoHash.java:83-92. SHA512.new(truncate='256') reports plain SHA-512 even though PycaDigestMapper already has SHA512_224/SHA512_256 ready; BLAKE2b/s lose their digest size.

19. [Domain] The HPKE rule likely cannot firePythonCryptoCipher.java:269-285 uses two positional parameters, but PyCryptodome's HPKE.new() is keyword-only. HPKETestFile.py uses invented syntax (from Crypto.Protocol.HPKE.AEAD import AES128_GCMAEAD is a class, not a submodule), so the green test does not prove real-world matching. Worth confirming against a real install.

20. [Tests] Three new components have no test at allPythonCryptoRandom (both rules), PycaRandomContextTranslator (0% coverage; only reachable from it), and PycaCurveMapper (20 branches, no test anywhere in mapper/src/test/). The missing curve test is exactly what would have caught MUST FIX 2.

21. [Tests] The Cryptodome.* branch is structurally untested — zero of the 55 new .py files use the Cryptodome prefix. DSS_CRYPTODOME and ECDSA_CRYPTODOME exist solely for it and are never exercised. Given the PR's headline claim of pycryptodomex support, this is the largest coverage gap.

22. [Tests] The new core-engine matching mode has no engine testwithOtherParameters() / SOME_WITH_REMAINDER / prefixMatchesParameters appear nowhere under any src/test/. Used 15 times, verified only indirectly through Python integration tests.

23. [Architecture] Unchecked (PublicKeyEncryption) casts on Optional<? extends Algorithm>PycaKeyContextTranslator.java:79, PycaPublicKeyContextTranslator.java:79, PycaPrivateKeyContextTranslator.java:74. Safe only by accident today; a ClassCastException here aborts the Sonar analysis of the file. Fix: narrow PycaCurveMapper.parse to Optional<EllipticCurveAlgorithm>.

24. [Architecture] The key-algorithm switch is duplicated across three translators and is already driftingELGAMAL was added to two of them, FERNET/EC_IMPORT to only one. Fix: extract an IMapper into mapper/.../mapper/pyca/.

25. [Docs] withOtherParameters() is absent from the documented builder grammardocs/DETECTION_RULE_STRUCTURE.md:172,178. Third parties following this doc will not discover the new DSL step.

26. [Breaking Change] MethodMatcherSerializer does not emit prefixMatch — exported rule JSON cannot distinguish prefix from exact arity, misrepresenting all 15 new rules.

27. [Domain] PycaSecretContextTranslator is now dead code — unwired from PythonTranslator (replaced by PycaKeyContextTranslator for KeyContext) but still present. Confirm no pyca regression, then delete.

28. [Domain] bcrypt cost is log2 of rounds, mapped straight to ITERATIONSPythonCryptoKDF.java:269-282 reports cost=10 as 10 iterations instead of 1024.

29. [Docs] Broken README linkREADME.md:39: [PyCryptodome(x)][https://www.pycryptodome.org/] is reference-link syntax with no defined reference, so GitHub renders the literal text. Use (...) instead. Also revert the trailing-whitespace-only change on line 38.

SUGGESTIONS

  1. scrypt's r is a block-size factor (block = 128·r bytes), reported as raw bytes — PythonCryptoKDF.java:258-261
  2. Append SOME_WITH_REMAINDER at the end of the enum rather than inserting it — safe today (no ordinal()/serialization use anywhere), defensive convention only
  3. Guard prefixMatchesParameters against an empty expectedTypes list (it would match every invocation) — currently unreachable, but one refactor away
  4. PycaCipherContextTranslator.java:109-122 compares raw asString() while neighbouring switches use .toUpperCase().trim()
  5. Crypto.Protocol.DH.key_agreement is hard-coded to "ECDH" even though the X25519/X448 import rules feed it
  6. PycaCurveMapper has phantom aliases (PRIME192V1/PRIME224V1/PRIME384V1 are not real X9.62 names) and partial brainpool coverage — suggests the list was extrapolated rather than taken from PyCryptodome's curve table
  7. Delete the ~20 lines of commented-out original code in SignatureReorganizer.java:194-205,257-268

What's good here

Worth calling out explicitly — several things were done carefully:

  • The 95-file rename is flawless. Zero stale FQN references repo-wide, including rules/, plugin resources, and rule metadata; all 96 PythonCheckVerifier.verify(...) path strings resolve. The pyca move is a pure rename with 0 changed lines in the rule bodies.
  • The memoization contract is followed perfectly across all 8 new classes, including all four suppliers in PythonCryptoPublicKey.
  • Layering is respected — no rule class imports com.ibm.mapper.*; PycaCurveMapper is correctly language-agnostic.
  • Translator dispatch ordering is correct and safeIDetectionContext.is() uses exact class equality rather than isAssignableFrom, and the new KeyContext branch was placed after the subtype branches anyway.
  • Test assertions are genuinely exhaustive (12–46 assertions per class, covering both the detection store and the translated nodes) — no stubs. BcryptTest's empty-node assertion is honest documentation of a gap, not a cheat.
  • The broad ENCRYPT/DECRYPT/SIGN/VERIFY rules are correctly excluded from top-level registration, so there is no repo-wide match on every .encrypt() call.
  • prefixMatch is provably inert for Java/Go — gated behind the new opt-in builder step and used only in pycrypto/.

Bottom line: the reorganization and test discipline are solid, but the 8 blocking issues mean the feature does not fully work as advertised — one entire rule class is unregistered, three rule groups are dead detections, and the shared curve mapper regresses existing pyca output. The engine and mapper/ciphersuites.json changes would be better split into separate PRs.


This review was generated by AI (Claude). Findings may contain errors — please verify before acting on them.

san-zrl added 13 commits August 14, 2026 13:07
… DSL

- Add SOME_WITH_REMAINDER to CapturedParameterScope so a rule matches
  calls whose actual arity is >= the declared arity
- Expose withOtherParameters() on every terminal builder stage in
  IDetectionRule; implement in DetectionRuleBuilderImpl
- MethodMatcher: add boolean prefixMatch field and prefixMatchesParameters();
  keep 3-arg array constructor as a backward-compatible delegate (prefixMatch=false)
- MethodMatcherSerializer: emit prefixMatch boolean field in exported JSON
- DetectionRuleStore: append prefixMatch flag to matcher ID to prevent
  silent deduplication of rules that differ only in arity mode

Signed-off-by: san-zrl <san@zurich.ibm.com>
Signed-off-by: san-zrl <san@zurich.ibm.com>
…natureReorganizer

Algorithm(Algorithm,Class) and Key(Key,Class) assigned the source node's
live children map by reference; mutating the re-kinded copy corrupted the
original node's children. Use new HashMap<>(src.getChildren()) instead.

SignatureReorganizer.moveNodesFromUnder*: snapshot node.getChildren() into
an ArrayList before iterating and remove each child individually via
removeChildOfType() to avoid ConcurrentModificationException on the live map.

Signed-off-by: san-zrl <san@zurich.ibm.com>
…yca mappers

PycaCurveMapper: centralises curve-string to model mapping. Correctly splits
Edwards curves (ED25519->Edwards25519, ED448->Edwards448) from Montgomery
curves (CURVE25519->Curve25519, CURVE448->Curve448). Fixes SECP521R1 aliases
(was SECP512R1) and re-adds the missing SECP256K1 case.

PycaKeyBasedAlgorithmMapper: extracts the RSA/DSA/DH/ElGamal/Fernet algorithm
switch that was duplicated across the three key-context translators.

PycaCipherMapper: adds DES, RC2, Salsa20, AES128_GCM/AES256_GCM, RC4 alias,
CHACHA20_POLY1305 alias.
PycaDigestMapper/PycaMacMapper: adds MD2, MD4, RIPEMD-160, KMAC, TupleHash,
cSHAKE, Keccak, KangarooTwelve for PyCryptodome rules.
RIPEMD: adds (asKind, RIPEMD) copy constructor for MAC usage.

Signed-off-by: san-zrl <san@zurich.ibm.com>
Relocate all existing pyca rule classes (aead, asymmetric, fernet, hash,
kdf, keyagreement, mac, padding, symmetric, wrapping) into the new
detection/pyca/ sub-package to match the pycrypto/ layout and avoid future
naming collisions. No logic changes.

Signed-off-by: san-zrl <san@zurich.ibm.com>
New detection-rule classes under detection/pycrypto/:
  cipher    - AES, DES, 3DES, Blowfish, CAST5, RC2, RC4, ChaCha20,
              ChaCha20-Poly1305, Salsa20, PKCS1-OAEP, PKCS1-v1.5, HPKE
  hash      - MD2/4/5, SHA-1/2/3, RIPEMD-160, SHAKE, BLAKE2b/s, KMAC,
              TupleHash, cSHAKE, Keccak, KangarooTwelve, Poly1305
  mac       - HMAC (2-arg and 3-arg), CMAC (2-arg and 3-arg)
  kdf       - PBKDF1 (5 arity variants), PBKDF2 (5 arity variants),
              scrypt, HKDF, SP800-108
  keyagreement - DH.key_agreement, X25519/X448 import helpers
  publickey - RSA, DSA, ECC, ElGamal generate/import (RSA/DSA/ECC reached
              as dependent rules from Signature; only ElGamal is top-level)
  random    - Crypto.Random.get_random_bytes, Crypto.Random.random methods
  signature - PKCS1v15 (emits RSA-PKCS1V15), PSS, DSS, ECDSA, EdDSA

All rules use .inBundle(() -> "PyCrypto") for correct CBOM attribution.

Signed-off-by: san-zrl <san@zurich.ibm.com>
…lugin

PythonDetectionRules: register all 8 new PyCryptodome rule bundles (Hash,
  Mac, Random, Cipher, PublicKey, Signature, KDF, KeyAgreement).

PythonTranslator: replace dead PycaSecretContextTranslator with new
  PycaKeyContextTranslator; wire PycaRandomContextTranslator for PRNGContext;
  move KeyContext handling after Private/PublicKey so more-specific contexts
  are checked first.

PythonReorganizerRules: add four moveNodesFromUnderFunctionalityUnderParent
  rules for Sign/Verify under Signature and PSS.

PycaKeyContextTranslator (new): handles KeyContext for RSA/DSA/EC/ElGamal/
  Fernet keys; delegates to PycaKeyBasedAlgorithmMapper and PycaCurveMapper.
PycaRandomContextTranslator (new): maps PRNG to PseudorandomNumberGenerator.
PycaPrivateKeyContextTranslator: delegate curve mapping to PycaCurveMapper;
  add ElGamal and EC_IMPORT cases; use PycaKeyBasedAlgorithmMapper.
PycaPublicKeyContextTranslator: add ElGamal; add Curve branch via
  PycaCurveMapper; use PycaKeyBasedAlgorithmMapper.
PycaCipherContextTranslator: add MODE_* aliases; PKCS1_OAEP/PKCS1_v1_5/
  HPKE ValueAction cases.
PycaSignatureContextTranslator: add RSA-PKCS1V15, DSS, ECDSA, EDDSA cases.
PycaKeyDerivationContextTranslator: add PBKDF1/PBKDF2/HKDF root nodes;
  IterationCount/SaltSize translation; pycrypto-pbkdf* digest pa  IterationCount/SaltSize translation; pycrypto-pbkdf* digest pa  Iterationn to Algorithm.

Signed-off-by: san-zrl <san@zurich.ibm.com>
Mirror the production-code reorganisation. No assertion changes; only
package declarations and fixture paths updated.

Signed-off-by: san-zrl <san@zurich.ibm.com>
Covers: cipher (AES, DES, 3DES, Blowfish, CAST5, RC2, RC4, ChaCha20,
ChaCha20-Poly1305, Salsa20, PKCS1-OAEP, PKCS1-v1.5), hash (MD2/4/5,
SHA-1/2/3, RIPEMD-160, BLAKE2b/s, TupleHash128, cSHAKE256), KDF (PBKDF1,
PBKDF2, scrypt, HKDF), MAC (HMAC, CMAC), key agreement (DH, X25519/X448),
public key (RSA, DSA, ECC, ElGamal), signature (PKCS1v15, PSS, DSS,
ECDSA, EdDSA).

Signed-off-by: san-zrl <san@zurich.ibm.com>
…yca mappers

PycaCurveMapper: centralises curve-string to model mapping. Correctly splits
Edwards curves (ED25519->Edwards25519, ED448->Edwards448) from Montgomery
curves (CURVE25519->Curve25519, CURVE448->Curve448). Fixes SECP521R1 aliases
(was SECP512R1) and re-adds the missing SECP256K1 case.

PycaKeyBasedAlgorithmMapper: extracts the RSA/DSA/DH/ElGamal/Fernet algorithm
switch that was duplicated across the three key-context translators.

PycaCipherMapper: adds DES, RC2, Salsa20, AES128_GCM/AES256_GCM, RC4 alias,
CHACHA20_POLY1305 alias.
PycaDigestMapper/PycaMacMapper: adds MD2, MD4, RIPEMD-160, KMAC, TupleHash,
cSHAKE, Keccak, KangarooTwelve for PyCryptodome rules.
RIPEMD: adds (asKind, RIPEMD) copy constructor for MAC usage.

KeyAgreementReorganizer: add REPLACE_ECDH_WITH_X25519_WHEN_CURVE25519 and
REPLACE_ECDH_WITH_X448_WHEN_CURVE448 rules to convert a generic ECDH node
into the correct XDH algorithm when both key children carry the matching curve.

Signed-off-by: san-zrl <san@zurich.ibm.com>
Relocate all existing pyca rule classes (aead, asymmetric, fernet, hash,
kdf, keyagreement, mac, padding, symmetric, wrapping) into the new
detection/pyca/ sub-package to match the pycrypto/ layout and avoid future
naming collisions. No logic changes.

Signed-off-by: san-zrl <san@zurich.ibm.com>
Delete the original flat detection/{aead,asymmetric,fernet,hash,kdf,
keyagreement,mac,padding,symmetric,wrapping} trees and the superseded
PycaSecretContextTranslator now that everything has been moved to
detection/pyca/ and replaced by the new translators.

Signed-off-by: san-zrl <san@zurich.ibm.com>
@san-zrl
san-zrl force-pushed the feat/python-crypto-detection branch from 253117b to 87c5284 Compare August 14, 2026 11:08
@san-zrl

san-zrl commented Aug 14, 2026

Copy link
Copy Markdown
Contributor Author

Hi @n1ckl0sk0rtge,

Thanks for the review. I modified the code in response and pushed my changed .
Note that I rebased the branch and organized the commit history into logical, grouped steps.

Here's what I did:

Must Fix

M1: PythonCryptoPublicKey.rules() was redesigned — it now only contains ElGamal. The are marked as obsolete keys and should be detected stand-alone. A class-level comment documents this.

RSA_GENERATE, DSA_GENERATE, ECC_GENERATE, etc. are intentionally not top-level rules. They are reached as dependent rules from PythonCryptoSignature — for example, PKCS1V15 and PSS call PythonCryptoPublicKey.RSARules(), DSS call PythonCryptoPublicKey.DSARules(), and ECDSA, EDDSA and DH_KEY_AGREEMENT call PythonCryptoPublicKey.ECCRules().

M2: The typo — SECP512R1 → SECP521R1 (and all its aliases) — fixed

  1. The original broken case replaced with "SECP521R1", "PRIME521V1", "P-521", "P521", "NIST P-521"
    All five label strings now correctly identify the 521-bit NIST prime curve.
  2. The missing SECP256K1 case — re-added

M3: Fixed.

Concern Status
ED25519 mapped to Curve25519 (wrong) Fixed — now maps to Edwards25519
CURVE25519 still maps to Curve25519 Correct — preserved
Same split applied to 448-bit family Done — ED448 → Edwards448, CURVE448 → Curve448
ECCTestFile.py marker updated Done — EC-Edwards25519
ECCTest.java assertion updated Done — asserts Edwards25519 throughout. If the mapper still conflated ED25519 → Curve25519, these assertions would fail.

M4: Fixed. Removed bcrypt rule and test for the moment.
Bcrypt support could be an issue on its own. Support exists in BouncyCastle (not detected) and pycryptodome.
For python (complementing python cryptography) and for golang there are separate bcrypt libs. Our mapper does
not provide a BCrypt algorithm, so there's currently no translation target.

M5: Fixed. Both detection rules now emit "EC" instead of "ECC" so that detections produce translated output. Tests
could remain unchanged. The suggested modification to emit Ed25519/Ed448 keys does not work since the EDDSA_IMPORT_PUBLIC/PRIVATE_KEY rules cannot infer the precise key type from the call site alone. Therefore the rules produce Elliptic Curve keys which is the correct generalization.

M6: Fixed the typo.

M7: Fxed. All five PBKDF2 variants are now active in buildRules(): PBKDF2, PBKDF2_WITH_HASH,
PBKDF2_WITH_COUNT, PBKDF2_WITH_HASH_AND_COUNT, PBKDF2_WITH_COUNT_AND_HASH
Non-existent PKCS1_OAEP_NEW_WITH_HASH removed.

M8: Fixed as prescribed.

SHOULD FIX

S9: All KDF rules use now .inBundle(() -> "PyCrypto") as suggested.

S10: Fixed, but no multi-child regression tests for Java and Go.

  1. Both moveNodesFromUnderFunctionalityUnderNode and moveNodesFromUnderFunctionalityUnderParent now snapshot the children into an ArrayList before iterating, and remove each entry individually with removeChildOfType rather than bulk clear(). This eliminates both the ConcurrentModificationException and the aliased-map corruption in a single change.
  2. Root-cause fix: The copy constructors at Algorithm.java:39 and Key.java:48 now both do new HashMap<>(...) instead of assigning the live map reference directly. Any future code that mutates a re-kind copy no longer corrupts the source node's children.

S11: No change

S12: DES_3 rule removed. Update of Cipher rules AES, DES, DES3, BLOWFISH, CAST, ARC2.

S13: Fixed. Both rules now end with .withDependingDetectionRules(List.of(ENCRYPT, DECRYPT)), consistent with all other cipher rules.

S14: Fixed.

  1. Detection rule — "RSA" → "RSA-PKCS1V15".
  2. PycaSignatureContextTranslator now handles "RSA-PKCS1V15" in the main ValueAction switch.

S15: Fixed. Both DSS (line 140) and DSS_CRYPTODOME (line 156) now have .withOtherParameters() after the typed key parameter, with a comment explicitly listing what the extra args cover. This means DSS.new(key, 'fips-186-3'), DSS.new(key, 'fips-186-3', encoding='der'), and any other arity variant all match. The fix is symmetric with ECDSA (line 172) and ECDSA_CRYPTODOME (line 188), which already had it.

S16: Fixed. The original single-rule approach — which bound the algorithm to the 2nd positional parameter and thus confused msg with the hash/cipher module — was replaced with a two-rule pattern for both HMAC and CMAC.

S17: "KMAC128", "KMAC256", and "Poly1305" were wrongly detected as hashes and emitted as if they were digests.
"KMAC128", "KMAC256", and "Poly1305" were moved to PythonCryptoMac.java which correctly detects them under MacContext. PycaMacContextTranslator.java routes them to PycaMacMapper.java which correctly maps them as MACs. Tests for "KMAC128", "KMAC256", and "Poly1305" were added to python/src/test/java/com/ibm/plugin/rules/detection/pycrypto/mac.

S18: Not addressed, accepted as a limitation.

  1. SHA512 allows two named parameters "data" and "truncate", both of type str. When using detection rules with positional heuristics there's no way to distinguish the parameters.
  2. BLAKE2s/BLAKE2b allow digest size to be passed as bit or byte values via different named parameters. Number of
    detection rules would explode.

S19: Removed attempted detection of HPKE and corresponding test. The case is similar to M4 (Bcrypt).
In our mapper model there is no HPKE algorithm and thus no translation target for a HPKE detection rule.
I guess this is also the reason as to why our pyca detection rules don't detect HPKE either even though
the pyca library implements it.

S20, S21, S22: Missing tests not addressed.

S23: Narrowed PycaCurveMapper.parse to Optional, cast exception now impossible. Removed cast in PycaKeyContextTranslator. The other two casts are necessary.

S24: Fixed. Created PycaKeyBasedAlgorithmMapper and wired it into PycaKeyContextTranslator, PycaPrivateKeyContextTranslator and PycaPublicKeyContextTranslator replacing the drifting switch statements.

S25: Fixed. Added withOtherParameters() to the grammar and explained it in the text below.

S26: Fixed.

  1. MethodMatcher — stored the prefixMatch flag as a proper field across all constructors and exposed it via isPrefixMatch().
  2. MethodMatcherSerializer — emits "prefixMatch": true/false into the exported rule JSON so readers can distinguish prefix-arity from exact-arity matchers.
  3. DetectionRuleStore — appends the prefixMatch flag to the matcher ID string, preventing two rules that differ only in arity mode from being silently deduplicated during graph export.

S27: Fixed. Removed dead PycaSecretContextTranslator..java

S28: Void. Bcrypt removed in response to M4.

S29. Fixed broken link and trailing whitespace in README.md.

Suggestions

S30: Left as is
S31: Left as is
S32: Added !expectedTypes.isEmpty() as guard to prefixMatchesParameters()
S33: Fixed. Use toUpperCase().trim()
S34: Update detection rules in PyhonCryptoKeyAgreement.java that generate ECDH detections with ECC keys.
Added two reorganiser rules that convert the ECDH to X25519 and X448 if the curves in the ECC keys are 25519 or 448.
S35: The PRIME curve names were there before. Left as is
S36: Deleted commented out code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants