Finish ZeroKMS vs HSM comparison + add comparison diagrams

[coderdan]

Follow-up to #31, which migrated the ZeroKMS comparison content from cipherstash-js-suite PR #364. Some of it landed as work-in-progress and needs finishing.

1. Fill in the WIP TODOs in zerokms-vs-hsm.mdx

content/stack/reference/comparisons/zerokms-vs-hsm.mdx still carries the original draft's TODO markers:

• Benefits of HSM section — currently just TODO
• Operational complexity — TODO: key ceremonies, replication, uptime (eg. power), reliability etc
• Compliance section — TODO: Talk about PCI/DSS etc
• Physical access "ZeroKMS Advantage" callout — TODO: this could be worded better
• Cryptographic custody / distributed trust link — TODO (may warrant its own "Trust Model" article)

2. Replace the diagram placeholder

The "Hybrid solutions" section of zerokms-vs-hsm.mdx originally used a D2 diagram, but this repo has no D2 renderer. It was converted to a {/* TODO */} comment with the original D2 source preserved inline:

• zerokms-vs-hsm.mdx — "Hybrid solutions" client/SDK/HSM diagram

Needs a decision on how diagrams are rendered in this repo (image export, mermaid support, or a diagram component).

Note: the original ZeroKMS vs Cloud KMS page also had a placeholder diagram, but that page was merged into aws-kms.mdx and the placeholder diagram (which was sample content) was dropped.

---

Once both are addressed, cipherstash-js-suite PR #364 can be closed.

[coderdan]

Two performance-claim follow-ups surfaced while rewriting the comparison pages for an agent-research audience (#31):

• Source the "14× faster than AWS KMS" throughput claim. It's currently unsourced (flagged with an inline {/* TODO */} in aws-kms.mdx). The FHE page cites an open-source benchmark harness; this figure should too, with conditions. The sub-5ms latency claim is now linked to the Artillery benchmark.
• Reconcile 14× vs 100× across the docs. Standardized on 14× in the comparison pages, but cipherstash/kms/index.mdx and reference/what-is-cipherstash.mdx still say 100×. A reader cross-referencing pages will hit the contradiction — pick one repo-wide once the benchmark is confirmed.

[coderdan]

Benchmark results — conditioning the "14×" claim

The ZeroKMS-vs-AWS-KMS benchmark now exists: cipherstash/benches#18 (report with charts). It gives us real data to source/reframe the unsourced, internally-contradictory "14× / 100× faster than AWS KMS" claims.

What the data says

The advantage is not a blanket multiple — it's a function of how many values you encrypt/decrypt per request, because the architectures differ at the root: ZeroKMS does a whole batch in one round-trip (bulk API, up to 10k keys); AWS KMS has no bulk API, so it's one call per value.

• Latency (median p95): ~8× at 100 records, ~15–17× at 500 records (3 encrypted fields each). AWS KMS throttle-fails the majority of requests past a few hundred values; ZeroKMS: 0 failures.
• Throughput: ZeroKMS scaled linearly to ≥30,000 values/s with 0 failures (and didn't saturate); AWS KMS peaked ~2,500 values/s then collapsed under load (more load → lower achieved throughput). ≥12× and widening.

⚠️ Don't cite these exact numbers yet

They're from a developer laptop on home Wi-Fi, which overstates the gap (network penalty hits AWS's per-value calls harder). The conservative, citable numbers come from the pending in-region EC2 run (runbook: kms-app/EC2.md).

Recommendation for the docs

1. Drop the blanket "14×" / "100×" in favour of a conditioned statement, e.g. "For bulk operations, ZeroKMS is roughly an order of magnitude faster than AWS KMS — and the gap widens with batch size, because AWS KMS has no bulk API and throttles past a few hundred values per request."
2. Reconcile the contradiction: kms/index.mdx and what-is-cipherstash.mdx say 100×; the comparison pages say 14×. Neither is supported as a blanket figure. Pick one conditioned framing repo-wide.
3. Hold the hard multiplier + citation until the EC2 numbers land, then cite cipherstash/benches.

[coderdan]

In-region benchmark is done — citable numbers

The conservative, in-region run (cipherstash/benches#18, report) is complete: two c7i.2xlarge in ap-southeast-2, with the load generator on a separate instance from the app (the earlier laptop numbers overstated the gap; a single-box attempt was CPU-bound and discarded).

Citable figures:

• Latency: ZeroKMS ~16× faster at a 100-record batch (52 ms vs 854 ms p95). AWS KMS throttle-fails at 500+ records.
• Throughput: ZeroKMS sustains ~21,000 values/s vs AWS KMS ~250 — a ~85× gap.

These supersede the laptop directional numbers in my earlier comment. The "≈14×" in the docs is in the right ballpark for mid-batch latency but undersells the throughput story — for bulk operations the gap is closer to two orders of magnitude. Recommendation stands: replace the blanket "14×/100×" with a conditioned statement (e.g. "for bulk operations ZeroKMS is ~16× lower latency and ~85× higher throughput than AWS KMS, which has no bulk API and throttles at scale") and cite cipherstash/benches.

Caveat for whoever writes the copy: ZeroKMS's ~21k throughput ceiling here was bounded by the app instance's CPU, not ZeroKMS — so it's a floor.