Skip to content

WIP: Ledger security release docs (2026.1)#329

Draft
evanorti wants to merge 24 commits into
mainfrom
security-release
Draft

WIP: Ledger security release docs (2026.1)#329
evanorti wants to merge 24 commits into
mainfrom
security-release

Conversation

@evanorti

@evanorti evanorti commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

WIP: Ledger security release docs (2026.1)

Documentation for the 2026.1 security release: post-quantum keys (ML-DSA),
zero-downtime consensus key rotation, and remote signing via Cosmos-KMS.
Every technical claim is code- or test-verified before its page merges.

Nav: three groups under Run a Node (SDK next): Key Rotation, Post-quantum Keys,
Remote Signing: Cosmos-KMS.

Key Rotation

Post-quantum Keys

  • Post-quantum keys — keys and algorithms survey, ML-DSA, costs, EVM note
  • Enable ML-DSA keys — genesis and governance paths; governance flow test-verified
  • Migrate a validator to ML-DSA — keygen step blocked on tooling (TODO(ERIC) in page)
  • Create an ML-DSA account — test-verified on a localnet

Remote Signing: Cosmos-KMS

  • Cosmos-KMS and remote signing (overview)
  • Remote signing tutorial — test-verified e2e
  • Configure a signing backend (AWS KMS, PKCS#11, file) — AWS/PKCS#11 sections pending execution
  • Rotate a consensus key held in Cosmos-KMS — test-verified e2e
  • Cosmos-KMS configuration reference
  • Remote signing best practices
  • Migrate from TMKMS — skeleton; TODO(ERIC) sections pending content

Module references and release-impact fixes

  • x/staking README: MsgRotateConsPubKey, KeyRotationFee, rotation stores, End-Block
  • PoA api/governance/overview: RotateConsPubKey (provisional until #26590 merges)
  • run-production: TMKMS walkthrough replaced with Cosmos-KMS links + deprecation note
  • keyring: key algorithms corrected (secp256k1 + ml_dsa_65 via --key-type), new pages linked
  • run-node: state-sync-adjacent next steps, link fixes

Upgrading

  • 0.53/0.54 → 0.55 upgrade docs — blocked on UPGRADING content (FOU-754)

Verification

Five flows run end to end against local chains, commands exactly as published:
governance param change, staking rotation, ML-DSA account, kms file-backend
tutorial, and kms rotation. Each tested page carries a hidden verification
comment with environment and commits.

  • npx mint broken-links clean

Review

  • Per-page review issues with instructions live in Linear (Ledger Security Docs
    project, FOU-739 to FOU-755): rotation set → Matt, ML-DSA + kms sets → Eric,
    release notes → Alex. Open eng questions are tracked there, and as TODO(ERIC)/
    TODO(MATT) comments in the pages.

@evanorti
evanorti marked this pull request as draft July 10, 2026 19:42
@greptile-apps

greptile-apps Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This WIP PR adds 14 new documentation pages for the 2026.1 security release, covering post-quantum ML-DSA keys, zero-downtime consensus key rotation (staking and PoA), and Cosmos-KMS remote signing. It also updates existing pages (run-production, keyring, run-node, staking/README, PoA api and overview) to reflect the new release and correct stale content.

  • Key rotation (concept + staking + PoA procedures): step-by-step guides are test-verified on a local chain; the PoA page is marked provisional against feat(enterprise/poa): consensus key rotations cosmos-sdk#26590.
  • Post-quantum keys (survey, genesis/governance enable, validator migration, user account creation): governance flow and account creation are test-verified; the validator keygen step is a documented TODO pending release tooling.
  • Cosmos-KMS remote signing (overview, file-backend tutorial, backend configuration, key rotation, config reference, best practices, TMKMS migration): file-backend tutorial and KMS rotation are verified end-to-end; AWS KMS and PKCS#11 backend sections carry explicit test TODOs; the TMKMS migration guide is a partial skeleton with clearly marked gaps.
  • All new pages carry noindex: true and docs.json registration is complete.

Confidence Score: 5/5

Safe to continue reviewing; all changes are documentation only with no executable code paths affected.

Every changed file is an MDX documentation page or a work log entry. The procedural pages that are marked test-verified carry hidden verification comments with environment details. Incomplete sections (TMKMS migration skeleton, AWS/PKCS#11 backend, ML-DSA keygen) are consistently marked with noindex: true and explicit TODO comments, so unfinished content cannot reach readers. The only finding is a style guide violation in poa/api.mdx that follows the pre-existing bold-label pattern of that file and has no reader-facing impact.

sdk/next/kms/migrate-from-tmkms.mdx has three skeleton sections (softsign key format, state migration, YubiHSM) that must be filled before the page can be published. sdk/next/enterprise/poa/api.mdx and sdk/next/keys/rotate-validator-key-poa.mdx are provisional against cosmos/cosmos-sdk#26590 and need re-verification at merge.

Important Files Changed

Filename Overview
docs.json Adds three nav groups under Run a Node (Key Rotation, Post-quantum Keys, Remote Signing: Cosmos-KMS) with all 14 new pages correctly registered; no redirects needed since paths are new.
sdk/next/keys/key-rotation.mdx New concept page explaining consensus key rotation mechanics, safety rails, and PoA vs staking differences; accurate and well-linked. Bold list items flagged in a previous review pass.
sdk/next/keys/rotate-validator-key.mdx Step-by-step staking rotation guide with test-verified commands, correct state-sync pointer, and appropriate double-sign Danger callout.
sdk/next/keys/post-quantum-keys.mdx ML-DSA survey page with accurate size/cost numbers and correct EVM/EIP roadmap description; bold Example: previously flagged.
sdk/next/keys/enable-ml-dsa-keys.mdx Governance-path guide with test-verified jq command for building the consensus param proposal; Warning callout correctly explains the null-field failure mode.
sdk/next/keys/create-ml-dsa-account.mdx Test-verified ML-DSA account creation guide; content and commands are correct.
sdk/next/kms/tutorial-file-backend.mdx End-to-end verified tutorial for Cosmos-KMS file backend; correctly documents the signer-first startup requirement and the startup-timeout edge case.
sdk/next/kms/rotate-key-remote-signer.mdx Test-verified rotation guide for a KMS-held key; Danger callout correctly warns against using show-validator on a remote-signing node.
sdk/next/kms/configuration-reference.mdx Complete kms.yaml field reference transcribed from config.go; two open TODOs on algorithm default behavior and gRPC TLS requirements are clearly marked.
sdk/next/kms/migrate-from-tmkms.mdx Skeleton migration guide with correct config mapping table and double-sign Danger callout; softsign key format conversion, state migration, and YubiHSM sections are stubs with explicit TODOs — not yet safe to publish as-is.
sdk/next/enterprise/poa/api.mdx Adds RotateConsPubKey message reference (provisional against cosmos/cosmos-sdk#26590); new section uses bold labels matching the pre-existing file style but violating the no-bold rule.
sdk/next/node/run-production.mdx TMKMS walkthrough replaced with concise Cosmos-KMS pointer and TMKMS deprecation Note; clean substitution with correct links.
sdk/next/node/keyring.mdx Corrects two pre-existing errors: fixes broken in-page anchor and updates key algorithm list to match the actual keyring source (secp256k1 + ml_dsa_65, removing the stale ed25519/--algo claim).
sdk/next/kms/best-practices.mdx Security-oriented placement and transport guide; correctly explains the SecretConnection vs Noise asymmetry and the one-signer-per-key rule with an appropriately loud Danger callout.
work-log/security-release.md Work log for the branch; entries follow the required format (date, bullet points, brief descriptions) and are thorough.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Operator submits MsgRotateConsPubKey] --> B[Chain emits power update to CometBFT:
old key → 0, new key → full power]
    B --> C{2 heights pass}
    C --> D[CometBFT activates new key
Chain stores new key]
    D --> E[Shadow node starts signing]
    D --> F[Old node stops signing]
    E --> G[Retire old node / signer]

    subgraph Zero-downtime window
        H[Old node still signing] -.-> C
        I[Shadow node syncing
non-signing full node] -.-> C
    end

    subgraph Remote signer path
        J[New key in KMS backend] --> K[Second kms process
for shadow node]
        K --> I
    end

    subgraph ML-DSA path
        L[Generate ml_dsa_65 key] --> M[Replace priv_validator_key.json
on shadow node]
        M --> I
    end
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[Operator submits MsgRotateConsPubKey] --> B[Chain emits power update to CometBFT:
old key → 0, new key → full power]
    B --> C{2 heights pass}
    C --> D[CometBFT activates new key
Chain stores new key]
    D --> E[Shadow node starts signing]
    D --> F[Old node stops signing]
    E --> G[Retire old node / signer]

    subgraph Zero-downtime window
        H[Old node still signing] -.-> C
        I[Shadow node syncing
non-signing full node] -.-> C
    end

    subgraph Remote signer path
        J[New key in KMS backend] --> K[Second kms process
for shadow node]
        K --> I
    end

    subgraph ML-DSA path
        L[Generate ml_dsa_65 key] --> M[Replace priv_validator_key.json
on shadow node]
        M --> I
    end
Loading

Reviews (2): Last reviewed commit: "update todos" | Re-trigger Greptile

Comment thread sdk/next/keys/key-rotation.mdx
@mintlify

mintlify Bot commented Jul 10, 2026

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
cosmos-docs 🟢 Ready View Preview Jul 10, 2026, 7:56 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

docs(sdk): kms rotation guide, release-impact fixes, prerequisites pass

Add rotate-key-remote-signer: rotate a consensus key held in Cosmos-KMS
(second signer process, shadow node, pubkey from /status). Test-verified
e2e on a localnet; includes the show-validator stray-key trap as a Danger.

Release-impact fixes to existing pages:
- run-production: replace the deprecated TMKMS walkthrough with
  Cosmos-KMS links and a deprecation note
- keyring: correct key algorithms to secp256k1 + ml_dsa_65 via
  --key-type (was ed25519 via --algo), link the new keys pages,
  fix a broken in-page anchor
- run-node: next-step links to key rotation and remote signing;
  jq links moved to jqlang.org

Guide improvements:
- complete, linked prerequisites on every keys/ and kms/ guide
- migrate-validator-ml-dsa: step 2 split into labeled staking / PoA /
  remote signer paths with the exact deltas (PoA passes ml_dsa_65)
- state-sync pointers on both shadow-node rotation pages
- post-quantum-keys: EIP-7702/8051/8141 linked and corrected
- Cosmos-KMS naming in prose, bare "kms" replaced with "the signer"
- remote-signing next steps completed (resolves placeholder TODO)
@evanorti

Copy link
Copy Markdown
Contributor Author

@greptileai plz rereview

Comment thread sdk/next/keys/post-quantum-keys.mdx Outdated
{/* TODO(ERIC): confirm the node-side claim above. tmkms validator addr embeds a node ID (tcp://<node-id>@host:port); kms tcp:// takes a bare host:port. Confirm the listener works unchanged for both and whether the node ID part is simply dropped in the kms config. This guide needs to be tested and verified before it can be released. */}


{/* TODO(ERIC): TMKMS's direct YubiHSM, Fortanix, and Ledger integrations do not map one-to-one onto cosmos-kms. A YubiHSM likely moves to the PKCS11 backend, but confirm the path for Ledger and Fortanix. This migration guide should give a path for each prior backend. */}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

YubiHSM and Fortanix both suppot PKCS#11 so you should be able to configure them using the pkcs11 signer in this repo.

The Ledger signer in tmkms isn't implemented in cosmos/kms. We should call out that it's explicitly out of scope right now and request people to open an issue for a support as a feature request which we will evaluate based on demand going forward.


The validator node itself needs no changes: both signers speak the same privval protocol to the same `priv_validator_laddr` listener.

{/* TODO(ERIC): confirm the node-side claim above. tmkms validator addr embeds a node ID (tcp://<node-id>@host:port); kms tcp:// takes a bare host:port. Confirm the listener works unchanged for both and whether the node ID part is simply dropped in the kms config. This guide needs to be tested and verified before it can be released. */}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cosmos/kms currently supports either tcp://host:port with no node ID. This would connect using cometbft's custom secret conn for those using cometbft's p2p networking. Or if you are running comet using libp2p, provide noise://<peer-id>@host:port. Note that this only works for those using libp2p peer IDs.


The softsign backend keeps the key in a file, so this is a format question: whether the Cosmos-KMS file backend reads a tmkms softsign key directly, or the key needs conversion.

{/* TODO(ERIC): fill this section. The kms file backend accepts a CometBFT priv_validator_key.json or a raw base64 ed25519 key (config.go FileConfig); a tmkms softsign consensus key is its own format. State whether kms reads it directly, and if not, give the exact conversion, for example via `tmkms softsign export` if that exists, or a documented openssl/one-liner. Include the command and a warning about shredding intermediate plaintext copies. */}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tmkms softsign format is not compatible with cosmos/kms's file signer.

If the tmkms key came from a priv_validator_key.json (tmkms softsign import converts from it), just point the kms key_file at the original json.

Otherwise you'll have to decode the seed from the tmkms file and re-encode it as the kms file expects. There's not support for this in the repo right now, but it should be pretty straightforward to shell script this.


### From YubiHSM or another HSM

Keys generated inside an HSM are typically non-exportable, which is the point of an HSM. A key that cannot leave the device cannot be moved into AWS KMS or another token. For those keys, migrate by rotation instead: generate a new key in the target custodian and rotate the validator to it on chain, which retires the TMKMS-held key entirely. See [Rotate a consensus key, Staking](/sdk/next/keys/rotate-validator-key).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Keys generated inside an HSM are typically non-exportable, which is the point of an HSM. A key that cannot leave the device cannot be moved into AWS KMS or another token. For those keys, migrate by rotation instead: generate a new key in the target custodian and rotate the validator to it on chain, which retires the TMKMS-held key entirely. See [Rotate a consensus key, Staking](/sdk/next/keys/rotate-validator-key).
Keys generated inside an HSM are typically non-exportable, which is the point of an HSM. A key that cannot leave the device cannot be moved into AWS KMS.
You can point Cosmos-KMS at existing HSM keys via the same PKCS#11 interface and continuing using them. For keys not falling into this category, you can migrate by rotation instead: generate a new key in the target custodian and rotate the validator to it on chain, which retires the TMKMS-held key entirely. See [Rotate a consensus key, Staking](/sdk/next/keys/rotate-validator-key).


Keys generated inside an HSM are typically non-exportable, which is the point of an HSM. A key that cannot leave the device cannot be moved into AWS KMS or another token. For those keys, migrate by rotation instead: generate a new key in the target custodian and rotate the validator to it on chain, which retires the TMKMS-held key entirely. See [Rotate a consensus key, Staking](/sdk/next/keys/rotate-validator-key).

{/* TODO(ERIC): confirm this is the recommended path for YubiHSM-held keys, or document an export path if YubiHSM wrap/export is considered acceptable here. Also state whether kms's pkcs11 backend can talk to a YubiHSM via its PKCS#11 module as a no-move option, keeping the key on the same device with the new signer. */}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since YubiHSM support PKCS11 you should be able to just wire up the config in cosmos-kms and continue using the same key.


TMKMS and Cosmos-KMS both track the last signed height, round, and step per chain, and the protection only works if the new signer starts at or above the old signer's high-water mark.

{/* TODO(ERIC): fill this section. Either kms can consume the tmkms state file (state the command or config), or the documented safe procedure is manual: stop tmkms, read its last signed height from the state file, and start kms only after the chain height has advanced past it, with the exact steps to write or verify the initial kms state file. State which one is true, and give the procedure. */}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not the same format.
LLM produces jq '{height, round: (.round|tonumber), step: (.step + 1)}' tmkms_state.json > <home>/state/<chain-id>.json as a one-liner to translate which we can put in the docs if we want.

- The validator stays dark after cutover: the new signer is not reaching the listener. Check the `validators[].addr` translation and the firewall between the hosts.
- The signer rejects the key file: the softsign format question above. Do not guess at conversions with a production key.

{/* TODO(ERIC): add the real error messages for the wrong-key-format and stale-state cases once the sections above are filled, so operators can match on them. */}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you try to point kms file signer at a tmkms softsign file you will see: file: parse key file "<path>": expected 64-byte ed25519 key, got 32

If you try to use tcp://node-id@host:port for the connection string you will see dial tcp: lookup node-id@host:port no such host

If you try to re-use the tmkms validator state file you will see chain "<chain-id>": reload sign-state: json: cannot unmarshal string into Go value of type int32


The alternative is running the signer on the validator host itself, with the key in a cloud KMS. The key still never touches disk, but the signer process and its credentials live on the exposed machine.

{/* TODO(ERIC): decision criteria between the separate-host and local-embedded layouts beyond the trust-domain argument; the PRD names both without saying when to pick which. */}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prefer to run the kms binary on a separate host with network firewall since this maximally reduces attack vectors. However, if your validator node is fronted by a sentry node then kms can be run locally as network isolation is already built into the setup. Additionally if cost efficiency is a concern, running both processes on the same host still gives the benefits of remote signing and may be an acceptable tradeoff.


Double-sign protection lives in the signer's per-chain state file, which records the highest height, round, and step ever signed. That protection assumes one writer. Two signer instances holding the same consensus key with separate or missing state files can each sign the same height, which is a double sign.

<Danger>Never run two signer instances against the same validator key. High availability by running a second signer is not redundancy; it is a double-sign generator. If availability matters more than custody, that is Horcrux's problem to solve, not a second signer's.</Danger>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not super convinced we should be recommending Horcrux to people.

| `tls_key` | string | no | TLS server private key file. |
| `keys` | list | yes | The keys the service exposes; see below. |

{/* TODO(ERIC): the repo README says TLS is mandatory for the gRPC server while config.go and the config template allow plaintext when both TLS fields are empty; confirm which is intended and update this table. */}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mandatory tls configuration was removed--it's optional now.

| `algorithm` | string | yes in practice | Key algorithm: `ed25519`, `secp256k1`, `secp256k1eth`, or `mldsa65`. Set it explicitly. |
| `key_id` | string | per backend | For `pkcs11`: hex `CKA_ID` of the key object. For `awskms`: KMS key ID, ARN, or `alias/<name>`. |

{/* TODO(ERIC): config.go documents algorithm as defaulting to ed25519, but a tested run failed validation without it on the file backend; confirm and update the Required column. */}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Defaults were removed--you need to explicitly configure the algorithm you want to use.

Comment on lines +41 to +42

{/* TODO(ERIC): the release presentation says user PQ accounts are "chain opt-in via a breaking release"; confirm whether running 0.55 is sufficient or an account-side switch exists, and document it here if so. */}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's sufficient to upgrade to 0.55 without doing any additional chain configuration, but you need the entire chain to be running it since a binary built with an older sdk version will not be able to handle mldsa signatures.

- PKCS#11: a hardware security module, signing on-device through the standard HSM interface.
- AWS KMS: a key that never leaves AWS KMS, using standard AWS credentials and IAM.

Cosmos-KMS is designed to work with existing validator infrastructure. If you run [TMKMS](https://github.com/iqlusioninc/tmkms) today, Cosmos-KMS is its successor: TMKMS is deprecated, and operators should [migrate](/sdk/next/kms/migrate-from-tmkms).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it's not deprecated, but we should say we're moving away from it and will be implementing signing support in cosmos/kms going forward.

- Signing: requests travel the privval connection, the backend signs in place, on disk, on the HSM, or inside AWS KMS, and only the signature returns. The private key never crosses the wire.
- Connection: the signer dials out, so the signing host needs no inbound ports. The address scheme selects the transport: `tcp://` uses CometBFT's SecretConnection, and `noise://` adds mutual peer pinning, where each side refuses any connection from an unexpected peer.
- Key types: `ed25519` and `mldsa65` post-quantum keys. For per-backend support, see [Configure a signing backend](/sdk/next/kms/configure-backend).
{/* TODO(ERIC): confirm which key types kms supports per backend; document it here and in the configure-backend guide. */}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

privval supports ed25519, secp256k1eth, and mldsa65

signer service suports ed25519, and secp256k1eth


The new key is `~/.kms-demo-scratch/config/priv_validator_key.json`. The rest of the scratch home is disposable.

{/* TODO(ERIC): confirm the per-backend mldsa65 support matrix for the release. kms @ 0007b0d ships mldsa65 under awskms/ and pkcs11/ (file backend run-verified 2026-07-17; HSM and AWS untested against real hardware). */}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mldsa is supported for file, awskms, and pkcs11 for privval.


The `algorithm` field is required for all backends.

{/* TODO(ERIC): config.go says algorithm defaults to ed25519; a tested run errored without it on the file backend. Confirm intended behavior, then edit the note above. */}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No defaults exist anymore.


## AWS KMS

{/* TODO(ERIC): please verify the content of this section and add any missing information. Can ML-DSA keys be used with AWS KMS? */}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes awskms includes all keytypes that cosmos-kms supports.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants