Document generator

[unnawut]

Just an idea.

Currently we have markdown docs that are slowly getting out of sync. Then there's documentation in docstrings at module, function and field levels.

Having a document generator that consumes python specs's docstrings and produce a human-friendly docs would be good to have. With docgen we can also avoid displaying non-specs code like imports and package inits, making the readability closer to the current consensus-specs. Then we can get rid of the markdown files entirely and documentation lives in sync with the python specs.

Having the view options between individual fork diffs and merged diffs would be nice too.

[unnawut]

I tried bringing in https://github.com/SamWilsn/docc which is also used by the execution-specs but looks like I'll need some fixing.

---

1. Tried out of the box by setting [tool.docc.plugins."docc.python.discover"] to src/lean_spec:

[tool.docc.plugins."docc.python.discover"]
paths = [ "src/lean_spec" ]

and got this error:

  File "/Users/unnawut/.pyenv/versions/3.13.3/lib/python3.13/site-packages/libcst/metadata/name_provider.py", line 191, in on_visit
    FullyQualifiedNameVisitor._fully_qualify(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        self.module_name, self.package_name, qname
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/unnawut/.pyenv/versions/3.13.3/lib/python3.13/site-packages/libcst/metadata/name_provider.py", line 173, in _fully_qualify
    new_name = FullyQualifiedNameVisitor._fully_qualify_local(
        module_name, package_name, qname.name
    )
  File "/Users/unnawut/.pyenv/versions/3.13.3/lib/python3.13/site-packages/libcst/metadata/name_provider.py", line 157, in _fully_qualify_local
    raise ImportError("attempted relative import beyond top-level package")
ImportError: attempted relative import beyond top-level package

2. Setting [tool.docc.plugins."docc.python.discover"] to the project root dir:

[tool.docc.plugins."docc.python.discover"]
paths = [ "." ]

Got this error:

  File "/Users/unnawut/.pyenv/versions/3.13.3/lib/python3.13/site-packages/docc/plugins/html/__init__.py", line 913, in render_reference
    raise ReferenceError(reference.identifier, context=context) from error
docc.plugins.references.ReferenceError: in `src/lean_spec/subspecs/containers/state/state.py`, undefined identifier: `src.lean_spec.subspecs.containers.block.BlockHeader`

3. Tried replacing relative path imports with absolutes. For example:

- from ..block import Block, BlockBody, BlockHeader
+ from lean_spec.subspecs.containers.block import Block, BlockBody, BlockHeader

Got this error:

  File "/Users/unnawut/.pyenv/versions/3.13.3/lib/python3.13/site-packages/docc/plugins/html/__init__.py", line 913, in render_reference
    raise ReferenceError(reference.identifier, context=context) from error
docc.plugins.references.ReferenceError: in `src/lean_spec/subspecs/xmss/containers.py`, undefined identifier: `src.lean_spec.subspecs.koalabear.Fp`

So I stopped here because it's now touching other subspecs.

[mananuf]

I’m in favor of moving toward docstring-driven documentation so the Python specs become the single source of truth. This would eliminate the current drift between markdown and code and ensure documentation stays versioned, reviewable, and automatically updated alongside spec changes.

I'm suggesting looking at Mintlify because it supports Git-synced, versioned documentation and can render generated MDX cleanly. This makes it possible to produce human-friendly docs directly from spec docstrings, while also enabling features like per-fork and merged views without maintaining separate markdown files.

The main motivation behind this suggestion is to reduce maintenance overhead and prevent documentation divergence, while keeping the output readable and close to the current consensus-specs style by filtering out non-spec boilerplate during generation.

I think it would be worth prototyping this on a fork first to validate readability, diff views, and overall workflow before committing to a tooling change.

[unnawut]

I’m in favor of moving toward docstring-driven documentation so the Python specs become the single source of truth. This would eliminate the current drift between markdown and code and ensure documentation stays versioned, reviewable, and automatically updated alongside spec changes.

I'm suggesting looking at Mintlify because it supports Git-synced, versioned documentation and can render generated MDX cleanly. This makes it possible to produce human-friendly docs directly from spec docstrings, while also enabling features like per-fork and merged views without maintaining separate markdown files.

The main motivation behind this suggestion is to reduce maintenance overhead and prevent documentation divergence, while keeping the output readable and close to the current consensus-specs style by filtering out non-spec boilerplate during generation.

I think it would be worth prototyping this on a fork first to validate readability, diff views, and overall workflow before committing to a tooling change.

I don't really have a strong preference right now I'll defer to @tcoratger for his taste, or do you know who from STEEL team might be able to input on this?

[this-vishalsingh]

This might be helpful: https://deepwiki.com/leanEthereum/leanSpec

I use to understand/overview the protocols.