Auto-publish PyDABs reference docs to gh-pages on release - #6729
Sankalp-Mittal wants to merge 7 commits into
Conversation
Add an update-pydabs-docs workflow that rebuilds the PyDABs Python reference docs and publishes the HTML to the gh-pages branch on each release tag (and via workflow_dispatch), replacing the manual copy. Modeled on update-schema-docs.yml: same runner group, JFrog/Go/uv setup, and gh-pages worktree + github-actions[bot] commit + direct push pattern. Co-authored-by: Isaac <no-reply@databricks.com>
Sankalp-Mittal
changed the title
Integration test reportCommit: 4db1247
Top 6 slowest tests (at least 2 minutes):
|
Waiting for approvalBased on git history, these people are best suited to review:
Eligible reviewers: Suggestions based on git history. See OWNERS for ownership rules. |
…ocs' into sankalp-mittal/pydabs-docs-gitworkflow
Add -W to the pydabs-docs sphinx-build command so warnings (e.g. malformed upstream descriptions or unresolved cross-references introduced by an SDK bump) fail the build instead of passing silently. This makes both the existing python_docs CI job and the new gh-pages publish workflow strict, surfacing docs breakage at bump time. Fix the one pre-existing warning this exposes: the Resources docstring referenced add_job with :meth:, but add_job is a generated method Sphinx cannot resolve. Use an inline literal instead. Co-authored-by: Isaac <no-reply@databricks.com>
Refer to the generated per-resource adders as ``add_<resource>`` rather than singling out add_job, which was only an example among add_cluster, add_app, etc. Co-authored-by: Isaac <no-reply@databricks.com>
| - name: Setup JFrog | ||
| uses: ./.github/actions/setup-jfrog | ||
|
|
||
| - name: Setup Go |
There was a problem hiding this comment.
do we need go here? you can just call uv and then we also don't need jfrog
There was a problem hiding this comment.
We can run uv here but then this would be a secondary source of truth for the flags, the first one being in the Taskfile, over time I think that could cause divergence
| - docs/_output/** | ||
| cmds: | ||
| - "uv run --python 3.12 sphinx-build docs docs/_output --show-traceback --nitpicky --fresh-env --keep-going" | ||
| - "uv run --python 3.12 sphinx-build docs docs/_output --show-traceback --nitpicky --fresh-env --keep-going -W" |
There was a problem hiding this comment.
I think this will fail .github/workflows/python_push.yml on PRs and main (which is a good signal to gain) - could you please double check?
There was a problem hiding this comment.
Since this PR is based on the PR in which I updated the docs, so this passes on CI here
| on: | ||
| push: | ||
| tags: | ||
| - "v[0-9]+.[0-9]+.[0-9]+*" |
There was a problem hiding this comment.
what if this workflow fails, will it be retried? How will we know docs aren't updated?
There was a problem hiding this comment.
it won't be autoretried (can be done manually), if the docs aren't updated (push to gh-docs fails) this should fail, and if the docs itself on main are not updated (shouldn't happen as check on each PR) then the stale content is published. But this shouldn't happen as all PR merged would have to check for the updated docs
Align with the release-* workflow family that runs post-release, and update the internal name: to match. Co-authored-by: Isaac <no-reply@databricks.com>
PyDABs doc generation is a lightweight Sphinx build, so the -large protected runner is overkill. Drop to the standard protected runner (matching tagging.yml, which is also tag-triggered and pushes to the repo). Co-authored-by: Isaac <no-reply@databricks.com>
Full history is not needed: the build reads only the current main tree, and the release-tag lookup uses tag refs (fetch-tags), not file content at historical tags. Full history was inherited from update-schema-docs, whose since_version.go reads `git show <tag>:...` per release. Co-authored-by: Isaac <no-reply@databricks.com>
This branch has not been deployed
3 participants
Stack
.rstdoc pages from the generated modules (base)gh-pages+ make the docs build strict ← this PRChanges
Two related pieces of the PyDABs docs pipeline:
1. Auto-publish the docs to
gh-pageson release. A newupdate-pydabs-docsGitHub Actions workflow rebuilds the PyDABs Python reference docs and publishes the HTML to thegh-pagesbranch on every release tag (and on demand viaworkflow_dispatch), replacing the manual copy that had left https://databricks.github.io/cli/python/ stale (5 pages vs. the 32 the build now emits). Modeled onupdate-schema-docs.yml:pushtagsv[0-9]+.[0-9]+.[0-9]+*+workflow_dispatch.contents: write,id-token: write(setup-jfrog GOPROXY exchange).main, set up JFrog / Go / uv, thengit worktree addorigin/gh-pages.go tool -modfile=tools/task/go.mod task pydabs-docs.gh-pagespython/directory wholesale frompython/docs/_output/, so a dropped resource loses its page; top-level.nojekyll,index.html, andexperimental/are untouched.github-actions[bot]and push directly togh-pages, no-op when there is no diff.2. Make the docs build strict (
-W). Add-Wto thepydabs-docssphinx-build command so warnings — e.g. malformed upstream descriptions or unresolved cross-references introduced by an SDK bump — fail the build instead of passing silently. This makes both the existingpython_docsCI job (every PR, including SDK bumps) and the new publish workflow strict, surfacing docs breakage at bump time. Fixes the one pre-existing warning this exposes: theResourcesdocstring referencedadd_jobwith:meth:, butadd_jobis a generated method Sphinx cannot resolve — changed to an inline literal.Stacked on top of #6727, so the strict build already includes its generated pages and the override for malformed upstream descriptions.
Tests
From repo root,
./task pydabs-codegenthen./task pydabs-docsboth exit 0 under-W(theadd_jobfix + #6727's override clear the only warnings). YAML validated locally.This pull request and its description were written by Isaac.