Improve the monorepo release process

[yaythomas]

Problem

Python has no native tooling for releasing multiple packages from one monorepo. There is no equivalent of npm workspaces with per-package publish orchestration. This repository works around the gap with a GitHub Actions workflow, pypi-publish.yml. The workflow builds and publishes every package on every GitHub release. This design has three problems.

1. The release tag does not control what gets published.

The workflow triggers on release: published. It uses the tag only as a checkout ref. No filter connects the tag name to the package matrix. So a release tagged sdk-v2.0.0 builds and uploads all four packages: aws-durable-execution-sdk-python, -otel, -testing, and -insight.

The real selector is each package's __about__.py version. A package publishes when its version is absent from PyPI. A package fails when its version is already there. So the set of published packages depends on which version bumps sit on the tagged commit. It does not depend on what the tag says. A maintainer who intends to release only the SDK can unintentionally publish other packages.

One workflow does read the tag. lambda-layer-publish.yml gates on contains(tag_name, 'otel-v'). That gate forced the combined tag convention sdk-v1.7.0,otel-v0.3.0. The convention packs several package names into one git tag with commas.

2. Already-published versions fail the workflow instead of being skipped.

The publish step does not check PyPI before uploading. It does not set skip-existing. A package with an unchanged version is rebuilt and re-uploaded. PyPI rejects the upload with 400: Uploading new files to releases older than 14 days is not allowed. In practice the -testing package hits this on most releases. So every release since the monorepo conversion has finished with a failed workflow run. This holds even when the intended packages published successfully. A release workflow that is always red trains maintainers to ignore failures. Ignored failures hide real ones.

3. There is no pre-flight plan and no post-publish verification.

Nothing reports which packages will publish before uploads begin. Nothing confirms afterward that PyPI serves the new versions. The first signal of a wrong or missing publish is a user report, or a red job found by reading logs.

How the JS repository avoids this

aws-durable-execution-sdk-js has the same outer shape. A release event triggers a workflow that iterates all publishable packages. The difference is its publish script, iterate-publish-npm.sh. The script runs in two phases.

Phase one is a pre-flight. Before any upload, the script reads the npm registry state for every package. It decides a plan per package: publish under a computed dist-tag, skip because the version is already published, or reject for a policy violation. A reject aborts the run before any upload. An unreadable registry also aborts the run.

Phase two publishes according to the plan. A skip is a deliberate decision and the run stays green. After each upload the script polls the registry until the dist-tag points at the published version.

This makes releases idempotent. A release evaluates six packages and publishes only the ones whose versions moved. Single-package releases such as sdk-2.3.1 and combined releases such as sdk-2.4.0/test-1.1.4/otel-1.0.0 go through the same workflow.

The JS model still does not make the tag authoritative. Package selection there is also version-file-driven. The proposal below goes one step further.

Proposal

1. Make the tag the source of truth for package selection. Define one tag grammar. For example, sdk-v2.0.0 names a single package and sdk-v2.0.0+otel-v1.0.0 names a combined release. A preflight script parses the tag. Only the named packages are built and published. If a tagged version disagrees with the package's __about__.py, the preflight fails before any upload.
2. Add a preflight plan job. For each selected package, compare __about__.py against the PyPI JSON API. Emit a plan of publish, skip, or error to the job summary. Downstream jobs consume the plan.
3. Make publishing idempotent. Set skip-existing: true on pypa/gh-action-pypi-publish. A re-run of a partially failed release then republishes only what is missing.
4. Verify after publishing. Poll the PyPI JSON API until the new version is served. Bound the retries.
5. Apply the same tag gate to the downstream release workflows. These are the Lambda layer and ECR emulator image workflows. An SDK-only tag must not rebuild or retag unrelated artifacts.

Acceptance criteria

1. No tag/publish discrepancy. A release tagged for package X publishes exactly X. Publishing a package not named in the tag is impossible. This holds regardless of which version bumps exist on the tagged commit.
2. Protection against accidental and inconsistent publishes. The preflight aborts before any upload when a tagged version disagrees with the source version. It also aborts when a policy check fails. Nothing is published on abort.
3. Successful workflow runs on every legitimate release. An already-published version is a reported skip, not a failure. A green run means everything intended was published and verified. A red run means something went wrong. Re-running a release is safe and idempotent.