ci: run integration tests against N, N-1, N-2 Dapr runtime versions#1091
ci: run integration tests against N, N-1, N-2 Dapr runtime versions#1091ashoksmore wants to merge 3 commits into
Conversation
Signed-off-by: Ashok More <ashoksmore7@gmail.com>
ashoksmore
force-pushed
the
feature/version-compat-matrix
branch
from
b689805 to
dbaabad
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1091 +/- ##
==========================================
- Coverage 86.63% 81.92% -4.71%
==========================================
Files 84 117 +33
Lines 4473 9671 +5198
==========================================
+ Hits 3875 7923 +4048
- Misses 598 1748 +1150 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
sicoyle
left a comment
sicoyle
left a comment
There was a problem hiding this comment.
this is amazing - thank you!!! could you pls mv the python script into its own py file that we can then test and run locally more easily?
There was a problem hiding this comment.
Pull request overview
Updates the CI workflow to validate the Python SDK against the Dapr runtime compatibility window (N, N-1, N-2) by dynamically computing a runtime-version test matrix from the SDK version and dapr/dapr GitHub releases.
Changes:
- Adds a
preparejob that computes a compatibility matrix (Python 3.10–3.14 × runtime patch versions for N/N-1/N-2). - Updates
validateto consume the computed matrix and run integration + example test suites per runtime version. - Pins
setup-dapr-runtimeto an explicit runtime version per matrix entry (instead of always using “latest”).
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| run: | | ||
| if [ ${{ github.event.client_payload.command }} = "ok-to-test" ]; then | ||
| echo "CHECKOUT_REPO=${{ github.event.client_payload.pull_head_repo }}" >> $GITHUB_ENV | ||
| echo "CHECKOUT_REF=${{ github.event.client_payload.pull_head_ref }}" >> $GITHUB_ENV | ||
| fi |
| run: | | ||
| CLI_VERSION=$(curl -fsS -H "Authorization: Bearer $GITHUB_TOKEN" \ | ||
| "https://api.github.com/repos/dapr/cli/releases?per_page=10" | \ | ||
| jq -r 'map(select(.prerelease == false)) | sort_by(.created_at) | reverse | .[0].tag_name | ltrimstr("v")') | ||
| if [ -z "$CLI_VERSION" ] || [ "$CLI_VERSION" = "null" ]; then | ||
| echo "Failed to resolve Dapr CLI version" && exit 1 | ||
| if [ ${{ github.event.client_payload.command }} = "ok-to-test" ]; then | ||
| echo "CHECKOUT_REPO=${{ github.event.client_payload.pull_head_repo }}" >> $GITHUB_ENV | ||
| echo "CHECKOUT_REF=${{ github.event.client_payload.pull_head_ref }}" >> $GITHUB_ENV | ||
| fi |
| def latest_patch(runtime_minor: str) -> str | None: | ||
| prefix = f'{runtime_minor}.' | ||
| for prerelease in (False, True): | ||
| versions = [ | ||
| release['tag_name'].removeprefix('v') | ||
| for release in releases | ||
| if release.get('prerelease') == prerelease | ||
| and release['tag_name'].removeprefix('v').startswith(prefix) | ||
| ] | ||
| if versions: | ||
| return sorted(versions, key=version_key)[-1] | ||
| return None |
| matrix_include = [] | ||
| for runtime_minor in runtime_minors: | ||
| runtime_version = latest_patch(runtime_minor) | ||
| if runtime_version is None: | ||
| print(f'Warning: no Dapr runtime release found for {runtime_minor}, skipping') | ||
| continue | ||
| for python_version in python_versions: | ||
| matrix_include.append( | ||
| { | ||
| 'python_ver': python_version, | ||
| 'runtime_version': runtime_version, | ||
| } | ||
| ) | ||
|
|
||
| if not matrix_include: | ||
| raise SystemExit('No Dapr runtime releases found for compatibility matrix') |
|
@ashoksmore Great idea! I was thinking this is potentially useful for other dapr repos too, so I think it'd a good idea to make it reusable. jobs:
versions:
runs-on: ubuntu-latest
outputs:
versions: ${{ steps.v.outputs.versions }} # ["1.18.2","1.17.5","1.16.8"]
steps:
- id: v
uses: dapr/.github/.github/actions/dapr-versions@main
test:
needs: versions
strategy:
matrix:
version: ${{ fromJson(needs.versions.outputs.versions) }}
steps:
- uses: dapr/.github/.github/actions/setup-dapr-cli@main
with: { version: ${{ matrix.version }} }What do you think? |
Signed-off-by: Ashok More <ashoksmore7@gmail.com>
|
@sicoyle Thanks for your review. I've addressed your feedback. Matrix logic is in tools/compute_compat_matrix.py with unit tests in tests/test_compute_compat_matrix.py. |
|
Hi @acroca, thanks for the suggestion on a shared dapr-versions action in dapr/.github. That makes a lot of sense for reuse across repos. I kept this PR focused on the python-sdk script per @sicoyle’s feedback, but I’d be happy to help on a follow-up PR in dapr/.github if that’s the direction you’d like to go. Let me know what you prefer. |
|
Hey @ashoksmore, I really like the initiative here, but I think there's a gap in the core approach we should sort out before it lands. The matrix runs the existing tests/integration and tests/examples suites unchanged against N, N-1 and N-2. The problem is those suites already include tests for APIs that only exist in recent runtimes, like conversation (the Llama setup step), jobs, mcp, workflow, crypto and distributed-lock. Running them against 1.17 (which is what N-2 resolves to today, with VERSION at 1.19.0.dev) they'll fail just because the feature didn't exist yet, not because anything regressed. Right now there's no mechanism in the repo to skip a test by runtime version, so the matrix would be red from day one. The thing we need to solve is making a red job mean something, so it tells us "we broke a supported runtime" and not "this feature is newer than that runtime". The way I read the N-2 guarantee is that existing behavior keeps working on older runtimes, not that new features work on them, but it'd be good to agree on that explicitly. My preference would be minimum runtime markers, something like One more thing worth putting on the table. This matrix tests the newest SDK against older runtimes. The reverse case, an older SDK against the latest runtime (for example someone on SDK 1.17 upgrading their dapr to 1.19), is probably the more common real world scenario, and it doesn't need any markers because the old tests only touch features that already existed. We could cover that by having the release-1.X branches run their suites against the latest runtime. Pinning release-1.17 to runtime 1.17 specifically isn't worth much, since that pair was already green at release. Not necessarily for this PR, but we should decide which directions we actually want to guarantee. This is also separate from the shared dapr/.github action idea from before, which is more about where the version resolution lives, so both can land. What do you think? |
Description
Updates
run-tests.yamlto validate SDK compatibility against Dapr runtime versions N, N-1, and N-2 per the SDK compatibility policy.Changes:
preparejob that reads the SDKVERSIONand derives three runtime minorsdapr/daprreleasesvalidatejob matrix to run existing integration and example tests for each runtime × Python version (3.10-3.14)versiontosetup-dapr-runtimeinstead of always using the latest runtimeThe same workflow file should work on backported release branches without modification.
Issue reference
Fixes #1067
Checklist
build.yamllint/unit checks applytests/integration/andtests/examples/suites across runtime matrix (no new test files; coverage is via CI matrix)