Skip to content

feat(pyamber): add workflow core runtime - #8583

Draft
carloea2 wants to merge 3 commits into
apache:mainfrom
carloea2:codex/feat-pytexera-workflow-runtime
Draft

carloea2 wants to merge 3 commits into
apache:mainfrom
carloea2:codex/feat-pytexera-workflow-runtime

Conversation

@carloea2

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

Add the core runtime for generated Python workflow drivers: a local variable heap, explicit input/output boundaries, field ownership checks, and a fresh global-binding namespace per execution. Invalid boundary IDs and incomplete output contracts fail explicitly.

The tuple-operator adapter and native binding helpers follow in PR 3B.

Any related issues, documentation, discussions?

Related to #8319 (PR 3A). Depends on #8433; keep this PR in draft until the codec prerequisite merges and this branch is rebased.

The runtime slice adds 611 lines across two files (303 implementation, 308 tests). The current diff also includes the codec prerequisite: 938 additions across five files.

How was this PR tested?

From amber/, Python 3.12:

python -m pytest src/test/python/pytexera/workflow -m "not integration" -q -p no:cacheprovider --cov=pytexera.workflow.runtime --cov-branch --cov-report=term-missing
python -m ruff check src/main/python/pytexera/workflow src/test/python/pytexera/workflow
python -m ruff format --check src/main/python/pytexera/workflow src/test/python/pytexera/workflow

54 tests passed; core runtime statement and branch coverage is 100%. Lint and formatting passed. Regression tests reproduced acceptance of non-string boundary IDs before the validation fix. Coverage includes missing/duplicate boundaries, ownership conflicts, absent bindings, malformed payload recovery, and execution namespace isolation.

Local verification used an existing generated-protobuf directory on a secondary Python path; the runtime and codec were loaded from this release checkout. The coverage run emitted a NumPy reload warning. Full integration and cross-version results will come from CI.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (GPT-6)

@github-actions

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • No candidates found from git blame history.

@github-actions github-actions Bot added dependencies Pull requests that update a dependency file pyamber labels Sep 18, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.04950% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.77%. Comparing base (378a3b5) to head (10d2c9e).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
amber/src/main/python/pytexera/workflow/codec.py 86.30% 5 Missing and 5 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##               main    #8583    +/-   ##
==========================================
  Coverage     92.76%   92.77%            
  Complexity     4898     4898            
==========================================
  Files          1236     1238     +2     
  Lines         51962    52164   +202     
  Branches       6381     6420    +39     
==========================================
+ Hits          48205    48394   +189     
- Misses         2185     2192     +7     
- Partials       1572     1578     +6     
Flag Coverage Δ *Carryforward flag
access-control-service 71.78% <ø> (ø) Carriedforward from 378a3b5
agent-service 99.32% <ø> (ø) Carriedforward from 378a3b5
amber 88.60% <ø> (ø) Carriedforward from 378a3b5
computing-unit-managing-service 54.61% <ø> (ø) Carriedforward from 378a3b5
config-service 87.37% <ø> (ø) Carriedforward from 378a3b5
file-service 81.53% <ø> (ø) Carriedforward from 378a3b5
frontend 96.60% <ø> (ø) Carriedforward from 378a3b5
notebook-migration-service 83.73% <ø> (ø) Carriedforward from 378a3b5
pyamber 98.27% <95.04%> (-0.21%) ⬇️
workflow-compiling-service 74.09% <ø> (ø) Carriedforward from 378a3b5

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 2 better · 🔴 6 worse · ⚪ 7 noise (<±5%) · 0 without baseline

Compared against main 378a3b5 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 430 0.263 22,682/27,114/27,114 us 🟢 -22.4% / 🔴 +90.0%
🔴 bs=100 sw=10 sl=64 922 0.563 108,388/136,982/136,982 us 🔴 +7.4% / 🔴 +37.1%
🔴 bs=1000 sw=10 sl=64 1,095 0.669 906,955/1,052,465/1,052,465 us 🔴 +8.5% / 🔴 +9.7%
Baseline details

Latest main 378a3b5 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 430 tuples/sec 454 tuples/sec 839.58 tuples/sec -5.3% -48.8%
bs=10 sw=10 sl=64 MB/s 0.263 MB/s 0.277 MB/s 0.512 MB/s -5.1% -48.7%
bs=10 sw=10 sl=64 p50 22,682 us 20,664 us 11,937 us +9.8% +90.0%
bs=10 sw=10 sl=64 p95 27,114 us 34,955 us 14,976 us -22.4% +81.1%
bs=10 sw=10 sl=64 p99 27,114 us 34,955 us 19,204 us -22.4% +41.2%
bs=100 sw=10 sl=64 throughput 922 tuples/sec 968 tuples/sec 1,096 tuples/sec -4.8% -15.9%
bs=100 sw=10 sl=64 MB/s 0.563 MB/s 0.591 MB/s 0.669 MB/s -4.7% -15.8%
bs=100 sw=10 sl=64 p50 108,388 us 100,892 us 94,033 us +7.4% +15.3%
bs=100 sw=10 sl=64 p95 136,982 us 132,616 us 99,939 us +3.3% +37.1%
bs=100 sw=10 sl=64 p99 136,982 us 132,616 us 110,148 us +3.3% +24.4%
bs=1000 sw=10 sl=64 throughput 1,095 tuples/sec 1,114 tuples/sec 1,128 tuples/sec -1.7% -2.9%
bs=1000 sw=10 sl=64 MB/s 0.669 MB/s 0.68 MB/s 0.688 MB/s -1.6% -2.8%
bs=1000 sw=10 sl=64 p50 906,955 us 896,300 us 921,191 us +1.2% -1.5%
bs=1000 sw=10 sl=64 p95 1,052,465 us 970,150 us 959,735 us +8.5% +9.7%
bs=1000 sw=10 sl=64 p99 1,052,465 us 970,150 us 988,604 us +8.5% +6.5%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,464.79,200,128000,430,0.263,22681.57,27113.90,27113.90
1,100,10,64,20,2169.81,2000,1280000,922,0.563,108388.12,136981.72,136981.72
2,1000,10,64,20,18259.62,20000,12800000,1095,0.669,906955.49,1052465.39,1052465.39

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file pyamber

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants