Skip to content

fix: Forward the create-time step symbol table to openjd-sessions - #237

Open
leongdl wants to merge 6 commits into
OpenJobDescription:mainlinefrom
leongdl:fix/template-scope-let-path-format
Open

fix: Forward the create-time step symbol table to openjd-sessions#237
leongdl wants to merge 6 commits into
OpenJobDescription:mainlinefrom
leongdl:fix/template-scope-let-path-format

Conversation

@leongdl

@leongdl leongdl commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

What changed

The CLI now forwards each step's create-time resolved symbol table to openjd-sessions, at all three entry points: Session.run_task, Session.enter_environment and Session.exit_environment. An exit is keyed to the table its enter used, so an onExit resolves in the same scope as its onEnter.

A step's template-scope let (RFC 0007 §3.6) is resolved once at job creation with PathFormat.POSIX, so a create-time value cannot depend on the host that created the job. openjd-model used to also merge those bindings into script.let, and sessions re-evaluated the merged list at session time in the host's format, re-rendering its PATH values. On Windows, startswith(path("/foo/bar"), "/foo") flips from true to false between the two evaluations. Worse, the seeded value and the re-evaluated one land in the same symbol table and the re-evaluation writes last, so it overwrote the correct value.

With the merge and the re-evaluation both gone upstream, resolved_symtab is the only channel those values have into a session. create_job_with_symbol_tables replaces create_job so the tables built during instantiation come back instead of being discarded; LocalSession carries them keyed by step name, and each action looks up the one it needs.

This follows the Deadline worker agent's public commit 08a5878b, "feat: forward the resolved symbol table to the v0 session", where it replaced extra_let_bindings outright. A convergent fix that adopts an existing upstream design, not a new one.

It also fixes a live break

The CLI was still passing extra_let_bindings, which sessions removed in 0.12.0. The CLI never followed, so against a current sessions build any step with a let raised TypeError: Session.enter_environment() got an unexpected keyword argument 'extra_let_bindings'.

Users saw that as Cannot exit unknown Environment. The failed enter left the environment unregistered in the session but already recorded in the CLI's own entered list, so the step's finally exit tripped over it and masked the real error.

extra_let_bindings is removed outright rather than kept beside the new channel.

Dependency floors

Both had to move; neither existing range contained a release with the needed API.

openjd-sessions goes to >= 0.12.0, < 0.13. 0.12.0 is the first release with resolved_symtab on all three methods, and the release that removed extra_let_bindings. No single version carries both channels, so the floor has to move rather than being feature-detected.

openjd-model goes to >= 0.11.4, < 0.12, the first release exporting create_job_with_symbol_tables and JobWithSymbolTables.

Migration

Removing the merge upstream is a breaking change for any consumer that calls resolve_syntax_sugar() and does not forward a resolved symbol table to its session. Such a consumer silently loses step-level bindings rather than failing. The worker agent forwards, and this PR is the CLI following. A third-party consumer would not, and there is no in-process fallback, because no Python Step carries the resolved table.

Verification

Suite went from 2 failed / 311 passed to 318 passed. The two prior failures were the extra_let_bindings break described above.

Conformance through this branch's CLI: 1172 passed, 2 failed. expr2.2.1--string-conversion and expr2.3.2--path-construction both pass — the two fixtures that pull in opposite directions, one wanting a value that has left path-space to stay /mnt/out and the other wanting a path to render in the host's format. The 2 failures are pre-existing range-normalization cases, unrelated to this change and blocked on a spec ruling.

Upstream suites: model 5496, sessions 999.

Not verified

Windows. The companion sessions PR, OpenJobDescription/openjd-sessions-for-python#362, has had its Windows CI legs fail-fast cancelled on every round, so Windows has never been exercised by CI there. The behaviour was proven by simulation instead: with the host format forced to Windows in-process, a path-typed binding renders \a\b while a value that has left path-space stays /mnt/out.

The conformance runner concatenates a fixture's output block with its output_<os> block, so on a POSIX host an output_windows block is never asserted. A green POSIX conformance run proves nothing about the Windows expectations.


Stack note. The first three commits (af52126, a328431, ae5bdaa) are preceding local work on the RFC 0007/0008 test and comment surface, carried on this branch. 1b90918 is the change described above.

@leongdl
leongdl requested a review from a team as a code owner August 29, 2026 08:02
Comment thread src/openjd/cli/_run/_run_command.py
job: Job,
job_parameter_values: JobParameterValues,
session_id: str,
step_symbol_tables: Optional[dict[str, "SerializedSymbolTable"]] = None,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

step_symbol_tables is optional with a None default, and every downstream lookup is self._step_symbol_tables.get(step.name). Combined with the invariant this PR documents extensively — that the resolved table is the only channel a step's template-scope let has into the session — that means any caller that omits it (or any step name missing from the mapping) gets a session where the step's let names are silently undefined rather than an error. The failure surfaces as an unresolved-symbol error from deep inside the session, or worse, as an action that runs with the wrong scope.

do_run is the only production caller and it always has the tables, so consider making the parameter required (keyword-only, no default) so the type checker catches an omission at the call site instead of at action-evaluation time. _run_local_session (_run_command.py:328) has the same Optional[...] = None default for the same reason.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit, not taken. do_run is the only first-party caller and it always passes the full mapping returned by create_job_with_symbol_tables, so a required keyword-only parameter would buy type-checker coverage on a call site that is already correct while breaking any out-of-tree caller of _run_local_session. The degradation is documented at the assignment and the failure is loud rather than silent — an unresolved name surfaces as ActionState.FAILED with Undefined variable in the action log, not as a wrong-scope run.

Comment thread test/openjd/cli/conftest.py Outdated
# their variables and actions. The source expressions are not
# re-evaluated anywhere downstream, so a missing table here means the
# step's `let` names are undefined rather than merely stale.
resolved_symtab = self._step_symbol_tables.get(step.name)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both lookup sites (run_step here and run_task at line 391) use .get(step.name), so a step name absent from step_symbol_tables degrades to resolved_symtab=None silently. Per this PR's own documented invariant, that is not a benign degradation — the step's let names become undefined, and the error surfaces later as an unresolved-symbol failure inside the session action, pointing at the template rather than at the missing table.

Since create_job_with_symbol_tables is expected to return a table for every step of the job it built, a missing key means the mapping and the Job came from different creations (or a step name was transformed between them). Consider self._step_symbol_tables[step.name] — a KeyError at the boundary is far more diagnosable than a template-scope let quietly evaluating to nothing.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit, not taken, for the same reason as the sibling thread. Switching to self._step_symbol_tables[step.name] would trade a documented None degradation that already fails loudly — ActionState.FAILED carrying Undefined variable — for a KeyError at the boundary, with no user-facing gain given the single first-party caller always supplies a table for every step of the job it built. Left as .get() with the invariant recorded at the assignment.

Comment thread src/openjd/cli/_run/_local_session/_session_manager.py Outdated
Comment thread test/openjd/cli/test_run_command.py Outdated
Comment thread test/openjd/cli/templates/job_sleep_exit_normal.yaml
@leongdl
leongdl enabled auto-merge (squash) August 31, 2026 23:36

@seant-aws seant-aws left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent edge-case testing — 14 templates, all pass

Checked out the PR locally, resolved openjd-sessions 0.12.0 + openjd-model 0.11.6, and ran the PR's own tests (5/5 pass) plus 14 additional templates targeting the untested surface of the _entered_env_symtabs dict-keying and symbol-table scoping:

Dict-keying (env_id has no step component):

  • Duplicate step-env names in one step → model rejects ("Duplicate values for name are not allowed") — double-pop bug is schema-unreachable
  • Two steps sharing a step-env name with different let values → AlphaExit=ALPHAVAL / BetaExit=BETAVAL, not crossed (each step fully exits before the next enters)
  • Three distinct step envs → LIFO exit pairing correct, all resolve
  • onExit after onEnter action fails (exit code 3) → ExitAfterFail=FAILVAL — retained key works as designed
  • Pre-registration failure (RFC 0008 wrap conflict on a step env with let) → stale key created but provably inert (job aborts, cleanup never pops it)

Scoping:

  • Cross-step let contamination → model rejects ("Variable avar does not exist... Did you mean: bvar")
  • Step with no let but with a step env → table present anyway (Step.Name resolves) — confirmed create_job_with_symbol_tables populates a table for every step unconditionally
  • Chained bindings (a=2, b=a*5) → Chained=10
  • 3-task parameter space → all tasks see the let value
  • PATH-typed binding (path("/foo/bar"), startswith) → resolves correctly on Linux
  • let shadowing a job parameter → structurally impossible (capitalized names are reserved namespaces)
  • Step named My Step-Name → works, including --step selection

No bugs found. The forwarding chain is correct end-to-end. The .get() / optional decisions are defensible — degradation is loud (ActionState.FAILED / Undefined variable), and the table is always present for any real step. LGTM.

…er-failure cleanup

Six behaviours the RFC 0007/0008 CLI work introduced were unpinned: reverting
the template-declared-extensions fix entirely still passed all 307 tests. Each
new test drives a real template through do_run and asserts on observed output.

- Extension behaviours activate only when a template DECLARES the extension,
  never from the CLI's --extensions accept-list. A job template declaring no
  extensions must not get REDACTED_ENV_VARS even though --extensions defaults
  to every supported one. Kills: supported_extensions=extensions in
  _run_command (the pre-fix behaviour).
- The declared set is the UNION over the job template and every external
  environment template. Only the environment template declares
  REDACTED_ENV_VARS, and redaction is active for the whole session. Kills:
  dropping the environment-template union loop.
- Job.Name is seeded into the session symbol table, so a step-level EXPR `let`
  binding resolves it. Asserts the printed VALUE equals the job's name, so
  neither omitting job_name= from Session() nor passing an empty string
  satisfies it.
- The step name reaches Session.run_task and feeds WrappedStep.Name inside an
  active onWrapTaskRun hook. Asserts the resolved value equals the step's real
  name. Kills: passing a wrong constant for step_name.
- Step.Name is out of scope outside a step: a job environment referencing it is
  rejected. NOTE this is openjd-model static validation at template-read time,
  so it does NOT pin the CLI's non-seeding of step_name on job/external enters
  -- no loadable template can observe that, because the same rejection applies
  to every environment-template location including RFC 0008 wrap hooks. The
  docstring records the limit; the CLI's decision stays pinned only by
  test_localsession_step_env_enter_receives_step_name.
- An enter that raises BEFORE the session registers the environment (two
  external wrap-hook environments hitting RFC 0008's "at most one") must drop
  it from the CLI's entered list, so the real error surfaces instead of a
  masking "Cannot exit unknown Environment". Kills: never popping on raise.

Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
The enter-failure comment named two triggers that cannot reach it: at the
pinned openjd-sessions floor (0.10.11) enter_environment has no `raise`
after it registers the environment -- every failure it detects past that
point routes through _fail_action_before_start() and returns normally
(verified by AST-walking the function: raises at 780/782/793, append at
812, none after). The post-registration branch is therefore defensive, and
now says so rather than implying a reachable path.

The dependency-floor comment named run_task as what breaks below 0.10.11,
but `job_name` is passed to Session() unconditionally, so construction
raises TypeError first; `extra_let_bindings` is also 0.10.11-only.

The forwarding comment explained that None is not passed without saying
that pthat pthat pthat pteqthat pthat pthat pthat pteqthat pthat pthat pthat pteqthat pthat pthat pthat pteqthat pthat pthat pthat pteqthat pthat pthaion pins it.

Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
Both tests asserted on a race against a tight action timeout and failed
intermittently once this branch added 12 tests to the 12-worker parallel
run: job_sleep_exit_normal left under 2s for interpreter startup before
the timeout killed the action, so 'SLEEP' was never printed; and
feature_bundle_1_timeout gave a bash action that sleeps 1s a 5s budget,
which a loaded run exceeded, turning exit 0 into exit 1.

Measured before: the full suite failed 2 of 3 runs on this branch while
the base commit passed 3 of 3 (301 tests vs 313 -- the added contention,
not a behaviour change; both tests pass 5/5 in isolation on both trees).
After: 3 of 3 runs green.

Neither test's subject changes. TaskTimeout still times out, because the
sleep is now an order of magnitude longer than the timesleep is now an order of magnitude longer than the timesleep is now an order of magnitude longer than the timesleep is now an order of magnitude longer thanit
asserts moves, 5s to 20s.

Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
A step's template-scope `let` (RFC 0005 §3.6) is resolved once at job
creation. openjd-model no longer merges the resolved bindings into
`script.let` and openjd-sessions no longer re-evaluates the source
expressions, so the only channel those values have into a session is the
`resolved_symtab` argument. The CLI forwarded nothing, so a step-level
`let` produced no bindings at all.

It also still passed `extra_let_bindings`, which openjd-sessions removed in
0.12.0 in favour of `resolved_symtab`. That raised TypeError for any step
with a `let`, which surfaced as "Cannot exit unknown Environment" once the
failed enter left the environment unregistered.

Take the tables from `create_job_with_symbol_tables` (same job as
`create_job`, tables returned instead of discarded), carry them on
LocalSession keyed by step name, and forward the right one at all three
entry points: `run_task`, `enter_environment` for a step's environments,
and `exit_environment` with the same table the enter used. The values are
already `SerializedSymbolTable`, which is what the sessions API takes, so
nothing is converted. `extra_let_bindings` is removed rather than kept
beside the new channel.

Raise both floors, since neither range contained a release with the API:
openjd-sessions to >= 0.12.0 (first release with `resolved_symtab` on all
three entry points; 0.10.11 through 0.11.0 have `extra_let_bindings` and no
`resolved_symtab`, so no version satisfies both) and openjd-model to
>= 0.11.4 (first release exporting `create_job_with_symbol_tables`).

Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
The note in `sample_job_and_dirs` sent readers to a
`sample_job_with_symtabs` fixture "below" that exists nowhere in the
suite. The fixture that actually carries the per-step resolved symbol
tables is `step_let_job` in test_step_symbol_tables.py, which builds them
through a real `job_from_template` call.

Comment only; no behaviour change.

The CodeQL "unused import" report for `SerializedSymbolTable` in
_run/_run_command.py is a false positive and is deliberately not acted
on: the import is already guarded by `if TYPE_CHECKING`, and the quoted
annotation on `_run_local_session`'s `step_symbol_tables` parameter needs
it. Removing it makes mypy fail with `Name "SerializedSymbolTable" is not
defined`.

Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
Three text corrections, no behaviour change.

The enter_environment rationale cited openjd-sessions 0.10.11 for "nothing
raises post-registration" while pyproject now pins >= 0.12.0. The claim still
holds at 0.12.0 (verified against the released wheel: every raise in
enter_environment precedes registration, and the post-registration
resolved_symtab failure calls _fail_action_before_start and returns), so only
the citation moves.

test_do_run_job_name_in_step_let_binding claimed neither omitting job_name from
the Session nor passing an empty one would satisfy it. Measured otherwise:
deleting job_name=str(job.name) leaves the test passing, because the binding is
step-level and openjd-model resolves it at job creation, so the value is already
a literal in the resolved table before Session.__init__ runs. Reworded to the
create-time forward path it does pin: model to step_symbol_tables to CLI to
step-environment enter.

test_do_run_step_name_in_step_environment had the same shape, calling itself
"the end-to-end proof of the feature". Dropping step_name=step_name on the
enter also leaves it passing (measured); that keyword is pinned by
test_localsession_step_env_enter_receives_step_name. Scoped accordingly.

Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
@leongdl
leongdl force-pushed the fix/template-scope-let-path-format branch from c3cc217 to 65c4ad1 Compare September 1, 2026 20:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants