fix: Forward the create-time step symbol table to openjd-sessions - #237
fix: Forward the create-time step symbol table to openjd-sessions#237leongdl wants to merge 6 commits into
Conversation
| job: Job, | ||
| job_parameter_values: JobParameterValues, | ||
| session_id: str, | ||
| step_symbol_tables: Optional[dict[str, "SerializedSymbolTable"]] = None, |
There was a problem hiding this comment.
step_symbol_tables is optional with a None default, and every downstream lookup is self._step_symbol_tables.get(step.name). Combined with the invariant this PR documents extensively — that the resolved table is the only channel a step's template-scope let has into the session — that means any caller that omits it (or any step name missing from the mapping) gets a session where the step's let names are silently undefined rather than an error. The failure surfaces as an unresolved-symbol error from deep inside the session, or worse, as an action that runs with the wrong scope.
do_run is the only production caller and it always has the tables, so consider making the parameter required (keyword-only, no default) so the type checker catches an omission at the call site instead of at action-evaluation time. _run_local_session (_run_command.py:328) has the same Optional[...] = None default for the same reason.
There was a problem hiding this comment.
Nit, not taken. do_run is the only first-party caller and it always passes the full mapping returned by create_job_with_symbol_tables, so a required keyword-only parameter would buy type-checker coverage on a call site that is already correct while breaking any out-of-tree caller of _run_local_session. The degradation is documented at the assignment and the failure is loud rather than silent — an unresolved name surfaces as ActionState.FAILED with Undefined variable in the action log, not as a wrong-scope run.
| # their variables and actions. The source expressions are not | ||
| # re-evaluated anywhere downstream, so a missing table here means the | ||
| # step's `let` names are undefined rather than merely stale. | ||
| resolved_symtab = self._step_symbol_tables.get(step.name) |
There was a problem hiding this comment.
Both lookup sites (run_step here and run_task at line 391) use .get(step.name), so a step name absent from step_symbol_tables degrades to resolved_symtab=None silently. Per this PR's own documented invariant, that is not a benign degradation — the step's let names become undefined, and the error surfaces later as an unresolved-symbol failure inside the session action, pointing at the template rather than at the missing table.
Since create_job_with_symbol_tables is expected to return a table for every step of the job it built, a missing key means the mapping and the Job came from different creations (or a step name was transformed between them). Consider self._step_symbol_tables[step.name] — a KeyError at the boundary is far more diagnosable than a template-scope let quietly evaluating to nothing.
There was a problem hiding this comment.
Nit, not taken, for the same reason as the sibling thread. Switching to self._step_symbol_tables[step.name] would trade a documented None degradation that already fails loudly — ActionState.FAILED carrying Undefined variable — for a KeyError at the boundary, with no user-facing gain given the single first-party caller always supplies a table for every step of the job it built. Left as .get() with the invariant recorded at the assignment.
seant-aws
left a comment
There was a problem hiding this comment.
Independent edge-case testing — 14 templates, all pass
Checked out the PR locally, resolved openjd-sessions 0.12.0 + openjd-model 0.11.6, and ran the PR's own tests (5/5 pass) plus 14 additional templates targeting the untested surface of the _entered_env_symtabs dict-keying and symbol-table scoping:
Dict-keying (env_id has no step component):
- Duplicate step-env names in one step → model rejects ("Duplicate values for name are not allowed") — double-pop bug is schema-unreachable
- Two steps sharing a step-env name with different
letvalues →AlphaExit=ALPHAVAL/BetaExit=BETAVAL, not crossed (each step fully exits before the next enters) - Three distinct step envs → LIFO exit pairing correct, all resolve
onExitafteronEnteraction fails (exit code 3) →ExitAfterFail=FAILVAL— retained key works as designed- Pre-registration failure (RFC 0008 wrap conflict on a step env with
let) → stale key created but provably inert (job aborts, cleanup never pops it)
Scoping:
- Cross-step
letcontamination → model rejects ("Variable avar does not exist... Did you mean: bvar") - Step with no
letbut with a step env → table present anyway (Step.Nameresolves) — confirmedcreate_job_with_symbol_tablespopulates a table for every step unconditionally - Chained bindings (
a=2, b=a*5) →Chained=10 - 3-task parameter space → all tasks see the
letvalue - PATH-typed binding (
path("/foo/bar"),startswith) → resolves correctly on Linux letshadowing a job parameter → structurally impossible (capitalized names are reserved namespaces)- Step named
My Step-Name→ works, including--stepselection
No bugs found. The forwarding chain is correct end-to-end. The .get() / optional decisions are defensible — degradation is loud (ActionState.FAILED / Undefined variable), and the table is always present for any real step. LGTM.
…er-failure cleanup Six behaviours the RFC 0007/0008 CLI work introduced were unpinned: reverting the template-declared-extensions fix entirely still passed all 307 tests. Each new test drives a real template through do_run and asserts on observed output. - Extension behaviours activate only when a template DECLARES the extension, never from the CLI's --extensions accept-list. A job template declaring no extensions must not get REDACTED_ENV_VARS even though --extensions defaults to every supported one. Kills: supported_extensions=extensions in _run_command (the pre-fix behaviour). - The declared set is the UNION over the job template and every external environment template. Only the environment template declares REDACTED_ENV_VARS, and redaction is active for the whole session. Kills: dropping the environment-template union loop. - Job.Name is seeded into the session symbol table, so a step-level EXPR `let` binding resolves it. Asserts the printed VALUE equals the job's name, so neither omitting job_name= from Session() nor passing an empty string satisfies it. - The step name reaches Session.run_task and feeds WrappedStep.Name inside an active onWrapTaskRun hook. Asserts the resolved value equals the step's real name. Kills: passing a wrong constant for step_name. - Step.Name is out of scope outside a step: a job environment referencing it is rejected. NOTE this is openjd-model static validation at template-read time, so it does NOT pin the CLI's non-seeding of step_name on job/external enters -- no loadable template can observe that, because the same rejection applies to every environment-template location including RFC 0008 wrap hooks. The docstring records the limit; the CLI's decision stays pinned only by test_localsession_step_env_enter_receives_step_name. - An enter that raises BEFORE the session registers the environment (two external wrap-hook environments hitting RFC 0008's "at most one") must drop it from the CLI's entered list, so the real error surfaces instead of a masking "Cannot exit unknown Environment". Kills: never popping on raise. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
The enter-failure comment named two triggers that cannot reach it: at the pinned openjd-sessions floor (0.10.11) enter_environment has no `raise` after it registers the environment -- every failure it detects past that point routes through _fail_action_before_start() and returns normally (verified by AST-walking the function: raises at 780/782/793, append at 812, none after). The post-registration branch is therefore defensive, and now says so rather than implying a reachable path. The dependency-floor comment named run_task as what breaks below 0.10.11, but `job_name` is passed to Session() unconditionally, so construction raises TypeError first; `extra_let_bindings` is also 0.10.11-only. The forwarding comment explained that None is not passed without saying that pthat pthat pthat pteqthat pthat pthat pthat pteqthat pthat pthat pthat pteqthat pthat pthat pthat pteqthat pthat pthat pthat pteqthat pthat pthaion pins it. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
Both tests asserted on a race against a tight action timeout and failed intermittently once this branch added 12 tests to the 12-worker parallel run: job_sleep_exit_normal left under 2s for interpreter startup before the timeout killed the action, so 'SLEEP' was never printed; and feature_bundle_1_timeout gave a bash action that sleeps 1s a 5s budget, which a loaded run exceeded, turning exit 0 into exit 1. Measured before: the full suite failed 2 of 3 runs on this branch while the base commit passed 3 of 3 (301 tests vs 313 -- the added contention, not a behaviour change; both tests pass 5/5 in isolation on both trees). After: 3 of 3 runs green. Neither test's subject changes. TaskTimeout still times out, because the sleep is now an order of magnitude longer than the timesleep is now an order of magnitude longer than the timesleep is now an order of magnitude longer than the timesleep is now an order of magnitude longer thanit asserts moves, 5s to 20s. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
A step's template-scope `let` (RFC 0005 §3.6) is resolved once at job creation. openjd-model no longer merges the resolved bindings into `script.let` and openjd-sessions no longer re-evaluates the source expressions, so the only channel those values have into a session is the `resolved_symtab` argument. The CLI forwarded nothing, so a step-level `let` produced no bindings at all. It also still passed `extra_let_bindings`, which openjd-sessions removed in 0.12.0 in favour of `resolved_symtab`. That raised TypeError for any step with a `let`, which surfaced as "Cannot exit unknown Environment" once the failed enter left the environment unregistered. Take the tables from `create_job_with_symbol_tables` (same job as `create_job`, tables returned instead of discarded), carry them on LocalSession keyed by step name, and forward the right one at all three entry points: `run_task`, `enter_environment` for a step's environments, and `exit_environment` with the same table the enter used. The values are already `SerializedSymbolTable`, which is what the sessions API takes, so nothing is converted. `extra_let_bindings` is removed rather than kept beside the new channel. Raise both floors, since neither range contained a release with the API: openjd-sessions to >= 0.12.0 (first release with `resolved_symtab` on all three entry points; 0.10.11 through 0.11.0 have `extra_let_bindings` and no `resolved_symtab`, so no version satisfies both) and openjd-model to >= 0.11.4 (first release exporting `create_job_with_symbol_tables`). Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
The note in `sample_job_and_dirs` sent readers to a `sample_job_with_symtabs` fixture "below" that exists nowhere in the suite. The fixture that actually carries the per-step resolved symbol tables is `step_let_job` in test_step_symbol_tables.py, which builds them through a real `job_from_template` call. Comment only; no behaviour change. The CodeQL "unused import" report for `SerializedSymbolTable` in _run/_run_command.py is a false positive and is deliberately not acted on: the import is already guarded by `if TYPE_CHECKING`, and the quoted annotation on `_run_local_session`'s `step_symbol_tables` parameter needs it. Removing it makes mypy fail with `Name "SerializedSymbolTable" is not defined`. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
Three text corrections, no behaviour change. The enter_environment rationale cited openjd-sessions 0.10.11 for "nothing raises post-registration" while pyproject now pins >= 0.12.0. The claim still holds at 0.12.0 (verified against the released wheel: every raise in enter_environment precedes registration, and the post-registration resolved_symtab failure calls _fail_action_before_start and returns), so only the citation moves. test_do_run_job_name_in_step_let_binding claimed neither omitting job_name from the Session nor passing an empty one would satisfy it. Measured otherwise: deleting job_name=str(job.name) leaves the test passing, because the binding is step-level and openjd-model resolves it at job creation, so the value is already a literal in the resolved table before Session.__init__ runs. Reworded to the create-time forward path it does pin: model to step_symbol_tables to CLI to step-environment enter. test_do_run_step_name_in_step_environment had the same shape, calling itself "the end-to-end proof of the feature". Dropping step_name=step_name on the enter also leaves it passing (measured); that keyword is pinned by test_localsession_step_env_enter_receives_step_name. Scoped accordingly. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
c3cc217 to
65c4ad1
Compare
What changed
The CLI now forwards each step's create-time resolved symbol table to openjd-sessions, at all three entry points:
Session.run_task,Session.enter_environmentandSession.exit_environment. An exit is keyed to the table its enter used, so anonExitresolves in the same scope as itsonEnter.A step's template-scope
let(RFC 0007 §3.6) is resolved once at job creation withPathFormat.POSIX, so a create-time value cannot depend on the host that created the job. openjd-model used to also merge those bindings intoscript.let, and sessions re-evaluated the merged list at session time in the host's format, re-rendering its PATH values. On Windows,startswith(path("/foo/bar"), "/foo")flips from true to false between the two evaluations. Worse, the seeded value and the re-evaluated one land in the same symbol table and the re-evaluation writes last, so it overwrote the correct value.With the merge and the re-evaluation both gone upstream,
resolved_symtabis the only channel those values have into a session.create_job_with_symbol_tablesreplacescreate_jobso the tables built during instantiation come back instead of being discarded;LocalSessioncarries them keyed by step name, and each action looks up the one it needs.This follows the Deadline worker agent's public commit
08a5878b, "feat: forward the resolved symbol table to the v0 session", where it replacedextra_let_bindingsoutright. A convergent fix that adopts an existing upstream design, not a new one.It also fixes a live break
The CLI was still passing
extra_let_bindings, which sessions removed in 0.12.0. The CLI never followed, so against a current sessions build any step with aletraisedTypeError: Session.enter_environment() got an unexpected keyword argument 'extra_let_bindings'.Users saw that as
Cannot exit unknown Environment. The failed enter left the environment unregistered in the session but already recorded in the CLI's own entered list, so the step'sfinallyexit tripped over it and masked the real error.extra_let_bindingsis removed outright rather than kept beside the new channel.Dependency floors
Both had to move; neither existing range contained a release with the needed API.
openjd-sessionsgoes to>= 0.12.0, < 0.13. 0.12.0 is the first release withresolved_symtabon all three methods, and the release that removedextra_let_bindings. No single version carries both channels, so the floor has to move rather than being feature-detected.openjd-modelgoes to>= 0.11.4, < 0.12, the first release exportingcreate_job_with_symbol_tablesandJobWithSymbolTables.Migration
Removing the merge upstream is a breaking change for any consumer that calls
resolve_syntax_sugar()and does not forward a resolved symbol table to its session. Such a consumer silently loses step-level bindings rather than failing. The worker agent forwards, and this PR is the CLI following. A third-party consumer would not, and there is no in-process fallback, because no PythonStepcarries the resolved table.Verification
Suite went from 2 failed / 311 passed to 318 passed. The two prior failures were the
extra_let_bindingsbreak described above.Conformance through this branch's CLI: 1172 passed, 2 failed.
expr2.2.1--string-conversionandexpr2.3.2--path-constructionboth pass — the two fixtures that pull in opposite directions, one wanting a value that has left path-space to stay/mnt/outand the other wanting a path to render in the host's format. The 2 failures are pre-existing range-normalization cases, unrelated to this change and blocked on a spec ruling.Upstream suites: model 5496, sessions 999.
Not verified
Windows. The companion sessions PR, OpenJobDescription/openjd-sessions-for-python#362, has had its Windows CI legs fail-fast cancelled on every round, so Windows has never been exercised by CI there. The behaviour was proven by simulation instead: with the host format forced to Windows in-process, a path-typed binding renders
\a\bwhile a value that has left path-space stays/mnt/out.The conformance runner concatenates a fixture's
outputblock with itsoutput_<os>block, so on a POSIX host anoutput_windowsblock is never asserted. A green POSIX conformance run proves nothing about the Windows expectations.Stack note. The first three commits (
af52126,a328431,ae5bdaa) are preceding local work on the RFC 0007/0008 test and comment surface, carried on this branch.1b90918is the change described above.