test: Add 27 EXPR expression-language conformance fixtures by leongdl · Pull Request #160 · OpenJobDescription/openjd-specifications

leongdl · 2026-08-02T02:38:30Z

Summary

27 new EXPR fixtures, 1 strengthened, 3 renamed, from a coverage audit of RFC 0005 / the published Expression Language spec. Suite: EXPR 350 → 379 passing, FEATURE_BUNDLE_1 55 → 57, 0 failing.

Areas covered

Direct path property access on all four routes — direct {{Task.File.<n>.name}}, embedded with surrounding text, inside a function call, and Env.File direct. This is the exact shape of the validator bug fixed in openjd-rs PR #292; the only pre-existing fixture reached the property through a let binding, the one route that never failed. All four now green on both implementations.
unresolved[T] progressive evaluation (zero prior fixtures mentioned it): valid op on an unresolved symbol passes check; type error on an unresolved value rejected at check; and the failing-arm suppression case — x if <unresolved bool> else <type-error arm> must validate, catching eager over-rejection.
let scope must-rejects: step-template let referencing Session.WorkingDirectory / Task.Param.* / Task.File.*; environment-script let referencing Task.Param.* — every §3.6.2 must-reject cell was previously empty.
Scope negatives inside expression trees (not just bare format strings): in-function-call and in-let variants.
Whole-field null-drop: an EXPR-computed null drops an optional scalar field (FEATURE_BUNDLE_1 timeout — audited every @fmtstring annotation; it is the only optional scalar expression field), plus the non-null control.
Operator precedence and associativity (zero prior fixtures): -2**2 == -4, 2**3**2 == 512 (right-assoc), a + b * c, not/and/or binding — 11 pins in one fixture.
Literal-table rows: \xhh/\uhhhh/\Uhhhhhhhh/\N{name} escapes; uppercase 0X/0O/0B/R prefixes; 00 valid; 1e10 is float (discriminated via §1.3.4 pass-through); three misplaced-underscore rejects.
Cross-type errors: 1 < "a", not 1, path(...)[0] rejected.
IEEE floats: 0.1 + 0.2 == 0.30000000000000004 asserted (the spec's own worked example); 0.0/0.0 rejected; float("-inf") rejected.
Session.HasPathMappingRules true branch via the runner's pathMapping: key — previously only ever asserted false.
Boundaries: exactly-50 let bindings, 512-char identifier, leading underscore, forward-reference chain; int64-max accept twins for param default and supplied value; supplied 2^63 reject (verified green on BOTH implementations in a dual run).
Strengthened: expr2.1.6--and-or-value-returning gained or-sentinel null discrimination and the false and x case.
Renamed: three expr1.3.11--* fixtures → expr1.3.12--* (they test §1.3.12 range field extensions; §1.3.11 is Error Handling).

Companion PR

Five spec-correct fixtures that FAIL today (int64 param-default acceptance, the openjd-rs 2^62 false-reject, and others) are in the expected-failures PR (branch conformance-expr-lang-expected-failures).

leongdl · 2026-08-02T02:49:24Z

CI note: the two failing fixtures on the Rust leg (7.3--task-file-property-direct / -embedded) are expected against the RELEASED openjd-rs — they pin the Task.File property-access fix merged in openjd-rs PR #292, which has not shipped in a release yet. They pass against openjd-rs main. Separately, mainline CI is currently red for every PR due to two pre-existing broken fixtures (wrap-env-let-bindings-in-hooks, wrap-failed-enter-still-runs-wrap-exit) unrelated to this change.

leongdl · 2026-08-15T17:45:08Z

Quorum verdict: GOOD (5/5) — spec: §3.6/§3.6.1 (exactly 50 bindings, exactly 512-char identifier, [a-z_] start, forward chains — all mechanically verified). Nit: header references ../proposed/ (the 513 reject twin lives on PR #164's branch, not here) — note the dependency.

leongdl · 2026-08-15T17:45:11Z

Quorum verdict: GOOD (5/5) — spec: §3.6.2 table (EnvironmentScript.let excludes Task.Param.*). Nit: no accept twin exercises a valid EnvironmentScript.let, so an implementation lacking env-script let entirely also passes; consider one accept fixture.

leongdl · 2026-08-15T17:45:13Z

Quorum verdict: GOOD (5/5) — spec: §3.6.2 (Session.* out of StepTemplate.let scope even nested in an expression tree). len/string are valid calls, so rejection isolates the buried symbol. Coverage nit: marginal over the bare-reference twin, kept for tree-walk validators.

leongdl · 2026-08-15T17:45:15Z

Quorum verdict: GOOD (5/5) — spec: §3.6.2 (StepTemplate.let may reference only Param/RawParam/Job.Name/Step.Name/earlier bindings). Minimal single defect.

leongdl · 2026-08-15T17:45:17Z

Quorum verdict: GOOD (5/5) — spec: §3.6.2 (Task.File.* scoped to StepScript.let only). The "no Task.File support at all" wrong-reason risk is closed by this PR's own 7.3 Task.File accept fixtures.

leongdl · 2026-08-15T17:45:19Z

Quorum verdict: GOOD (5/5) — spec: §3.6.2 (Task.Param.* unavailable in StepTemplate.let). parameterSpace defines Frame correctly, isolating the scope violation. The four 3.6.2 negatives cover three distinct excluded-symbol families — mainline had zero.

leongdl · 2026-08-15T17:45:22Z

Quorum verdict: GOOD (5/5) — spec: §7.3.1 (Session.* not in hostRequirements scope), buried in a call. upper/string are valid, so rejection traces to the scoped symbol. Nits: filename says 7.3 while citing §7.3.1 (matches its sibling; repo mixes prefixes); marginal over the existing bare-reference negative.

leongdl · 2026-08-15T17:45:24Z

Quorum verdict: GOOD (5/5) — spec: EL §1.1.6 (no underscore adjacent to the decimal point). No alternate tokenization yields a valid parse; accept twin (valid underscores) on mainline.

leongdl · 2026-08-15T17:45:26Z

Quorum verdict: GOOD (5/5, one nit) — spec: EL §1.1.6 (no underscore adjacent to the exponent marker). Nit: 1_e10 may be rejected via the trailing-underscore or symbol-resolution path rather than the named rule — inherent to exit-code testing; also likely shares a lexer rule with the trailing-underscore sibling.

leongdl · 2026-08-15T17:45:28Z

Quorum verdict: GOOD (5/5) — spec: EL §1.1.6 (no trailing underscore). Invalid under every tokenization. The three underscore negatives map to three distinct spec clauses — not duplicates.

leongdl · 2026-08-15T17:45:30Z

Quorum verdict: GOOD-WITH-NITS — one logic check requested. Spec: EL §1.2.1 unresolved-condition rule (failing arm's error suppressed under unresolved[bool]). The assertion direction (must validate) can't over-reject, but one reviewer challenges the premise: §1.2.3's single-scalar-target coercion may make upper(Task.Param.Frame) succeed (int→string in call position), in which case the fixture never actually exercises arm-failure suppression. Please verify; if coercion rescues it, switch the failing arm to something unrescuable (e.g. a subscript on an int).

leongdl · 2026-08-15T17:45:32Z

Quorum verdict: GOOD (5/5) — spec: EL §1.2.1 + e432ebe asymmetry rule (reject only when no resolved value could succeed): bool+int has no overload and no bool→int coercion exists, so every concrete value fails too — validation-time rejection is correct. Wrong-reason risk closed by the two accept twins in this PR.

leongdl · 2026-08-15T17:45:34Z

Quorum verdict: GOOD (5/5) — spec: EL §1.2.1 / §7.4 (valid ops on unresolved values must type-check). The anti-over-rejection direction — exactly what the unresolved trio needed.

leongdl · 2026-08-15T17:45:37Z

Quorum verdict: GOOD (5/5) — R100 rename tracking the spec renumber (§1.3.11→§1.3.12, Task Parameter Range Field Extensions); no stale in-file refs. Same for the path/string siblings.

leongdl · 2026-08-15T17:45:39Z

Quorum verdict: GOOD (5/5, one nit) — spec: EL §1.3.4, the spec's own 0.0/0.0 example. Nit: a generic div-by-zero guard also rejects it without any NaN check — but such rejection is still conforming (inf-producing ops are errors too), so immaterial.

leongdl · 2026-08-15T17:45:41Z

Quorum verdict: GOOD (5/5) — spec: EL §2.1.4 (ordering allows only int/float and string/path cross-pairs). Literal operands, no scope confound; no coercion path exists.

leongdl · 2026-08-15T17:45:44Z

Quorum verdict: GOOD (5/5) — spec: EL §2.1.6 note ("not remains strictly boolean"; no truthiness). Verbatim rule, minimal fixture.

leongdl · 2026-08-15T17:45:46Z

Quorum verdict: GOOD (5/5) — spec: EL §2.1.7 / RFC 0005 (path supports no subscript; the coercion escape hatch was checked — subscript receivers get an unconstrained target so path→string can't rescue it). Accept twin on mainline (string indexing).

leongdl · 2026-08-15T17:45:48Z

Quorum verdict: GOOD (5/5, coverage nit) — spec: EL §1.3.4 (float('-inf') explicitly an error; distinct spelling from inf). Low marginal value over the existing inf/nan siblings, but completes the family.

leongdl · 2026-08-15T17:45:50Z

Quorum verdict: GOOD-WITH-NITS — spec: §7.3.1 Env.File scope + EL §2.3.1 path properties; deterministic given the pinned filename:. Nit (adversarial): DIRECT_STEM:envconf is a prefix of the wrong answer envconf.txt, so a stem→name bug passes that line. Add the :END terminator this PR's own 7.3--task-file-property-embedded already uses.

leongdl · 2026-08-15T17:45:52Z

Quorum verdict: GOOD (5/5) — spec: §7.3.1 Session.HasPathMappingRules is bool under EXPR; true branch. Previously indistinguishable from a constant false — real gap. Both assertions flip completely on the false branch.

leongdl · 2026-08-15T17:45:54Z

Quorum verdict: GOOD-WITH-NITS — spec: §7.3.1 + EL §2.3.1, whole-field route (the known validator-bug shape). Nit: STEM:config is a prefix of STEM:config.txt — terminate the STEM line so the stem→name confusion can't pass.

leongdl · 2026-08-15T17:45:57Z

Quorum verdict: GOOD (5/5) — spec: EL §1.3.2 embedded stringification + §2.3.1. All three assertions carry :END terminators — this is the pattern the two sibling fixtures should copy.

leongdl · 2026-08-15T17:45:59Z

Quorum verdict: GOOD-WITH-NITS — spec: EL §2.2.1/§2.2.4 (upper/len over path properties). Nits: UPPER:CONFIG is a prefix of UPPER:CONFIG.TXT, and LEN:10 is a prefix of LEN:10x — add terminators or assert LEN_EQ:{{ len(...) == 10 }}. Coverage: the direct/embedded/in-call trio is at diminishing returns; direct is the known-bug shape and clearly justified.

leongdl · 2026-08-15T17:46:01Z

Quorum verdict: GOOD (5/5, one comment fix) — spec: EL §1.1 grammar; every expected value traced through the BNF and every wrong-parse alternative produces a non-prefix-compatible value or a type error. First precedence pins in the suite. Fix the garbled header phrase: "2 ** 512's exponent tree" should read "2 ** (3 ** 2) = 2 ** 9 = 512 (right-assoc), not 64".

leongdl · 2026-08-15T17:46:03Z

Quorum verdict: GOOD (5/5) — spec: EL §1.1.5 escape table (\xhh, \uhhhh, \Uhhhhhhhh, \N{name}). Unprocessed escapes yield raw backslash text that never matches the decoded expectations — discriminates both ways.

leongdl · 2026-08-15T17:46:05Z

Quorum verdict: GOOD (5/5, minor nit) — spec: EL §1.1.6 (uppercase prefixes, 00 explicitly valid, 1e10 float pass-through consistent with the suite's established literal-preservation reading). Nit: ZEROZERO:0 is a prefix of a literal pass-through 00; acceptance is the primary claim so low stakes.

leongdl · 2026-08-15T17:46:08Z

Quorum verdict: GOOD (5/5) — spec: EL §1.3.4 (IEEE 754 doubles; the spec's own 0.1+0.2 example). Service-compat reviewer verified the formatting is safe for BOTH engines: Python repr and Rust {} both emit shortest-round-trip 0.30000000000000004, and BIG_LIT uses literal pass-through, avoiding the 1e+308-vs-expanded divergence.

leongdl · 2026-08-15T17:46:10Z

Quorum verdict: GOOD (5/5) — spec: EL §1.2.1 (2^63-1 accepted on the parameter-value axis; DEF_IS_MAX:true pins exact equality — float overflow would print 9.223372036854776e+18 and fail). Nit: header references ../proposed/ (twin lives on PR #164's branch).

leongdl · 2026-08-15T17:46:13Z

Quorum verdict: GOOD (5/5, standard .invalid caveat) — spec: EL §1.2.1 (supplied 2^63 rejected at job creation; no lazy path could succeed). On the mainline runner any non-zero exit passes, so the near-identical accept twin (int64-max, same supplied-parameter mechanism) is the guard — keep them together.

leongdl · 2026-08-15T17:46:15Z

Quorum verdict: GOOD (5/5) — modification strengthens: original AND_NULL couldn't distinguish null from empty string; new AND_NULL_IS_NULL:true pins actual null and AND_FALSE:false can't be satisfied by "" or "hello". No assertion weakened.

leongdl · 2026-08-15T17:46:17Z

Quorum verdict: GOOD-WITH-NITS — spec: §5 timeout @fmtstring under FB1 + EL §1.3.2; placement in FEATURE_BUNDLE_1/jobs is correct (both extensions declared; FB1 sibling precedent). Nits: taskFailure: {} alone would also pass if the implementation rejected expression-valued timeouts outright — the null-drops-field twin is what rules that out (the pair works jointly; keep together); the 1s-timeout/10s-sleep margin assumes cancellation actually terminates python before the sleep completes — safe with default SIGTERM, but the spec doesn't bound the grace period.

leongdl · 2026-08-15T17:46:19Z

Quorum verdict: GOOD (5/5) — spec: EL §1.3.2 (optional field evaluating to null is omitted) + §5 (onRun default is no timeout). Requires success plus DONE_NO_TIMEOUT, so null-coercion to 0/""/None fails either by rejection or by kill. First real timeout-semantics pair in the suite.

leongdl · 2026-08-15T17:46:22Z

Quorum review (5 independent agents). 30 new fixtures + 3 renames + 1 strengthened; per-fixture verdicts posted as file comments. Net: no NOT-GOODs — the strongest PR of the set. Highlights: operator-precedence pins (first in suite), the unresolved-value trio (new semantic area, properly paired), HasPathMappingRules true-branch, first real timeout-enforcement pair, IEEE 0.1+0.2 pin verified safe for both Python repr and Rust shortest-round-trip formatting. Recurring actionable nit: three fixtures assert unterminated substrings that are prefixes of the most plausible wrong answer (stem-vs-name) — copy the :END terminator pattern from 7.3--task-file-property-embedded. Two headers reference a ../proposed/ dir not on this branch (lives in PRs #164/#166). One logic check requested on the unresolved-condition fixture.

leongdl · 2026-08-15T19:28:52Z

Quorum-review fixes applied and pushed (rebased onto mainline 3069673):

:END terminators added to 7.3--env-file-property-direct, 7.3--task-file-property-direct, and 7.3--task-file-property-in-call (stem-vs-name and LEN:10 prefix traps), matching the pattern 7.3--task-file-property-embedded already used.
Added 3.6.2--env-script-let-valid-scope.yaml — the accept twin of the env-script let-scope negatives (without it, an implementation lacking env-script let entirely passed every negative for the wrong reason).
expr1.2.1--unresolved-condition-failing-arm: the reviewer's coercion concern was checked empirically — upper(int) really has no matching signature (no int→string call-position coercion; verified against openjd-rs), so the failing arm is real; documented in the header.
Fixed the garbled operator-precedence comment (2 ** (3 ** 2) = 512).
Reworded two headers that referenced a ../proposed/ directory that lives on the expected-failures PR, not this branch.

Verified: 39 affected fixtures pass against openjd-rs upstream/main 8ae4fb9. PR is now 31 new + 1 strengthened + 3 renames.

…gaps 27 new fixtures, 1 strengthened, 3 renamed: direct path property access on all four routes (pins PR #292), unresolved[T] progressive evaluation incl. the failing-arm suppression case, let-scope must-rejects, in-expression scope negatives, whole-field null-drop (FEATURE_BUNDLE_1 timeout), operator precedence and associativity pins, string escapes and uppercase numeric prefixes, misplaced-underscore rejects, cross-type ordering/membership/not errors, IEEE float precision and NaN-arithmetic, HasPathMappingRules true branch, let boundary edges, int64-max accept twins, and the supplied-value 2^63 reject verified green on both implementations. Suite: 350 -> 379 passing. Spec-correct fixtures that FAIL today are split to conformance-expr-lang-expected-failures. Review: quorum-review fixes — :END terminators close stem-vs-name and LEN prefix traps in the Env.File/Task.File property fixtures; added 3.6.2--env-script-let-valid-scope.yaml as the accept twin of the env-script let-scope negatives; corrected the garbled operator-precedence comment (2 ** (3 ** 2) = 512); verified and documented that upper(int) has no matching signature (the unresolved-condition fixture's failing arm is real); reworded two headers that referenced a ../proposed/ directory which lives on the expected-failures PR, not this branch. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>

5--timeout-expr-value-honored.test.yaml was the single red test on windows-latest in run 33936907488. The action timeout was never at fault: the action was killed on time on both platforms (1.39s on Linux, 1.51s on Windows, against a 10-second sleep), and the runner's taskFailure check passed, so the run did exit non-zero. Only the forbidden-output check tripped. That check is a plain substring search over the combined stdout+stderr of the whole `openjd run`. openjd-sessions logs the command it is about to run at INFO, and builds that log line from different things on each platform (_subprocess.py, the cmd_line_for_logger branch): POSIX _runner_base.py writes the action into a temp .sh wrapper, so the logged line is shlex.join([<wrapper path>]). The command text appears only at DEBUG, which the CLI does not emit. Windows There is no wrapper, so the logged line is list2cmdline(args) -- the full argument list, including the python -c source body. So on Windows the echoed command line carried the fixture's own literal SHOULD_NOT_PRINT, and the check matched the echo rather than the program's output. On POSIX the same marker was invisible, which is why this passed on Linux and failed on Windows. Building the marker as 'SHOULD_NOT' + '_PRINT' keeps it out of the command line while still printing it verbatim if the action is ever allowed to finish, so the assertion is unchanged in substance. Verified with the CI versions (openjd-cli 0.7.7, openjd-model 0.11.9, openjd-sessions 0.12.1): baseline 1 passed, 0 failed mutant: timeout line removed 0 passed, 1 failed mutant: condition inverted to null 0 passed, 1 failed FEATURE_BUNDLE_1/jobs suite 12 passed, 0 failed Both mutants printed SHOULD_NOT_PRINT after the full 10 seconds and exited 0, so the fixture still fails when the timeout is not honored. Windows is verified by construction only -- list2cmdline of the new source does not contain the marker -- and needs a CI run to confirm. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>

Both headers now run in three paragraphs: the spec rule, what the fixture does, and what must happen. 23 comment lines down to 16. Cut the verbatim §1.3.2 quote, the restatement of what the runner's forbidden check does, and the POSIX .sh-wrapper mechanism. Kept the reason the marker is joined at run time: a reader who folds it back into a literal reintroduces the Windows failure fixed in 374c4d3. Comment-only change. FEATURE_BUNDLE_1/jobs is 12 passed, 0 failed. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>

leongdl requested a review from a team as a code owner August 2, 2026 02:38

leongdl commented Aug 15, 2026

View reviewed changes

leongdl mentioned this pull request Aug 15, 2026

test: [expected failures] EXPR fixtures needing implementation fixes #164

Draft

leongdl force-pushed the conformance-expr-lang-gaps branch from 8208897 to 2e7adba Compare August 15, 2026 19:26

leongdl force-pushed the conformance-expr-lang-gaps branch from 2e7adba to f6c1f8e Compare September 5, 2026 01:41

leongdl added 2 commits September 4, 2026 21:08

Uh oh!

Conversation

leongdl commented Aug 2, 2026

Summary

Areas covered

Companion PR

Uh oh!

leongdl commented Aug 2, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leongdl commented Aug 15, 2026

Uh oh!

leongdl commented Aug 15, 2026

Uh oh!

Reviewers