fix(workflows): evaluate parenthesised expressions - #4417
fix(workflows): evaluate parenthesised expressions#4417NgoQuocViet2001 wants to merge 2 commits into
Conversation
The operator scans in _evaluate_simple_expression skip over bracketed text, so an operator inside a nested operand is never split on. Nothing then unwrapped a group spanning the whole expression: `(a or b) and c` split at the top-level `and`, evaluated `(a or b)` as a dot path, found no such key, and got None. The `or` was never evaluated and the expression read false. So adding parentheses to make precedence explicit — the usual reason to add them — silently inverted the result: `inputs.a or inputs.b and inputs.c` was true while `(inputs.a or inputs.b) and inputs.c` was false. A step gated on such a condition is skipped with nothing reported; the malformed-condition validators do not flag it, because the syntax is valid. Unwrap a group that spans the whole expression, quote-aware and only when the opening paren closes at the very end, so `(a) and (b)` and a literal paren inside a string are untouched.
There was a problem hiding this comment.
🟡 Changes recommended
Parenthesis handling must also be mirrored in _unresolvable_term() with remediation coverage.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds correct evaluation of fully parenthesized workflow expressions.
Changes:
- Unwraps complete parenthesized groups before evaluation.
- Adds grouping, nesting, filtering, and quote-awareness tests.
File summaries
| File | Review |
|---|---|
tests/test_workflows.py |
Adds regression coverage for grouped expressions and edge cases. |
src/specify_cli/workflows/expressions.py |
Implements parenthesis unwrapping. The remediation parser remains inconsistent with evaluator behavior for grouped bare conditions (moderate; 2 votes). |
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 1
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
mnriem
left a comment
There was a problem hiding this comment.
Please address Copilot feedback
Addressing review feedback. The evaluator now unwraps a parenthesised
group, but _unresolvable_term did not, so a grouped operand reached the
path check as literal text. A condition the evaluator resolves was
therefore reported unresolvable and format_condition_remediation
withheld the wrap correction from it:
inputs.a or inputs.b -> Wrap the expression: ...
(inputs.a or inputs.b) and inputs.c -> No correction is offered
because '(inputs.a or inputs.b)'
is not a name ...
Mirror the unwrap branch, and cover grouped bare conditions both ways:
a valid group keeps the correction, and an unresolvable name inside a
group still refuses it.
|
Addressed — the Copilot finding was correct, thanks.
The unwrap branch is mirrored now, placed to match where the evaluator does it. Covered both directions in
|
|
Thanks — parentheses silently inverting a result is exactly the kind of correctness bug worth fixing, and mirroring the unwrap in |
Two regression tests for the property this refactor is for, both of which a second copy of the grammar in the gate would break while every existing test stayed green: - widening _INDEXED_SEGMENT alone reaches the gate (the negative-index shape from github#4416) - when the evaluator stops treating something as a leaf, the gate stops checking it, with no gate edit (the grouped-operand shape from github#4417) Both were checked by reintroducing the drift: giving the gate its own segment regex again fails the first with the real message rather than an import error.
|
Green review, but this is part of the |
…4460) * refactor(workflows): let the evaluator report its own leaves (#4274) _unresolvable_term answered one question -- does every operand in this condition resolve to something? -- by walking the expression itself: filters, then or/and/not, then comparisons, then list literals, down to the leaves. That walk was a second implementation of the parsing in _evaluate_simple_expression, kept in step with it by hand. Two helpers existed only to restate rules the evaluator already had. _looks_numeric mirrored the float()-only-when-a-dot-is-present rule because a bare float() accepts 1e3 and the evaluator does not. _is_literal mirrored the matching-close-is-the-final-character string test because startswith/endswith accepts 'a' 'b' and the evaluator does not. Both docstrings said "mirror the evaluator exactly", which is the tell: when the two drift nothing breaks loudly, the gate just answers wrongly, and the wrong answer is a paste-ready correction that inverts a condition. Seven of the nine findings in #4230 were the same defect wearing different clothes -- the gate disagreeing with the evaluator about where the operands are. Each round fixed one shape. Nothing stopped a tenth. _evaluate_simple_expression has exactly one place where a substring stops being grammar and becomes a name to resolve: its final line, _resolve_dot_path. Literals return before it; operands, filter arguments and list elements all arrive there by construction. Record the leaf there, behind a ContextVar that is None outside a probe, and the gate applies namespace rules to that list instead of re-deriving it. It now contains no grammar at all. Two properties this rests on, both asserted rather than assumed: * or/and are not short-circuited -- both sides are evaluated and only then combined -- so a leaf is recorded whatever the other side is worth. If that ever changes the gate would go quietly blind, so there is a test for it. * A probe run can raise on its own placeholder values. The leaves seen before that point are real, so they are kept rather than discarded; discarding them would lose `bogus` in `inputs.tags | join(bogus)`, which an earlier round of #4230 had to add by hand. expressions.py is 109 lines lighter and 84 heavier. All 336 existing tests pass unchanged, including the 20 cases of test_operands_must_be_literals_or_known_paths that took eight rounds to get right. test_literal_test_mirrors_the_evaluator tested the mirror, so it becomes test_literal_handling_comes_from_the_evaluator and asserts the same knowledge about 1e3 and 'a' 'b' through the gate instead. Four mutations, each killed by the tests that should kill it -- removing the leaf report alone turns 38 red. ruff 0.15.0 clean. * refactor(workflows): let _resolve_dot_path define the indexed segment The gate no longer restates the operator grammar, but it still restated the shape of a path segment: _PATH_SEGMENT and an inline fullmatch both described the index form that _resolve_dot_path matches with its own regex. Three copies of one rule, kept in step by hand -- the same drift this refactor set out to remove, one layer down. Name the form once as _INDEXED_SEGMENT beside _resolve_dot_path and have the gate ask it. Behaviour is unchanged: the regex is copied verbatim. What changes is that widening indexing now reaches the gate for free. * test(workflows): pin that the gate reads the evaluator's definitions Two regression tests for the property this refactor is for, both of which a second copy of the grammar in the gate would break while every existing test stayed green: - widening _INDEXED_SEGMENT alone reaches the gate (the negative-index shape from #4416) - when the evaluator stops treating something as a leaf, the gate stops checking it, with no gate edit (the grouped-operand shape from #4417) Both were checked by reintroducing the drift: giving the gate its own segment regex again fails the first with the real message rather than an import error. * fix(workflows): keep collecting leaves after a probe error The refactor stopped the leaf walk at the first exception a probe value raised, so every leaf further along the chain was lost. That is the one thing the collection exists to report, and it was a step backwards from the hand-written walk this PR replaces: inputs.blob | from_json | contains(bogus) origin/main reports 'bogus' this PR before the fix MISSED this PR after the fix reports 'bogus' from_json receives the probe placeholder mapping and raises; the walk ended there and contains(bogus) was never reached. Carry on past a failing filter while the sink is armed. _apply_filter evaluates a filter argument before it can raise on the value, so the failing segment's own leaves are already recorded; a fresh placeholder goes into the next filter, matching what the probe namespace hands out. Scoped to the probe: the sink is armed only by _collect_leaves, and _evaluator_rejects runs its own probe without it, so a mis-wired filter is still rejected and a real evaluation still raises rather than quietly returning the unfiltered value.
Problem
Adding parentheses to a workflow expression silently inverts its result:
_find_top_leveldeliberately skips bracketed text so an operator inside a nested operand is not split on. Nothing then unwraps a group that spans the whole expression, so:(a or b) and csplits at the top-leveland(a or b)reaches the dot-path fallback_resolve_dot_pathlooks up a key literally named(a or b), finds nothing, returnsNoneThe
oris never evaluated and the expression reads false. Parentheses are added precisely to make precedence explicit, so this fires on the expressions an author was being careful with.Nothing catches it either: the syntax is valid, so
condition_has_malformed_expression_blockandcondition_is_never_evaluatedboth pass it. Anif:step gated on such a condition takes the wrong branch, and awhilestep never starts, with no diagnostic.Fix
Unwrap a group that spans the whole expression, before the operator scans run. The check is quote-aware and requires the opening paren to close at the very last character, so it does not touch:
(a) and (b)— the first group closes early, so the top-levelandstill splits(inputs.n) | default(9)— same reason; the filter still applies'a(b'and('(')— a paren inside a string literal is not a groupScope
One helper and one early return in
src/specify_cli/workflows/expressions.py. Unparenthesised expressions take exactly the path they did before.Test plan
pytest tests/test_workflows.py -k "parenthesised or indexing or Expressions"→ 54 passed.pytest tests/test_workflows.py→ 942 passed. The 20 failures are theTestWorkflowCliAlignmentsymlink cases, which fail identically on an unmodified checkout here (Windows, no symlink privilege).expressions.pyfails the new test withassert False is True.