Skip to content

fix(workflows): evaluate parenthesised expressions - #4417

Open
NgoQuocViet2001 wants to merge 2 commits into
github:mainfrom
NgoQuocViet2001:fix-parenthesised-grouping
Open

fix(workflows): evaluate parenthesised expressions#4417
NgoQuocViet2001 wants to merge 2 commits into
github:mainfrom
NgoQuocViet2001:fix-parenthesised-grouping

Conversation

@NgoQuocViet2001

Copy link
Copy Markdown
Contributor

Problem

Adding parentheses to a workflow expression silently inverts its result:

ctx = StepContext(inputs={"a": True, "b": False, "c": True})

evaluate_expression("{{ inputs.a or inputs.b and inputs.c }}",     ctx)  # True
evaluate_expression("{{ (inputs.a or inputs.b) and inputs.c }}",   ctx)  # False  ← same logic
evaluate_expression("{{ (inputs.n) }}",                            ctx)  # None   ← not 5

_find_top_level deliberately skips bracketed text so an operator inside a nested operand is not split on. Nothing then unwraps a group that spans the whole expression, so:

  1. (a or b) and c splits at the top-level and
  2. the left side (a or b) reaches the dot-path fallback
  3. _resolve_dot_path looks up a key literally named (a or b), finds nothing, returns None

The or is never evaluated and the expression reads false. Parentheses are added precisely to make precedence explicit, so this fires on the expressions an author was being careful with.

Nothing catches it either: the syntax is valid, so condition_has_malformed_expression_block and condition_is_never_evaluated both pass it. An if: step gated on such a condition takes the wrong branch, and a while step never starts, with no diagnostic.

Fix

Unwrap a group that spans the whole expression, before the operator scans run. The check is quote-aware and requires the opening paren to close at the very last character, so it does not touch:

  • (a) and (b) — the first group closes early, so the top-level and still splits
  • (inputs.n) | default(9) — same reason; the filter still applies
  • 'a(b' and ('(') — a paren inside a string literal is not a group

Scope

One helper and one early return in src/specify_cli/workflows/expressions.py. Unparenthesised expressions take exactly the path they did before.

Test plan

  • Ran: pytest tests/test_workflows.py -k "parenthesised or indexing or Expressions" → 54 passed.
  • Ran: pytest tests/test_workflows.py → 942 passed. The 20 failures are the TestWorkflowCliAlignment symlink cases, which fail identically on an unmodified checkout here (Windows, no symlink privilege).
  • Checked: reverting only expressions.py fails the new test with assert False is True.

The operator scans in _evaluate_simple_expression skip over bracketed
text, so an operator inside a nested operand is never split on. Nothing
then unwrapped a group spanning the whole expression: `(a or b) and c`
split at the top-level `and`, evaluated `(a or b)` as a dot path, found
no such key, and got None. The `or` was never evaluated and the
expression read false.

So adding parentheses to make precedence explicit — the usual reason to
add them — silently inverted the result: `inputs.a or inputs.b and
inputs.c` was true while `(inputs.a or inputs.b) and inputs.c` was
false. A step gated on such a condition is skipped with nothing reported;
the malformed-condition validators do not flag it, because the syntax is
valid.

Unwrap a group that spans the whole expression, quote-aware and only when
the opening paren closes at the very end, so `(a) and (b)` and a literal
paren inside a string are untouched.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Parenthesis handling must also be mirrored in _unresolvable_term() with remediation coverage.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds correct evaluation of fully parenthesized workflow expressions.

Changes:

  • Unwraps complete parenthesized groups before evaluation.
  • Adds grouping, nesting, filtering, and quote-awareness tests.
File summaries
File Review
tests/test_workflows.py Adds regression coverage for grouped expressions and edge cases.
src/specify_cli/workflows/expressions.py Implements parenthesis unwrapping. The remediation parser remains inconsistent with evaluator behavior for grouped bare conditions (moderate; 2 votes).
Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 1
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/specify_cli/workflows/expressions.py

@mnriem mnriem left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address Copilot feedback

Addressing review feedback. The evaluator now unwraps a parenthesised
group, but _unresolvable_term did not, so a grouped operand reached the
path check as literal text. A condition the evaluator resolves was
therefore reported unresolvable and format_condition_remediation
withheld the wrap correction from it:

  inputs.a or inputs.b                 -> Wrap the expression: ...
  (inputs.a or inputs.b) and inputs.c  -> No correction is offered
                                          because '(inputs.a or inputs.b)'
                                          is not a name ...

Mirror the unwrap branch, and cover grouped bare conditions both ways:
a valid group keeps the correction, and an unresolvable name inside a
group still refuses it.
@NgoQuocViet2001

Copy link
Copy Markdown
Contributor Author

Addressed — the Copilot finding was correct, thanks.

_unresolvable_term() did not mirror the evaluator's unwrap, so a grouped operand reached the path check as literal text and the wrap correction was withheld from a condition that would have worked:

inputs.a or inputs.b                 -> Wrap the expression: "{{ ... }}".
(inputs.a or inputs.b) and inputs.c  -> No correction is offered because
                                        '(inputs.a or inputs.b)' is not a name
                                        the evaluator can resolve

The unwrap branch is mirrored now, placed to match where the evaluator does it. Covered both directions in test_condition_expression_block.py, since the unwrap must not become a blanket pass for anything parenthesised:

  • (inputs.a or inputs.b) and inputs.c, (inputs.a), ((inputs.a)), (inputs.a) and (inputs.c) → correction offered
  • (bogus or inputs.b) and inputs.c, (inputs.a or bogus) → still refused, naming bogus

pytest tests/unit/test_condition_expression_block.py → 341 passed; reverting only expressions.py fails the four new grouped cases.

@mnriem mnriem added author-needs-disclosure AI use, or the agent/model/settings behind it, not disclosed per CONTRIBUTING author-awaiting Waiting on author response triage-nice-to-have Verdict: evidence-backed fix or greenlit feature — land after review labels Sep 8, 2026
@mnriem

mnriem commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

Thanks — parentheses silently inverting a result is exactly the kind of correctness bug worth fixing, and mirroring the unwrap in _unresolvable_term() (with both-direction tests so it's not a blanket pass) is the right call. Two things before merge: (1) please add the AI-disclosure per CONTRIBUTING — the body has none; (2) the Copilot review predates your fix, so I'll re-request it to confirm it's resolved. Green + disclosure and this is good to go.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The fix is focused, well-tested, and has no unresolved issues.

Review details
  • Files reviewed: 3/3 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

ntdatt812 added a commit to ntdatt812/spec-kit that referenced this pull request Sep 9, 2026
Two regression tests for the property this refactor is for, both of which
a second copy of the grammar in the gate would break while every existing
test stayed green:

- widening _INDEXED_SEGMENT alone reaches the gate (the negative-index
  shape from github#4416)
- when the evaluator stops treating something as a leaf, the gate stops
  checking it, with no gate edit (the grouped-operand shape from github#4417)

Both were checked by reintroducing the drift: giving the gate its own
segment regex again fails the first with the real message rather than an
import error.
@mnriem
mnriem requested a balanced review from Copilot September 9, 2026 15:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The implementation is focused, consistent across evaluation and validation, and adequately tested.

Review details
  • Files reviewed: 3/3 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

@mnriem

mnriem commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

Green review, but this is part of the expressions.py cluster and isn't independently mergeable yet: (a) CI is stale (Sep 8, pre-current-main); (b) it overlaps #4416 in _unresolvable_term, so those two can't both merge as-is; and (c) per ntdatt812's experiment on #4460, once that refactor lands your _unresolvable_term hunk becomes unnecessary. Plan: land #4460 first, then rebase this to just the evaluator-side hunk (it'll shrink), re-run CI, add the disclosure — then merge. Holding on the cluster sequencing.

mnriem pushed a commit that referenced this pull request Sep 10, 2026
…4460)

* refactor(workflows): let the evaluator report its own leaves (#4274)

_unresolvable_term answered one question -- does every operand in this
condition resolve to something? -- by walking the expression itself:
filters, then or/and/not, then comparisons, then list literals, down to
the leaves. That walk was a second implementation of the parsing in
_evaluate_simple_expression, kept in step with it by hand.

Two helpers existed only to restate rules the evaluator already had.
_looks_numeric mirrored the float()-only-when-a-dot-is-present rule
because a bare float() accepts 1e3 and the evaluator does not.
_is_literal mirrored the matching-close-is-the-final-character string
test because startswith/endswith accepts 'a' 'b' and the evaluator does
not. Both docstrings said "mirror the evaluator exactly", which is the
tell: when the two drift nothing breaks loudly, the gate just answers
wrongly, and the wrong answer is a paste-ready correction that inverts a
condition.

Seven of the nine findings in #4230 were the same defect wearing
different clothes -- the gate disagreeing with the evaluator about where
the operands are. Each round fixed one shape. Nothing stopped a tenth.

_evaluate_simple_expression has exactly one place where a substring stops
being grammar and becomes a name to resolve: its final line,
_resolve_dot_path. Literals return before it; operands, filter arguments
and list elements all arrive there by construction. Record the leaf
there, behind a ContextVar that is None outside a probe, and the gate
applies namespace rules to that list instead of re-deriving it. It now
contains no grammar at all.

Two properties this rests on, both asserted rather than assumed:

  * or/and are not short-circuited -- both sides are evaluated and only
    then combined -- so a leaf is recorded whatever the other side is
    worth. If that ever changes the gate would go quietly blind, so
    there is a test for it.

  * A probe run can raise on its own placeholder values. The leaves seen
    before that point are real, so they are kept rather than discarded;
    discarding them would lose `bogus` in `inputs.tags | join(bogus)`,
    which an earlier round of #4230 had to add by hand.

expressions.py is 109 lines lighter and 84 heavier. All 336 existing
tests pass unchanged, including the 20 cases of
test_operands_must_be_literals_or_known_paths that took eight rounds to
get right. test_literal_test_mirrors_the_evaluator tested the mirror, so
it becomes test_literal_handling_comes_from_the_evaluator and asserts the
same knowledge about 1e3 and 'a' 'b' through the gate instead.

Four mutations, each killed by the tests that should kill it -- removing
the leaf report alone turns 38 red. ruff 0.15.0 clean.

* refactor(workflows): let _resolve_dot_path define the indexed segment

The gate no longer restates the operator grammar, but it still restated
the shape of a path segment: _PATH_SEGMENT and an inline fullmatch both
described the index form that _resolve_dot_path matches with its own
regex. Three copies of one rule, kept in step by hand -- the same drift
this refactor set out to remove, one layer down.

Name the form once as _INDEXED_SEGMENT beside _resolve_dot_path and have
the gate ask it. Behaviour is unchanged: the regex is copied verbatim.
What changes is that widening indexing now reaches the gate for free.

* test(workflows): pin that the gate reads the evaluator's definitions

Two regression tests for the property this refactor is for, both of which
a second copy of the grammar in the gate would break while every existing
test stayed green:

- widening _INDEXED_SEGMENT alone reaches the gate (the negative-index
  shape from #4416)
- when the evaluator stops treating something as a leaf, the gate stops
  checking it, with no gate edit (the grouped-operand shape from #4417)

Both were checked by reintroducing the drift: giving the gate its own
segment regex again fails the first with the real message rather than an
import error.

* fix(workflows): keep collecting leaves after a probe error

The refactor stopped the leaf walk at the first exception a probe value
raised, so every leaf further along the chain was lost. That is the one
thing the collection exists to report, and it was a step backwards from
the hand-written walk this PR replaces:

  inputs.blob | from_json | contains(bogus)
    origin/main            reports 'bogus'
    this PR before the fix MISSED
    this PR after the fix  reports 'bogus'

from_json receives the probe placeholder mapping and raises; the walk ended
there and contains(bogus) was never reached.

Carry on past a failing filter while the sink is armed. _apply_filter
evaluates a filter argument before it can raise on the value, so the failing
segment's own leaves are already recorded; a fresh placeholder goes into the
next filter, matching what the probe namespace hands out.

Scoped to the probe: the sink is armed only by _collect_leaves, and
_evaluator_rejects runs its own probe without it, so a mis-wired filter is
still rejected and a real evaluation still raises rather than quietly
returning the unfiltered value.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author-awaiting Waiting on author response author-needs-disclosure AI use, or the agent/model/settings behind it, not disclosed per CONTRIBUTING triage-nice-to-have Verdict: evidence-backed fix or greenlit feature — land after review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants