Skip to content

Clarify property-testing RFC contracts (10.1.2–10.3.4) - #665

Draft
leynos wants to merge 3 commits into
mainfrom
property-test-rfc-fixes
Draft

Clarify property-testing RFC contracts (10.1.2–10.3.4)#665
leynos wants to merge 3 commits into
mainfrom
property-test-rfc-fixes

Conversation

@leynos

@leynos leynos commented Sep 2, 2026

Copy link
Copy Markdown
Owner

Summary

This branch proposes RFC 0012’s bounded property-testing dialect for generated
build scripts and resolves the verified documentation contracts required before
implementation. It makes action projections deterministic, protects environment
data in result views and regressions, aligns seeded replay, and makes Phase 10
work independently measurable.

Roadmap tasks: 10.1.2–10.3.4.

No ExecPlan is associated with this documentation-only proposal.

Review walkthrough

  • Start with RFC 0012 for the canonical action ordering, environment redaction, versioned examples, and deterministic replay contract.
  • Then review Phase 10 for observable completion criteria, seed-plus-tuple replay, and the split between adoption and documentation work.
  • Finish with the RFC index entry to confirm the proposal remains discoverable.

Validation

  • make fmt: completed without incidental tracked changes.
  • make check-fmt: passed.
  • make markdownlint: passed (34 tests; 0 Markdown errors).
  • make nixie: passed.
  • git diff --check: passed.

Notes

  • Two independent verification passes confirmed all supplied RFC and roadmap findings were still valid before the minimal repair was made.
  • The PR is intentionally draft because RFC 0012 remains a proposal.

References

Summary by Sourcery

Document the contracts and roadmap for bounded property testing of generated build scripts, with deterministic action projections, protected environment data, and replayable failures.

New Features:

  • Add RFC 0012 defining a bounded, declarative property-testing dialect for generated build scripts, including structured action observations, quantified assertions, deterministic generation, replay, and metamorphic relations.

Enhancements:

  • Define canonical ordering and environment-redaction contracts for action projections and diagnostics.
  • Establish deterministic seed-plus-tuple replay, bounded expansion, coverage completeness checks, and measurable Phase 10 adoption criteria.

Documentation:

  • Index RFC 0012 in the documentation contents and add its complete proposal and design contracts.
  • Add Phase 10 to the roadmap with implementation, adoption, measurement, and documentation tasks spanning 10.1.2–10.3.4.

@coderabbitai

coderabbitai Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Warning

Your free Security trial is over. An organization admin can upgrade to Advanced for continuous pull request security review or dismiss this notice.


Comment @coderabbitai help to get the list of available commands.

@sourcery-ai

sourcery-ai Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Reviewer's Guide

This documentation-only PR introduces and indexes RFC 0012, specifying deterministic action/environment projections, bounded seeded property generation, replayable regressions, metamorphic relations, and coverage linting, and adds independently measurable Phase 10 roadmap tasks for delivering and adopting the proposal.

Sequence diagram for deterministic property-test replay

sequenceDiagram
    participant Runner as Test runner
    participant Regression as Regression store
    participant Generator as Bounded generator
    participant Pipeline as Plan-mode pipeline
    participant Report as Failure reporter

    Runner->>Regression: Load persisted regression tuple
    alt Regression tuple exists
        Regression-->>Runner: Return seed and tuple inputs
    else No regression tuple
        Runner->>Generator: Expand or sample domains with seed
        Generator-->>Runner: Return drawn tuple
    end
    Runner->>Pipeline: Run manifest with tuple
    Pipeline-->>Runner: Return canonical result.actions
    Runner->>Report: Evaluate assertions and redact environment data
    Report-->>Regression: Persist failing seed and tuple
Loading

Flow diagram for RFC 0012 roadmap delivery

flowchart TD
    Shell[10.1.1 Pin recipe shell] --> Actions[10.1.2 Implement result.actions]
    Actions --> Quantified[10.1.3 Implement quantified assertions]
    Quantified --> Parse[10.2.1 Parse forall and domains]
    Parse --> Replay[10.2.2 Deterministic expansion and replay]
    Replay --> Mutations[10.3.1 Implement mutations]
    Mutations --> Coverage[10.3.2 Add coverage lint]
    Coverage --> Adoption[10.3.3 Dogfood example manifests]
    Adoption --> Documentation[10.3.4 Document dialect and measurements]
Loading

File-Level Changes

Change Details Files
Add and index RFC 0012 defining a bounded, declarative property-testing dialect for generated build scripts.
  • Specify the host-independent result.actions projection, canonical ordering, quantified assertions, and environment redaction.
  • Define version 1.1 forall domains, deterministic exhaustive-or-seeded sampling, tuple persistence, replay, and reduction.
  • Define closed metamorphic mutations and an advisory/strict coverage-completeness lint.
  • Document the proposal’s goals, requirements, compatibility, alternatives, open questions, and implementation sequencing.
docs/rfcs/0012-netsukefile-property-testing.md
docs/contents.md
Add Phase 10 roadmap work items and measurable completion criteria for adopting the property-testing dialect.
  • Introduce the phase hypothesis and split work into structured observation, bounded generation, and metamorphic/adoption tracks.
  • Add roadmap tasks 10.1.2–10.3.4 covering action views, quantified assertions, generation/replay, mutations, coverage lint, dogfooding, and documentation.
  • Define dependencies, scope boundaries, validation outcomes, and the seed-plus-tuple replay success criterion.
docs/roadmap.md

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

codescene-access[bot]

This comment was marked as outdated.

Draft RFC 0012, extending the RFC 0007 Netsukefile testing framework
with a lightweight property-testing capability scoped to the generated
build script — its command invocations and environment constructions:

- a pinned-shell `result.actions` view exposing each build edge's
  tokenized argv and constructed environment, closing the observability
  gap the technical design flags for raw Ninja text;
- `for_all_actions` and `for_all_targets` quantified assertions
  evaluated by the existing MiniJinja engine;
- a bounded, declarative `forall` generation block with deterministic
  expansion, seeded sampling, `--seed` replay, and persisted
  regressions;
- a closed `mutations` vocabulary of built-in metamorphic relations
  (permute, insert-irrelevant, rename, remove-input) mirroring the
  mutation strategy proven in `tests/workflow_contracts/`;
- an advisory coverage completeness lint following the
  `tests/makefile_test_target/rustdocflags.rs` convention.

Add roadmap phase 10 with steps 10.1-10.3 sequencing the delivery after
the phase 7 result views, and list the RFC in `docs/contents.md`.
Define deterministic action ordering, secret-safe property reports, and
consistent seed replay. Give the Phase 10 roadmap atomic scope and observable
completion criteria for the planned dialect.
@leynos
leynos force-pushed the property-test-rfc-fixes branch from 80e52d5 to c2496b8 Compare September 2, 2026 20:01
codescene-access[bot]

This comment was marked as outdated.

@codescene-access codescene-access Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No quality gates enabled for this code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant