Clarify property-testing RFC contracts (10.1.2–10.3.4) - #665
Draft
leynos wants to merge 3 commits into
Draft
Conversation
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueWarning Your free Security trial is over. An organization admin can upgrade to Advanced for continuous pull request security review or dismiss this notice. Comment |
Contributor
Reviewer's GuideThis documentation-only PR introduces and indexes RFC 0012, specifying deterministic action/environment projections, bounded seeded property generation, replayable regressions, metamorphic relations, and coverage linting, and adds independently measurable Phase 10 roadmap tasks for delivering and adopting the proposal. Sequence diagram for deterministic property-test replaysequenceDiagram
participant Runner as Test runner
participant Regression as Regression store
participant Generator as Bounded generator
participant Pipeline as Plan-mode pipeline
participant Report as Failure reporter
Runner->>Regression: Load persisted regression tuple
alt Regression tuple exists
Regression-->>Runner: Return seed and tuple inputs
else No regression tuple
Runner->>Generator: Expand or sample domains with seed
Generator-->>Runner: Return drawn tuple
end
Runner->>Pipeline: Run manifest with tuple
Pipeline-->>Runner: Return canonical result.actions
Runner->>Report: Evaluate assertions and redact environment data
Report-->>Regression: Persist failing seed and tuple
Flow diagram for RFC 0012 roadmap deliveryflowchart TD
Shell[10.1.1 Pin recipe shell] --> Actions[10.1.2 Implement result.actions]
Actions --> Quantified[10.1.3 Implement quantified assertions]
Quantified --> Parse[10.2.1 Parse forall and domains]
Parse --> Replay[10.2.2 Deterministic expansion and replay]
Replay --> Mutations[10.3.1 Implement mutations]
Mutations --> Coverage[10.3.2 Add coverage lint]
Coverage --> Adoption[10.3.3 Dogfood example manifests]
Adoption --> Documentation[10.3.4 Document dialect and measurements]
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Draft RFC 0012, extending the RFC 0007 Netsukefile testing framework with a lightweight property-testing capability scoped to the generated build script — its command invocations and environment constructions: - a pinned-shell `result.actions` view exposing each build edge's tokenized argv and constructed environment, closing the observability gap the technical design flags for raw Ninja text; - `for_all_actions` and `for_all_targets` quantified assertions evaluated by the existing MiniJinja engine; - a bounded, declarative `forall` generation block with deterministic expansion, seeded sampling, `--seed` replay, and persisted regressions; - a closed `mutations` vocabulary of built-in metamorphic relations (permute, insert-irrelevant, rename, remove-input) mirroring the mutation strategy proven in `tests/workflow_contracts/`; - an advisory coverage completeness lint following the `tests/makefile_test_target/rustdocflags.rs` convention. Add roadmap phase 10 with steps 10.1-10.3 sequencing the delivery after the phase 7 result views, and list the RFC in `docs/contents.md`.
Define deterministic action ordering, secret-safe property reports, and consistent seed replay. Give the Phase 10 roadmap atomic scope and observable completion criteria for the planned dialect.
leynos
force-pushed
the
property-test-rfc-fixes
branch
from
September 2, 2026 20:01
80e52d5 to
c2496b8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This branch proposes RFC 0012’s bounded property-testing dialect for generated
build scripts and resolves the verified documentation contracts required before
implementation. It makes action projections deterministic, protects environment
data in result views and regressions, aligns seeded replay, and makes Phase 10
work independently measurable.
Roadmap tasks: 10.1.2–10.3.4.
No ExecPlan is associated with this documentation-only proposal.
Review walkthrough
Validation
make fmt: completed without incidental tracked changes.make check-fmt: passed.make markdownlint: passed (34 tests; 0 Markdown errors).make nixie: passed.git diff --check: passed.Notes
References
Summary by Sourcery
Document the contracts and roadmap for bounded property testing of generated build scripts, with deterministic action projections, protected environment data, and replayable failures.
New Features:
Enhancements:
Documentation: