feat(provider-tck): emit a machine-readable conformance report

[aepfli]

Stacked on #1830 (feat/provider-tck). Part of open-feature/spec#424; the envelope this emits is defined by the schema in open-feature/spec#425.

Sibling of #1847 (flagd adoption), not stacked above or below it: this branch adds reporting without changing the adopter-facing API, so the two do not conflict and either can land first. Verified — git merge-tree reports zero collisions between them.

Also now carries the canonical-set guard. CanonicalScenarioGuard fails a run that is set up to execute less than the canonical set — a tag filter, a selector override, or a feature file shadowing a canonical one. It belongs here rather than on the base for the same reason the report does: both exist so a conformance claim can be checked rather than trusted, and the guard is what makes the report's scenario count mean something. It arrived via #1846, which is now closed; the extension point from that PR went to #1830 instead.

This has been reworked. The earlier revision defined its own per-scenario results format: a list of scenarios, each with a four-value outcome and the Examples row it came from. That was the wrong call, and the review that said so was right. A results format has to be maintained, versioned and reimplemented in four languages, and everything it carried is already specified by Cucumber Messages. The schema in #425 has been reshaped accordingly and so has this.

Set TCK_REPORT_DIR and each suite now writes two files:

File	What it is
<configuration>.json	the envelope — what was tested, and what the provider claims
<configuration>.ndjson	the results — a Cucumber Messages stream

The envelope's results.location names the stream and results.digest covers it. Unset means no report, and that is not an error.

Real output

Both flagd resolvers, on this branch, against the flagd testbed:

$ TCK_REPORT_DIR=./reports mvn -Ptck -pl providers/flagd test
...
Tests run: 58, Failures: 0, Errors: 0, Skipped: 2

$ ls reports/
flagd-in-process.json  flagd-in-process.ndjson  flagd-rpc.json  flagd-rpc.ndjson

{
  "schemaVersion" : "1",
  "provider" : {
    "name" : "flagd",
    "language" : "java",
    "configuration" : "flagd-rpc"
  },
  "sdk" : {
    "name" : "dev.openfeature:sdk",
    "version" : "1.22.1"
  },
  "tck" : {
    "implementation" : "java-sdk-contrib/tools/tck",
    "version" : "0.0.1",
    "specRevision" : "bda599f1db440aa8d395d1d3af7b9b3cc3103b98"
  },
  "backend" : {
    "description" : "Docker Compose stack docker-compose.yaml, service backend",
    "controlApi" : "http"
  },
  "declaration" : {
    "declared" : [ "@lifecycle", "@events", "@stale", "@configuration-change", "@object", "@unavailable" ]
  },
  "results" : {
    "format" : "cucumber-messages",
    "formatVersion" : "30.1.0",
    "location" : "flagd-rpc.ndjson",
    "digest" : "sha256:47c6e8297fd655200e07cdcc85f3314a1b22100dd7de7159d6e8716dd13699de"
  },
  "knownDeviations" : [ {
    "capability" : "@numeric-coercion",
    "issue" : "https://github.com/open-feature/flagd/issues/1996",
    "summary" : "The lossy half of the coercion rule is not enforced: evaluating float-flag (0.5) through the integer API returns 0 with no error code, rather than TYPE_MISMATCH with the code default, so the fractional part is discarded silently. Lossless coercion is permitted and is not the defect. Both resolvers behave identically, which places it in the shared provider layer rather than in either transport."
  } ]
}

Reading the outcomes out of the stream, which needs one thing understood — a scenario's outcome is the most severe result among its steps, hooks included, because testCaseFinished carries no status of its own:

$ jq -c 'select(.testStepFinished) | .testStepFinished
         | {c: .testCaseStartedId, s: .testStepResult.status}' reports/flagd-rpc.ndjson \
    | jq -s 'group_by(.c) | map({s: (map(.s) | if any(. == "FAILED") then "FAILED"
                                              elif any(. == "SKIPPED") then "SKIPPED"
                                              else "PASSED" end)})
             | group_by(.s) | map({(.[0].s): length}) | add'
{
  "PASSED": 28,
  "SKIPPED": 1
}

Identical for flagd-in-process. 29 pickles, 29 testCase, 29 testCaseStarted, 29 testCaseFinished, nothing started twice, nothing left unexecuted. The one skip in each is A float flag is not silently narrowed to an integer, and it is the aborted @Before hook that makes it a skip: its step result is SKIPPED and carries the gate's own message, Skipped: provider does not declare capability NUMERIC_COERCION (tag @numeric-coercion). That is the rule Appendix F cares about, and it is truthful in the stream.

What the stream carries, and what it replaced

Scenario Outline row identity. pickle.astNodeIds is [scenario id, table row id], and the second entry resolves in the gherkinDocument message to the Examples row the scenario was compiled from. For the eleven rows of Requesting the wrong type returns the code default in errors.feature:

$ jq -c 'select(.pickle) | .pickle
         | select(.name == "Requesting the wrong type returns the code default")
         | {id, row: .astNodeIds[1]}' reports/flagd-rpc.ndjson
{"id":"6c8debd2-...","row":"ab8b4a4b-..."}   # -> ["string-flag","Boolean","false"]
{"id":"a7c76b0a-...","row":"63d6d6c8-..."}   # -> ["string-flag","Integer","1"]
{"id":"fecd333d-...","row":"bbd7f5ee-..."}   # -> ["string-flag","Float","0.1"]
{"id":"1a999e43-...","row":"0f899f28-..."}   # -> ["wrong-flag","Boolean","false"]
{"id":"32a4239b-...","row":"f88ca988-..."}   # -> ["boolean-flag","String","fallback"]
{"id":"f5b60fb7-...","row":"0597f6a3-..."}   # -> ["boolean-flag","Integer","1"]
{"id":"1bdf7aa2-...","row":"adee07be-..."}   # -> ["boolean-flag","Float","0.1"]
{"id":"392a66f7-...","row":"3409ce6f-..."}   # -> ["integer-flag","Boolean","false"]
{"id":"c9efc3ab-...","row":"adba2cac-..."}   # -> ["integer-flag","String","fallback"]
{"id":"94c6c1a0-...","row":"48bbee15-..."}   # -> ["float-flag","Boolean","false"]
{"id":"6acf3b06-...","row":"ef0378ff-..."}   # -> ["float-flag","String","fallback"]

Eleven distinct row ids for eleven scenarios sharing one name. ScenarioExamples and the report's example field are deleted: they re-parsed the feature source Cucumber publishes on TestSourceRead and matched a pickle's reported line number, via TestCase.getLocation(), back against the Examples tables — an approximation, reverse-engineered from CucumberQuery.getLocationBy, of exactly the mechanism the format already provides. That is the clearest argument for adopting a standard format rather than defining one, so it is worth saying rather than quietly dropping.

Also deleted: the per-scenario list, the per-capability rollup, the four-value Outcome enum, and tck.assetsTree — the stream carries the source of every feature that executed, which is strictly better than a tree hash asserting which revision it came from. specRevision stays, because it identifies the two artifacts the stream does not carry, flags/canonical-flags.json and openapi/control-api.yaml.

Tags, including per-Examples-block tags, are on pickle.tags with the AST node each came from. Gherkin allows a tag on an individual Examples block, so two rows of one outline can differ in whether the capability gate stops them; the self-test fixture has exactly that shape and asserts that only the tagged row is skipped.

Cucumber's own formatter, at a path the run chooses

The stream is produced by io.cucumber.core.plugin.MessageFormatter — the same class the built-in message:<path> plugin instantiates — so the bytes are what --plugin message:... would have written. ConformanceReportPlugin registers it against the same publisher and writes only the envelope.

The built-in plugin is not used directly for one reason: a @ConfigurationParameter value is a compile-time constant, so cucumber.plugin=message:<path> cannot have a path derived from TCK_REPORT_DIR, and flagd's two suites in one module would write to the same file. Delegating gets the standard bytes without giving up per-suite naming or the zero-configuration adoption.

Two consequences worth naming. The stream is buffered in memory and written at the end, because the file name comes from the provider configuration, which the suite only reports once its runtime has started — after the first messages have been emitted. It is 330 KB here. And the envelope handler is registered after the formatter's, because Cucumber invokes handlers for one event type in registration order and the formatter closes its writer on the run-finished message; going second is what guarantees the digest covers a complete file.

What stays OpenFeature-specific, and why

The envelope is not a summary of the results. Every field in it answers a question no results format answers, because a Messages stream cannot say what it was a test of:

• provider — what the provider calls itself through its own metadata, not the suite name. The suite name reads well in a failure message (flagd-rpc), which makes it the configuration; one provider with two materially different modes produces two reports that are not interchangeable.
• sdk — read from the classpath rather than declared, because the TCK depends on an SDK version range so that adopting it can never force an upgrade. What a consumer actually ran against is only knowable at runtime.
• tck — which implementation asked the questions, and which revision of the artifacts.
• declaration — the capability set the provider claims. This is the load-bearing one, and it is an input to reading the results rather than a summary of them, which is why it cannot be derived from the stream. The stream says a scenario was skipped; only the declaration says whether that is because the provider declines the capability it needed. Given the declaration and a scenario's tags — both present — the reason for each skip follows, so it does not have to be transported per scenario, which is what let the whole per-scenario list go.

knownDeviations is the one thing neither the stream nor the declaration can express. Withholding a capability reads identically whether it describes a limitation or works around a bug, and the TCK cannot tell the two apart from the outside. So ProviderTckHarness.knownDeviations() lets the provider author say, and flagd says it: @numeric-coercion is withheld because flagd narrows a float to an integer with no error code, in both resolvers, which places the defect in the shared provider layer rather than in either transport.

Tracked, not merely named. The gap is flagd#1996, the ADR that settles what the rule is, so KnownDeviation.tracked(...) carries the link. Silence was the alternative and it is worse: a consumer would read flagd declining @numeric-coercion exactly as it reads a provider with no streaming transport declining @configuration-change, and one of those is a decision while the other is a bug. KnownDeviation.untracked(...) remains for a defect with nothing to point at yet.

Testing the property rather than the serialisation

ConformanceReportPluginTest no longer asserts over a report this code wrote. It runs a fixture suite through the real Cucumber engine on the JUnit Platform, with the real plugin registered, and reads the emitted stream back the way a consumer would. That change is the point: "a gated scenario is never reported as passed" is now a property of what Cucumber emits, and only a real run can demonstrate it. Asserting over hand-built messages would only have checked Cucumber's serialiser.

The fixture is shaped like the suite rather than minimal — a capability tag on the feature, one on a scenario, one on a single Examples block, and an outline whose rows share a name — because those are the shapes the properties depend on. Seventeen tests: every pickle executed exactly once, outcome counts, both gated scenarios SKIPPED and neither PASSED, the gate's reason present, Examples-block tags reaching the right row only, eleven distinct row ids resolving to the right cells, the executed source byte-identical to the file, the digest matching the stream, and the envelope carrying what the schema requires and nothing it forbids.

The capability gate moved to CapabilityGate.requireDeclared so that the gate producing the skip and the test proving the skip survives are looking at the same code. Inlined in the step definitions, the self-test could only have shown that some abort becomes a skip.

Verification

• mvn -pl tools/tck verify — green: 14 tests, 0 Checkstyle violations, PMD clean, SpotBugs BugInstance size is 0, spotless clean.
• mvn -Ptck -pl providers/flagd test with TCK_REPORT_DIR set — 58 scenarios, 0 failures, 2 skipped. Four files written, one pair per resolver, no collision.
• Both envelopes valid against the reshaped schema from chore(deps): update actions/cache digest to 704facf #425 with a Draft 2020-12 validator (python jsonschema 4.10.3).
• results.digest matches sha256sum of the corresponding .ndjson for both.
• Both streams valid against the published Cucumber Messages schema, jsonschema/messages.schema.json from cucumber/messages@main — 744 messages each, zero invalid.
• Stream accounting per resolver: 29 pickles / 29 testCase / 29 testCaseStarted / 29 testCaseFinished, no pickle executed twice or never, 28 PASSED + 1 SKIPPED, and the one scenario carrying an undeclared capability tag is the SKIPPED one.
• Both jq recipes in the README were run against the real output and produce what they claim.

Run on JDK 21 with Docker; the flagd suites need a Docker daemon, so they are not part of the module's own test run.

Three schema changes since this was opened

All three are in spec#425 and this branch emits against them.

• backend.controlApi is required. A provider with no backend still had its flag state manipulated somehow, and which of the two ways that was is the single most important thing a reader needs in order to know what the results are worth. The old shape made the whole backend block omissible, which made the one value most worth knowing — in-process, the case the enum exists for — the one that could never appear. It is now stated by the control rather than inferred: BackendControl.controlApi(), http from the HTTP control and in-process from the in-process one.
• A known deviation's summary is required, and its issue link is not. A deviation with no prose is a line of tags that tells a reader nothing; a deviation with no filed issue is an honest state to be in, and requiring one encourages filing a placeholder.
• The declaration no longer claims to explain every skip. It explains the skips that come from an undeclared capability. It does not explain a skip the implementation refuses — @large-integers here, because Java's integer accessor is a 32-bit Integer and no Java provider can be asked for 2^53 − 1 — which is a property of the SDK rather than of the provider, and a reader who conflated the two would read a language's limit as a provider's decision.

How this interacts with the rest of the stack

• feat(provider-tck): in-process control path and in-memory/multi-provider self-tests #1837 — the conflict has grown, not shrunk, and it is worth being precise: 4 files / 6 hunks / 124 conflicted lines before these changes, 6 files / 8 hunks / 163 after. Deleting a lot of code did not help because the deletions are in files feat(provider-tck): in-process control path and in-memory/multi-provider self-tests #1837 does not touch — ConformanceReport, ConformanceReportPlugin, ScenarioExamples, Outcome — while the additions landed in three of the four files it does. ProviderTckHarness grew from 21 to 41 conflicted lines (knownDeviations() sits next to capabilities(), which feat(provider-tck): in-process control path and in-memory/multi-provider self-tests #1837 also moves) and TckRuntime from 76 to 90 (recordRun). Two new files conflict: steps/ProviderSteps (one line, the gate call) and flagd's shared resolver base class (two lines). All of it is mechanical. pom.xml and README.md still merge cleanly despite both branches editing them. The controlApi() follow-up described before still applies: it should become a default method on BackendControl returning empty, overridden by HttpBackendControl with http and InProcessBackendControl with in-process.
• feat(provider-tck): source the spec artifacts from the open-feature/spec submodule #1838 — introduces the spec submodule. The one remaining pinned POM property should then be generated from git rev-parse HEAD in the same step that copies the artifacts. provider-tck.spec.assets-tree is gone, so there is one fewer thing to generate.

Since the last review: two changes to what the declaration says

@strict-numeric-typing is now @numeric-coercion, and the rule is corrected

The tag was named for a stricter rule than the specification wants. flagd is implementing an accepted numeric coercion ADR (open-feature/flagd#1996) whose rule is that coercion is permitted when it is lossless and must fail with TYPE_MISMATCH only when information would be lost: 10.0 requested as an integer succeeds, 0.5 does not. Appendix F said "does not coerce between integer and float", which forbids the case the ADR requires to work — the one scenario survives the difference only because it asks about 0.5, which does have a fractional part. flagd's own testbed is also gaining @numeric-coercion scenarios, so keeping the old name would have left the reference implementation and the specification disagreeing about what a rule is called.

The spec side is open-feature/spec@aa2ad24f on feat/provider-tck-appendix. This branch's vendored src/main/resources/{features,flags,openapi} are byte-identical to specification/assets/provider-tck at that revision modulo line endings, and provider-tck.spec.revision records it. That range also carried two unrelated changes now vendored here: the lifecycle readiness scenario's name, and the /start requirement that a 200 means the seeded flag state is already being served.

flagd's deviation moves from untracked to tracked against flagd#1996, and its summary now says which half of the rule is broken — the defect is the lossy case being silently accepted, not coercion as such.

No scenario was added at the time, and two gaps were recorded as open. One has since closed:

• The lossless case has no scenario. Closed. The canonical flag set gained integral-float-flag, and errors.feature now carries three @numeric-coercion scenarios — the lossy one, 10.0 requested as an integer, and 10 requested as a float. A provider declaring the tag must satisfy all three, which is what stops the shortcut of rejecting every float and calling it strictness. Two of the four OFREP adoptions turned out to be taking exactly that shortcut, so the two lossless scenarios earned their place immediately.

• Accessor width is modelled, and what remains open is narrower than it looks. The ADR
  distinguishes a 64-bit integer accessor from a 32-bit one, and flagd's testbed tags the 32-bit-only
  scenarios @int32-bounded. Appendix F draws the same boundary from the other side: every language's
  accessor can ask for 2^31 − 1, so that precision scenario is untagged and mandatory; only some
  can ask for 2^53 − 1, so that one carries @large-integers, which a provider on a 32-bit
  accessor leaves undeclared. Mandatory-versus-gated instead of two tags, and no tag needed for the
  common case.

  What is genuinely open is the negative half: nothing asserts what a 32-bit-accessor provider owes
  when asked for a value it cannot represent. The scenario simply skips, so a provider that silently
  truncates passes exactly like one that returns TYPE_MISMATCH. That is deliberate rather than
  overlooked — spec#430 has not settled what a
  provider owes a value that does not fit the requested accessor, and asserting it here would be this
  suite inventing a rule the specification does not have.

A reserved capability can no longer be declared

@caching is reserved: it exists in the vocabulary so every language's TCK spells the same property the same way, but no scenario carries the tag. @targeting was reserved too when this was written and stopped being so at spec 26362f85, once scenarios started carrying it — which is the shape a reserved tag is meant to have: a placeholder that either grows scenarios or is removed, never a claim a report can carry indefinitely. The examples below are from before that change and show both. Such a capability cannot gate anything — it produces no skip, so nothing in a run can confirm or contradict it — and the report schema in #425 now says it must not be declared and must not appear in declaration.declared.

It was appearing. Before, from this branch:

"declared": ["@lifecycle","@events","@stale","@configuration-change","@object","@unavailable","@targeting","@caching"]

After:

"declared": ["@lifecycle","@events","@stale","@configuration-change","@object","@unavailable"]

Nobody decided to claim the last two. the shared flagd base class said EnumSet.complementOf(EnumSet.of(NUMERIC_COERCION)), which reads as "everything except the one thing flagd cannot do" and in fact means "every other enum constant", collecting both reserved tags on the way past. Both published reports claimed capabilities nothing had examined — the vacuous conformance claim the capability vocabulary exists to prevent. ProviderTckHarness.capabilities() defaulted to EnumSet.allOf, which had the same defect for any adopter who never overrode it.

Capability now carries the reserved flag itself, so the list cannot drift from the rule, and offers the two forms that mean what complementOf looks like: declarable() and declarableExcept(...). The default becomes declarable().

Naming a reserved capability explicitly fails the run, rather than being dropped with a warning. The declaration is the one part of the report no result can check — everything else in it was observed, this is asserted by the provider author — and a report is read long after the log a warning would have gone to. Nothing is lost by refusing, because no scenario carries the tag, and the check runs before the Compose stack starts, so the mistake costs seconds rather than a suite. It is enforced in TckRunMetadata, the one place every path to a report passes through; the emitted list skips reserved capabilities as well, so "no reserved tag in a declaration" is a property of the code that writes the document and not only of a check upstream of it. Three tests cover it: the maximal declaration's emitted declared contains neither reserved tag, naming one throws with the tag in the message, and declarableExcept is contrasted with the complementOf it replaces.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}