test(ofrep): run the provider conformance suite against flagd's OFREP API

providers/openfeature-provider-ofrep/README.md

@@ -23,6 +23,57 @@ api.set_provider(OFREPProvider())
 
 <!-- TODO: add configuration options -->
 
+## Provider conformance suite
+
+This provider runs the [OpenFeature Provider Conformance Suite][tck] against a flagd-testbed stack
+serving OFREP, in `tests/tck`. The suite owns the container stack: `tests/tck/conftest.py` declares
+a Compose file and the port the provider connects to, and nothing else.
+
+`tests/tck/docker-compose.yaml` is one definition of the backend for the whole repository, and the
+flagd adoption carries a byte-identical copy — each provider package publishes its own distribution
+and must not read the other's files, so the two are kept in step by `diff` rather than by sharing a
+path. Change one, copy it to the other.
+
+**It is excluded from the default build, and a maintainer runs it by hand before merging a change to
+it.**
+
+```
+poe test-tck        # needs Docker
+poe test            # everything else, which is what CI runs
+```
+
+The exclusion lives in `pyproject.toml`: `--ignore=tests/tck` on the two tasks `build.yml` reaches,
+with the reason in a comment above them. Why a conformance suite is not a required gate is
+[Appendix F, "Running the suite in CI"][appendix-f], and is not restated here.
+
+Two things that are this provider's rather than the policy's:
+
+- **Docker is not what decides it.** The flagd package's `tests/e2e` needs Docker too and does run in
+  the default build. What decides it is the run: **2 failed, 45 passed, 17 skipped, 1 xfailed** —
+  both failures are canonical flags that no released flagd-testbed seeds yet, and the `xfail` is
+  the one genuine provider gap, recorded as a `KnownDeviation` rather than hidden.
+  `tests/tck/conftest.py` and `tests/tck/test_ofrep.py` account for each one, so a
+  reviewer running the suite can tell a new failure from a known one.
+- **The default build still collects the suite** — `poe test` and `poe test-cov` end in
+  `pytest tests/tck --collect-only`, which imports every module and starts no container. An excluded
+  suite that has quietly stopped importing against the harness is worse than one that runs and
+  fails, and `mypy` here is configured over `src` alone, so nothing else would notice.
+
+**This suite races a known backend defect, and deliberately does not compensate for it.**
+flagd-testbed's `POST /start` returns before it serves the reseeded flag set, which the control API
+forbids, and a stateless provider has no initialisation to hide that window behind — so a run can
+report `FLAG_NOT_FOUND` for flags the configuration plainly defines. The defect is
+[flagd-testbed#394](https://github.com/open-feature/flagd-testbed/pull/394).
+
+This suite used to wrap the control in a `SettledControl` that polled until the flags were served.
+That is removed. Compensating here made this suite's results incomparable with every other adoption
+run against the same backend: it read a clean floor while the others bounced, and the difference was
+the wait rather than the provider. **Read a red run against the documented floor and repeat it before
+blaming the provider** — the race moves between scenarios, a real defect does not.
+
+[tck]: ../../tools/openfeature-tck/README.md
+[appendix-f]: https://github.com/open-feature/spec/blob/main/specification/appendix-f-provider-conformance.md
+
 ## License
 
 Apache 2.0 - See [LICENSE](./LICENSE) for more information.

providers/openfeature-provider-ofrep/pyproject.toml

@@ -29,12 +29,22 @@ Homepage = "https://github.com/open-feature/python-sdk-contrib"
 dev = [
   "coverage[toml]>=7.10.0,<8.0.0",
   "mypy>=1.18.0,<2.0.0",
+  # The OpenFeature conformance suite. Ships the feature files, the flag set,
+  # the control-API client and the Compose harness that owns the container
+  # stack, and registers its step definitions through a pytest11 entry point --
+  # so tests/tck declares a Compose file and nothing else. The `compose` extra
+  # is what pulls testcontainers in for the harness.
+  "openfeature-tck[compose]",
   "poethepoet>=0.37.0",
   "pytest>=9.0.0,<10.0.0",
+  "pytest-bdd>=8.1.0,<9.0.0",
   "requests-mock>=1.12.0,<2.0.0",
   "types-requests>=2.32.0,<3.0.0",
 ]
 
+[tool.uv.sources]
+openfeature-tck = { workspace = true }
+
 [tool.uv.build-backend]
 module-name = "openfeature"
 module-root = "src"
@@ -69,8 +79,51 @@ disallow_any_generics = false
 strict = true
 
 [tool.poe.tasks]
-test = "pytest tests"
-test-cov = "coverage run -m pytest tests"
+# `tests/tck` is excluded from the default build on purpose, and a maintainer
+# runs `poe test-tck` by hand before merging a change to it. Why a conformance
+# suite is not a required gate is Appendix F, "Running the suite in CI" --
+# linked from tools/openfeature-tck/README.md -- and is deliberately not
+# restated here.
+#
+# What is local to this package: a full run is 2 failed, 45 passed, 17 skipped,
+# 1 xfailed. Both failures are canonical flags flagd-testbed does not seed yet,
+# and the xfail is the one genuine provider gap; tests/tck/conftest.py accounts
+# for them. Docker is not what decides the exclusion -- the flagd package's
+# `tests/e2e` needs Docker too and does run.
+#
+# The suite is still *collected* on every default build, so it cannot quietly
+# stop importing against the harness while nobody runs it. `--ignore` would
+# otherwise leave nothing checking that, since mypy here is configured over
+# `src` alone.
+#
+# `ignore_fail = "return_non_zero"` because poe aborts a sequence at its first
+# failing subtask, which would put the collect step behind the default suite's
+# result. It is green in this package today, and a check that only runs while
+# everything else passes is not a check. Every subtask runs and a non-zero exit
+# still propagates. The flagd package carries the same pair, where the default
+# suite is red and the collect step was in fact never reached.
+test = { sequence = ["test-default", "test-tck-collect"], ignore_fail = "return_non_zero" }
+test-cov = { sequence = ["test-cov-default", "test-tck-collect"], ignore_fail = "return_non_zero" }
+test-default = "pytest tests --ignore=tests/tck"
+test-cov-default = "coverage run -m pytest tests --ignore=tests/tck"
+# The sync is part of the run rather than something a maintainer remembers,
+# because Appendix F requires the revision check to be in force **where the
+# scenarios execute** and not only in the TCK implementation's own tests. This
+# is a uv workspace, so `openfeature-tck` is installed editable and the
+# canonical assets this suite collects are the gitignored copies under
+# `tools/openfeature-tck/src`: a rebase that moves the submodule pin leaves them
+# from the previous revision, and a suite run from here would neither notice nor
+# say so. That is exactly how an adoption in another language published a full
+# pass against stale scenarios. `hatch_build_sync.py` checks the submodule out
+# at the pin, refuses to copy if it cannot, and needs nothing but the standard
+# library.
+#
+# Not on `test-tck-collect`, deliberately: that runs on every default build, it
+# executes no scenario, and it publishes no result, so the rule does not reach
+# it and the default build stays free of a submodule operation.
+sync-spec-assets = "python ../../tools/openfeature-tck/hatch_build_sync.py"
+test-tck = ["sync-spec-assets", {cmd = "pytest tests/tck"}]
+test-tck-collect = "pytest tests/tck --collect-only -q"
 cov-report = "coverage xml"
 cov = [
     "test-cov",

providers/openfeature-provider-ofrep/tests/tck/conftest.py

@@ -0,0 +1,120 @@
+"""Session fixtures for the OFREP conformance suite, and one recorded deviation.
+
+The container lifecycle belongs to the TCK -- see its README for what
+``tck_backend`` does with the declaration below, and Appendix F, "The control
+API", for why the stack is started once and never restarted. What is left here is
+the declaration, the one wrapper this provider needs around the control, and the
+xfail for the single scenario it cannot satisfy.
+
+A full run is ``2 failed, 45 passed, 17 skipped, 1 xfailed``. Both failures ask
+for ``large-integer-flag``, which no released flagd-testbed seeds --
+open-feature/flagd-testbed#392 adds it, along with the two other canonical flags
+the image is missing, and says what each catches. Neither carries a
+``KnownDeviation``: the gap is the backend's flag set, not the provider's.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import pytest
+
+from openfeature.contrib.tools.tck import BackendControl, ComposeBackend, RunningBackend
+
+OFREP_PORT = 8016
+"""flagd's OFREP HTTP port, and the one port this provider connects to.
+
+flagd's own default, which the testbed's launchpad does not override. The
+launchpad's control port is exposed by the harness and must not be listed here.
+"""
+
+
+@pytest.fixture(scope="session")
+def compose_backend() -> ComposeBackend:
+    """The stack under test, as the TCK's ``tck_backend`` fixture wants it.
+
+    The Compose file is the same one the flagd adoption uses -- one definition of
+    the backend, copied per package -- so it publishes flagd's two resolver ports
+    as well. Nothing here declares them, and a port nobody declares is neither
+    waited on nor looked up.
+
+    The path is absolute so that pytest run from the repository root works too;
+    a relative one resolves against the working directory.
+    """
+    return ComposeBackend(
+        compose_file=Path(__file__).parent / "docker-compose.yaml",
+        backend_ports=[OFREP_PORT],
+    )
+
+
+@pytest.fixture(scope="session")
+def ofrep_base_url(tck_backend: RunningBackend) -> str:
+    """The origin flagd serves OFREP on, resolved once the stack is up.
+
+    A fixture rather than a constant the suite module reads, because the mapped
+    host port does not exist until the stack has started. The provider appends
+    ``ofrep/v1/evaluate/flags/{key}`` itself (``ofrep/__init__.py:115-119``), so
+    this is the bare origin.
+    """
+    endpoint = tck_backend.endpoint
+    return f"http://{endpoint.host}:{endpoint.port(OFREP_PORT)}"
+
+
+@pytest.fixture(scope="session")
+def ofrep_control(tck_backend: RunningBackend) -> BackendControl:
+    """The control API client, used exactly as the harness provides it.
+
+    ``tck_backend.control`` is the TCK's own ``HttpControl``, already pointed at
+    the launchpad's mapped port and awaited ready. The launchpad registers no
+    ``/reset``, so every ``prepare_scenario`` takes the harness's documented
+    ``/start`` fallback and one 404 is logged per session.
+
+    This suite used to wrap it in a ``SettledControl`` that polled the OFREP
+    endpoint until the reseeded flags were actually served, because this
+    backend's ``/start`` returns before that is true and a stateless provider
+    has no initialisation to hide the window behind. That wrapper is gone.
+    A backend returning before it serves breaks the control API contract, and
+    compensating for it here made this suite's results incomparable with every
+    other adoption run against the same backend -- this one read a clean floor
+    while the others bounced, and the difference was the wait, not the provider.
+    The defect is open-feature/flagd-testbed#394 and belongs there.
+
+    So this suite now races the window like the others do. Read a red run
+    against the documented floor and repeat it before blaming the provider: the
+    race moves between scenarios, a real defect does not.
+    """
+    return tck_backend.control
+
+
+# ---------------------------------------------------------------------------
+# One known deviation, recorded rather than hidden.
+#
+# A conformance suite that quietly goes green on a scenario it ran and failed is
+# as bad as one that goes green on a scenario it skipped. So the single scenario
+# this provider cannot satisfy is marked xfail(strict=True), which keeps it in
+# the report with its reason attached and fails the suite the moment it starts
+# passing -- so the marker is removed when the bug is fixed rather than
+# lingering as a lie. Same mechanism, and same bug, as the TCK's own self-test
+# (tools/openfeature-tck/tests/conftest.py).
+
+_BOOL_AS_INT = (
+    "test_requesting_the_wrong_type_returns_the_code_default[boolean-flag-Integer-1]"
+)
+
+_REASON = (
+    "bool satisfies an Integer request. OFREP is an untyped protocol -- the "
+    "backend returns the JSON value with no knowledge of the requested type -- so "
+    "the whole type check is the provider's, at ofrep/__init__.py:244-256: "
+    "FlagType.INTEGER maps to `int` and the check is isinstance(value, int), which "
+    "bool is a subclass of in Python. boolean-flag requested as an Integer "
+    "therefore returns True with reason STATIC and no error code, where the "
+    "specification requires the code default and TYPE_MISMATCH. The Python SDK "
+    "client type-checks the same way, so fixing only one of the two is not enough. "
+    "See https://github.com/open-feature/python-sdk/issues/619"
+)
+
+
+def pytest_collection_modifyitems(items: list[pytest.Item]) -> None:
+    for item in items:
+        if item.name == _BOOL_AS_INT:
+            item.add_marker(pytest.mark.xfail(reason=_REASON, strict=True))

providers/openfeature-provider-ofrep/tests/tck/docker-compose.yaml

@@ -0,0 +1,20 @@
+# The backend every conformance suite in this repository runs against: the unmodified
+# flagd testbed image, serving flagd and its launchpad control API.
+#
+# Not flagd-testbed's own compose file, which adds an envoy sidecar for its
+# forbidden-endpoint scenarios, names its service `flagd`, and bind-mounts a flags
+# directory the launchpad writes into. None of that is needed to drive the launchpad.
+#
+# This file exists twice, once per provider package, and the two are byte-identical:
+#   providers/openfeature-provider-flagd/tests/tck/docker-compose.yaml
+#   providers/openfeature-provider-ofrep/tests/tck/docker-compose.yaml
+# Each package publishes its own distribution and must not read the other's files, so
+# `diff` the two paths after changing either -- that diff is what catches drift.
+services:
+  backend:
+    image: ghcr.io/open-feature/flagd-testbed:v3.10.1
+    ports:
+      - 8013   # flagd RPC evaluation (gRPC)
+      - 8015   # flagd in-process sync (gRPC)
+      - 8016   # flagd's OFREP HTTP API
+      - 8080   # launchpad control API