test(ffe): align invalid config metrics

[leoromanovsky]

Motivation

Per-flag invalid configurations now return the caller default without error metadata. Existing metrics tests still require ERROR and PARSE_ERROR, which blocks DataDog/dd-trace-py#19390.

Changes and Decisions

• Expect DEFAULT and no error.type for invalid-regex and invalid-variant per-flag configurations.
• Rename the test node IDs and update the manifest references and FFE documentation.
• Activate the corrected Python behavior from v4.14.0.
• Keep malformed full-payload parsing as ERROR and PARSE_ERROR.

[github-actions]

CODEOWNERS have been resolved as:

manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
tests/ffe/README.md                                                     @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/ffe/test_flag_eval_metrics.py                                     @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core

[datadog-official]

 

✨ Unblock PR with BitsAI

⚠️ Warnings

🚦 80 Pipeline jobs failed

Testing the test | System Tests (dotnet, dev) / End-to-end #1 / uds 1     

🔧 Fix in code. This looks caused by changes in this PR.

2 failed tests. AssertionError: Expected reason 'default' for invalid regex, got tags: ['env:system-tests', 'feature_flag.result.reason:error'].

🧪 2 Tests failed · 🎯 related to the job error above

🎯 tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Metric_Invalid_Regex_Default.test_ffe_eval_metric_invalid_regex_default[uds] from system_tests_suite  

AssertionError: Expected reason 'default' for invalid regex, got tags: ['env:system-tests', 'instrumentation_scope_version:n/a', 'service:weblog', 'version:1.0.0', 'error.type:parse_error', 'feature_flag.key:eval-invalid-regex-default-flag', 'feature_flag.result.reason:error', 'feature_flag.result.variant:n/a', 'instrumentation_scope:Datadog.FeatureFlags.OpenFeature']
assert 'error' == 'default'
  - default
  + error

self = <tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Metric_Invalid_Regex_Default object at 0x7fd3f5a60ef0>

    def test_ffe_eval_metric_invalid_regex_default(self):
        """Test that invalid regex produces a default result without an error."""
        assert self.r.status_code == 200, f"Flag evaluation request failed: {self.r.text}"
...

🎯 tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Metric_Invalid_Variant_Default.test_ffe_eval_metric_invalid_variant_default[uds] from system_tests_suite  

AssertionError: Expected reason 'default' for invalid variant, got tags: ['env:system-tests', 'instrumentation_scope_version:n/a', 'service:weblog', 'version:1.0.0', 'error.type:parse_error', 'feature_flag.key:eval-invalid-variant-default-flag', 'feature_flag.result.reason:error', 'feature_flag.result.variant:n/a', 'instrumentation_scope:Datadog.FeatureFlags.OpenFeature']
assert 'error' == 'default'
  - default
  + error

self = <tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Metric_Invalid_Variant_Default object at 0x7fd3f5a62b10>

    def test_ffe_eval_metric_invalid_variant_default(self):
        """Test that an invalid variant produces a default result without an error."""
        assert self.r.status_code == 200, f"Flag evaluation request failed: {self.r.text}"
...

Testing the test | System Tests (dotnet, prod) / End-to-end #1 / poc 1     

🔧 Fix in code. This looks caused by changes in this PR.

2 failed tests due to assertion errors: expected reason 'default' for invalid regex/variant, but got 'error' for both.

🧪 2 Tests failed · 🎯 related to the job error above

🎯 tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Metric_Invalid_Regex_Default.test_ffe_eval_metric_invalid_regex_default[poc] from system_tests_suite  

AssertionError: Expected reason 'default' for invalid regex, got tags: ['env:system-tests', 'instrumentation_scope_version:n/a', 'service:weblog', 'version:1.0.0', 'error.type:parse_error', 'feature_flag.key:eval-invalid-regex-default-flag', 'feature_flag.result.reason:error', 'feature_flag.result.variant:n/a', 'instrumentation_scope:Datadog.FeatureFlags.OpenFeature']
assert 'error' == 'default'
  - default
  + error

self = <tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Metric_Invalid_Regex_Default object at 0x7ff5446c3470>

    def test_ffe_eval_metric_invalid_regex_default(self):
        """Test that invalid regex produces a default result without an error."""
        assert self.r.status_code == 200, f"Flag evaluation request failed: {self.r.text}"
...

🎯 tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Metric_Invalid_Variant_Default.test_ffe_eval_metric_invalid_variant_default[poc] from system_tests_suite  

AssertionError: Expected reason 'default' for invalid variant, got tags: ['env:system-tests', 'instrumentation_scope_version:n/a', 'service:weblog', 'version:1.0.0', 'error.type:parse_error', 'feature_flag.key:eval-invalid-variant-default-flag', 'feature_flag.result.reason:error', 'feature_flag.result.variant:n/a', 'instrumentation_scope:Datadog.FeatureFlags.OpenFeature']
assert 'error' == 'default'
  - default
  + error

self = <tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Metric_Invalid_Variant_Default object at 0x7ff5446c35c0>

    def test_ffe_eval_metric_invalid_variant_default(self):
        """Test that an invalid variant produces a default result without an error."""
        assert self.r.status_code == 200, f"Flag evaluation request failed: {self.r.text}"
...

Testing the test | System Tests (golang, dev) / End-to-end #1 / chi 1     

🔧 Fix in code. This looks caused by changes in this PR.

1 failed test. AssertionError: Expected reason 'default' for invalid variant, got tags: ['feature_flag.result.reason:error', ...] at tests/ffe/test_flag_eval_metrics.py:823

🧪 1 Test failed · 🎯 related to the job error above

🎯 tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Metric_Invalid_Variant_Default.test_ffe_eval_metric_invalid_variant_default[chi] from system_tests_suite  

AssertionError: Expected reason 'default' for invalid variant, got tags: ['env:system-tests', 'service:weblog', 'version:1.0.0', 'instrumentation_scope:github.com/DataDog/dd-trace-go/openfeature', 'instrumentation_scope_version:n/a', 'feature_flag.result.reason:error', 'feature_flag.result.variant:n/a', 'error.type:parse_error', 'feature_flag.key:eval-invalid-variant-default-flag']
assert 'error' == 'default'
  - default
  + error

self = <tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Metric_Invalid_Variant_Default object at 0x7f3330bf13d0>

    def test_ffe_eval_metric_invalid_variant_default(self):
        """Test that an invalid variant produces a default result without an error."""
        assert self.r.status_code == 200, f"Flag evaluation request failed: {self.r.text}"
...

View all 80 failed jobs.

📋 Copy prompt for your agent

CI on my pull request is failing. Help me find and fix the root cause of each failing job below — they were flagged as caused by changes in this PR, so focus on the diff. For each job, explain the failure and propose a fix.

Branch: agent/ffe-invalid-config-default-metrics

Testing the test | System Tests (dotnet, dev) / End-to-end #1 / uds 1
Commit: 68bd2eb00253e7f4b87543a6fea492a58945cb53
Error (code / test):
2 failed tests. AssertionError: Expected reason 'default' for invalid regex, got tags: ['env:system-tests', 'feature_flag.result.reason:error'].
CI job: https://github.com/DataDog/system-tests/actions/runs/30771581178/job/91565998704

Testing the test | System Tests (dotnet, prod) / End-to-end #1 / poc 1
Commit: 68bd2eb00253e7f4b87543a6fea492a58945cb53
Error (code / test):
2 failed tests due to assertion errors: expected reason 'default' for invalid regex/variant, but got 'error' for both.
CI job: https://github.com/DataDog/system-tests/actions/runs/30771581178/job/91565998578

Testing the test | System Tests (golang, dev) / End-to-end #1 / chi 1
Commit: 68bd2eb00253e7f4b87543a6fea492a58945cb53
Error (code / test):
1 failed test. AssertionError: Expected reason 'default' for invalid variant, got tags: ['feature_flag.result.reason:error', ...] at tests/ffe/test_flag_eval_metrics.py:823
CI job: https://github.com/DataDog/system-tests/actions/runs/30771581178/job/91565998749

Plus 77 more failing jobs not shown here.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

🔄 Datadog auto-retried 4 jobs - 0 passed on retry

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 39e7030 | Docs | Datadog PR Page | Give us feedback!}