test(ffe): enable Python agentless configuration

[pavlokhrebto]

Motivation

DataDog/dd-trace-py#19331 implements the Python agentless Feature Flagging configuration source (FFL-2699). This enables the shared configuration-source contract for Python, following the Node.js (#7355) and Java (#7300) activations.

Two Python parametric-controller gaps had to be fixed for the contract to be observable; both mirror what #7355 fixed for Node.js.

Changes

• Enable tests/parametric/test_ffe/test_configuration_sources.py for Python from v4.14.0-dev.
• Return the evaluation reason and errorCode from /ffe/evaluate. It used get_*_value(), which discards both, so the controller always reported reason=DEFAULT with no error code. The cold-state assertions require PROVIDER_NOT_READY/GENERAL/PARSE_ERROR or reason=ERROR; the library already returned PROVIDER_NOT_READY, but it never reached the test.
• Skip the module-level eager provider initialization when any Feature Flagging configuration variable is set. DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED=true caused the provider to be created at import, polling the CDN before /ffe/start and breaking the lazy-activation assertions.

Decisions

The eager initialization is gated, not removed. Tests that predate this contract still rely on it: test_dynamic_evaluation.py has 2 tests but 1 ffe_start call, and test_span_enrichment.py has 18 tests with 13. Gating on the new configuration variables keeps those paths byte-identical.

Only the parametric suite is activated. tests/ffe/test_agentless_configuration.py (end-to-end, weblog) stays missing_feature — it was not exercised here.

Validation

Run against dd-trace-py #19331 using binaries/python-load-from-local:

PYTEST_XDIST_AUTO_NUM_WORKERS=1 TEST_LIBRARY=python ./run.sh PARAMETRIC \
  tests/parametric/test_ffe/test_configuration_sources.py

29 passed in 130.99s (0:02:10)

Progression while fixing the controller: 21 passed / 6 failed → 27 passed → 29 passed. The suite was run sequentially; with parallel workers, two cases intermittently error with test-agent port-routing noise (Client sent an HTTP request to an HTTPS server), unrelated to the library.

Note the local run declared v4.12.0 in the manifest rather than v4.14.0-dev: with python-load-from-local, ddtrace.__version__ reports the pip-installed release (4.12.0) while PYTHONPATH supplies the branch under test, so the version gate cannot see the branch locally. CI installs the real artifact, so v4.14.0-dev is the correct declaration there.

Note for reviewers: this PR modifies utils/build/docker/python/parametric/apm_test_client/server.py, so it needs R&P team approval per the checklist below.

Workflow

1. ⚠️ Create your PR as draft ⚠️
2. Work on you PR until the CI passes
3. Mark it as ready for review
   • Test logic is modified? -> Get a review from RFC owner.
   • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

• Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
• A docker base image is modified?
  • the relevant build-XXX-image label is present
• A scenario is added, removed or renamed?
  • Get a review from R&P team

[github-actions]

CODEOWNERS have been resolved as:

manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
utils/build/docker/python/parametric/apm_test_client/server.py          @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ef1cb447c9

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Preserve eager provider startup for RC tests

When DD_FEATURE_FLAGS_CONFIGURATION_SOURCE=remote_config is present, this new guard leaves openfeature_client unset until /ffe/start, but several existing parametric FFE paths set Remote Config and call wait_for_rc_apply_state() before they call ffe_start() (for example test_dynamic_evaluation.py::test_ffe_flag_evaluation and test_configuration_sources.py::test_remote_config_positive_ignores_agentless_env). Those tests rely on the eager DataDogProvider() construction to subscribe to FFE_FLAGS and ACK the config; with this condition they never advertise/apply that product and the wait times out for Python versions newly enabled by the manifest.

Useful? React with 👍 / 👎.

[pavlokhrebto]

Checked this — it doesn't reproduce, because in dd-trace-py the FFE_FLAGS subscription isn't owned by the provider. ddtrace/internal/openfeature/product.py calls enable_featureflags_rc() at tracer startup whenever the resolved source is remote_config, independent of whether DataDogProvider() has been constructed. So skipping the eager construction still advertises the product and ACKs the config; /ffe/start only wires up the OpenFeature client.

You're right that the guard applies to test_dynamic_evaluation.py — its DEFAULT_ENVVARS do set DD_FEATURE_FLAGS_CONFIGURATION_SOURCE=remote_config. I ran the whole directory sequentially against dd-trace-py #19331 to confirm:

PYTEST_XDIST_AUTO_NUM_WORKERS=1 TEST_LIBRARY=python ./run.sh PARAMETRIC tests/parametric/test_ffe/
30 passed, 12 xfailed, 29 xpassed in 197.18s

Per file: test_dynamic_evaluation.py 24 passed / 0 failed, test_span_enrichment.py 6 passed + 12 xfailed (pre-existing manifest declarations), test_configuration_sources.py 29 xpassed. No wait_for_rc_apply_state() timeouts.

(The configuration-source cases show as xpassed only locally: with binaries/python-load-from-local, ddtrace.__version__ reports the pip-installed release rather than the branch, so the v4.14.0-dev gate can't match. They pass 29/29 when the manifest declares a version the local build satisfies.)

[datadog-prod-us1-5]

 

⚠️ Warnings

🚦 2 Pipeline jobs failed

Testing the test | System Tests (python, dev) / parametric / parametric (2)     

🧪 19 Tests failed

tests.parametric.test_ffe.test_configuration_sources.Test_Feature_Flag_Configuration_Source_Cold_Failure_And_Recovery.test_bad_to_good_cold_recovery[server-error-to-valid, parametric-python] from system_tests_suite  

mock FFE agentless backend status did not reach expected state: bad_to_good 500 to 200 recovery; status={'requests_total': 0, 'in_flight': 0, 'max_in_flight': 0, 'last_path': None, 'last_if_none_match': None, 'last_auth_present': False, 'last_status_code': None, 'status_codes': []}

tests.parametric.test_ffe.test_configuration_sources.Test_Feature_Flag_Configuration_Source_Cold_Failure_And_Recovery.test_bad_to_unchanged_cold_preserves_not_ready[server-error-to-unchanged, parametric-python] from system_tests_suite  

mock FFE agentless backend status did not reach expected state: bad_to_unchanged 500 to 304 cold sequence; status={'requests_total': 0, 'in_flight': 0, 'max_in_flight': 0, 'last_path': None, 'last_if_none_match': None, 'last_auth_present': False, 'last_status_code': None, 'status_codes': []}

View all failed tests

Testing the test | all-jobs-are-green     

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

🔄 Datadog auto-retried 1 job - 0 passed on retry

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: ef1cb44 | Docs | Datadog PR Page | Give us feedback!}