You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2026/05/16 18:54:29 UTC

[I] Hook lineage collector pollution: ObjectStoragePath-using tests leak into openlineage extractor tests [airflow]

potiuk opened a new issue, #67044:
URL: https://github.com/apache/airflow/issues/67044

   ### Apache Airflow version
   
   main
   
   ### What happened?
   
   `providers/openlineage/tests/unit/openlineage/extractors/test_base.py::test_default_extractor_uses_wrong_operatorlineage_class` fails on the `Compat 3.0.6:P3.10` and `Compat 3.1.8:P3.10` provider-distributions jobs, asserting on an `OperatorLineage` that contains leaked `Dataset` entries from `providers/common/ai/.../durable/test_storage.py`:
   
   ```
   FAILED providers/openlineage/tests/unit/openlineage/extractors/test_base.py::test_default_extractor_uses_wrong_operatorlineage_class
     Expected: OperatorLineage(inputs=[], outputs=[], run_facets={}, job_facets={})
     Got:      OperatorLineage(
       inputs=[Dataset(namespace='file', name='/tmp/pytest-of-root/pytest-0/test_save_and_load_roundtrips0/test_dag_my_task_run_1.json'), ...],
       outputs=[Dataset(namespace='file', name='/tmp/pytest-of-root/pytest-0/test_save_and_load_roundtrips0/test_dag_my_task_run_1.json'), ...],
       run_facets={}, job_facets={},
     )
   ```
   
   ### Root cause
   
   The hook lineage collector returned by `get_hook_lineage_collector()` is a **process-wide singleton**. The `DurableStorage` tests in `providers/common/ai/tests/unit/common/ai/durable/test_storage.py` exercise `ObjectStoragePath` reads/writes (via the `storage` fixture with `dag_id=\"test_dag\", task_id=\"my_task\", run_id=\"run_1\"`). On the Compat 3.0.x/3.1.x targets the underlying SDK ships a real `HookLineageCollector` (not the `NoOpCollector` used in current `main`), so those file operations register as input/output assets in the singleton.
   
   When `test_default_extractor_uses_wrong_operatorlineage_class` then calls `ExtractorManager().extract_metadata(...)`, the extractor returns an invalid lineage class, so `validate_task_metadata` yields `None` → empty `OperatorLineage()`. The manager then falls through to `get_hook_lineage()` (see `providers/openlineage/src/airflow/providers/openlineage/extractors/manager.py:130-135`), reads the polluted singleton, and merges those leaked datasets into the result. The assertion fails.
   
   This is invisible on `main` because:
   - `NoOpCollector` is returned by `get_hook_lineage_collector()` when no hook lineage reader plugin is installed (`task-sdk/src/airflow/sdk/lineage.py:340-347`), so locally / in non-Compat CI there's no state to leak.
   - Scheduled jobs on `main` only run `Compat 2.10.5:P3.9` and `Compat 3.0.0:P3.9`. The `common.ai` provider isn't installed there (see `PROVIDERS_COMPATIBILITY_TESTS_MATRIX` in `dev/breeze/src/airflow_breeze/global_constants.py`).
   - The `common.ai` durable storage path was added in #64199 (Add durable execution for `AgentOperator`), making this a recent regression on the Compat 3.0.6/3.1.8 matrix entries.
   
   ### What you think should happen instead?
   
   Either:
   1. The openlineage test should isolate itself from the global collector (done in the immediate-fix PR).
   2. Tests that exercise `ObjectStoragePath` (or other operations that register with the hook lineage collector) should reset the collector in teardown, so cross-test pollution isn't possible.
   
   The immediate fix is (1). (2) is the more durable fix and worth doing as a follow-up — any other extractor test that falls through to `get_hook_lineage()` is potentially affected by the same pattern.
   
   ### How to reproduce
   
   Trigger CI on any branch that touches a path picked up by the Compat 3.0.6/3.1.8 jobs (selective-checks decides). The failure is deterministic when both `common.ai` durable storage tests and `openlineage` extractor tests run in the same pytest session.
   
   ### Operating System
   
   Linux (CI containers)
   
   ### Versions of Apache Airflow Providers
   
   main
   
   ### Deployment
   
   Other
   
   ### Deployment details
   
   Not applicable — affects CI only.
   
   ### Anything else?
   
   Surfaced while rebasing #66825 (an unrelated FileTrigger test fix); blocking that PR's CI.
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   
   ---
   Drafted-by: Claude Code (Opus 4.7); reviewed by @potiuk before posting


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Hook lineage collector pollution: ObjectStoragePath-using tests leak into openlineage extractor tests [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.

kaxil closed issue #67044: Hook lineage collector pollution: ObjectStoragePath-using tests leak into openlineage extractor tests
URL: https://github.com/apache/airflow/issues/67044


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Hook lineage collector pollution: ObjectStoragePath-using tests leak into openlineage extractor tests [airflow]

Posted by "puneetdixit200 (via GitHub)" <gi...@apache.org>.

puneetdixit200 commented on issue #67044:
URL: https://github.com/apache/airflow/issues/67044#issuecomment-4467951303

   I’d like to work on this. I can reproduce the provider-distribution CI failure, isolate the leaked hook lineage collector state, and add regression coverage so ObjectStoragePath tests cannot affect OpenLineage extractor tests. Please assign this to me.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org