You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "henry3260 (via GitHub)" <gi...@apache.org> on 2026/05/20 16:50:24 UTC
[PR] Prevent durable storage tests from leaking hook lineage [airflow]
henry3260 opened a new pull request, #67252:
URL: https://github.com/apache/airflow/pull/67252
Isolate the hook lineage collector used by common.ai durable storage tests, so `ObjectStoragePath `does not leak collected assets into later tests in the same pytest process.
closes: #67044
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
-->
- [ ] Yes (please specify the tool below)
<!--
Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
* Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent durable storage tests from leaking hook lineage [airflow]
Posted by "henry3260 (via GitHub)" <gi...@apache.org>.
henry3260 commented on code in PR #67252:
URL: https://github.com/apache/airflow/pull/67252#discussion_r3278992172
##########
providers/common/ai/tests/unit/common/ai/durable/test_storage.py:
##########
@@ -35,7 +35,8 @@ def tmp_cache_path(tmp_path):
@pytest.fixture
-def storage(tmp_cache_path):
+def storage(tmp_cache_path, hook_lineage_collector):
+ # ObjectStoragePath IO records hook lineage on compat Airflow versions; keep it isolated.
Review Comment:
> The same `ObjectStoragePath` pollution pattern exists in two other places in this provider, and they'll keep flaking the openlineage extractor test on Compat 3.0/3.1 even after this fix lands:
>
> * `providers/common/ai/tests/unit/common/ai/operators/test_agent.py::TestAgentOperatorDurable::test_execute_durable_wraps_model_and_cleans_up` patches `_get_base_path` to a real `file://` path (line 469) and lets `DurableStorage` write through it, so the hook lineage collector picks up the same `Dataset` entries that issue [Hook lineage collector pollution: ObjectStoragePath-using tests leak into openlineage extractor tests #67044](https://github.com/apache/airflow/issues/67044) describes.
> * `providers/common/ai/tests/unit/common/ai/utils/test_file_analysis.py` is heavier: most tests in `TestBuildFileAnalysisRequest`, `TestFileAnalysisHelpers`, and `TestFormatReaders` (e.g. `test_text_file_analysis`, `test_detect_file_format`, `test_render_parquet_uses_lazy_import`, ...) construct `ObjectStoragePath(str(path))` against real `tmp_path` files and read through them, which registers inputs on the singleton collector.
>
> Would you mind extending this PR to either (a) add `hook_lineage_collector` to the affected fixtures/tests there too, or (b) introduce an autouse fixture at the `providers/common/ai/tests/unit/common/ai/conftest.py` level that resets the collector for every test in this provider? Option (b) matches the "more durable fix" the issue calls out and would prevent the next `ObjectStoragePath`-using test from re-introducing the same leak.
good catch! applied option b
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent durable storage tests from leaking hook lineage [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil merged PR #67252:
URL: https://github.com/apache/airflow/pull/67252
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent durable storage tests from leaking hook lineage [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #67252:
URL: https://github.com/apache/airflow/pull/67252#discussion_r3277909699
##########
providers/common/ai/tests/unit/common/ai/durable/test_storage.py:
##########
@@ -35,7 +35,8 @@ def tmp_cache_path(tmp_path):
@pytest.fixture
-def storage(tmp_cache_path):
+def storage(tmp_cache_path, hook_lineage_collector):
+ # ObjectStoragePath IO records hook lineage on compat Airflow versions; keep it isolated.
Review Comment:
The same `ObjectStoragePath` pollution pattern exists in two other places in this provider, and they'll keep flaking the openlineage extractor test on Compat 3.0/3.1 even after this fix lands:
- `providers/common/ai/tests/unit/common/ai/operators/test_agent.py::TestAgentOperatorDurable::test_execute_durable_wraps_model_and_cleans_up` patches `_get_base_path` to a real `file://` path (line 469) and lets `DurableStorage` write through it, so the hook lineage collector picks up the same `Dataset` entries that issue #67044 describes.
- `providers/common/ai/tests/unit/common/ai/utils/test_file_analysis.py` is heavier: most tests in `TestBuildFileAnalysisRequest`, `TestFileAnalysisHelpers`, and `TestFormatReaders` (e.g. `test_text_file_analysis`, `test_detect_file_format`, `test_render_parquet_uses_lazy_import`, ...) construct `ObjectStoragePath(str(path))` against real `tmp_path` files and read through them, which registers inputs on the singleton collector.
Would you mind extending this PR to either (a) add `hook_lineage_collector` to the affected fixtures/tests there too, or (b) introduce an autouse fixture at the `providers/common/ai/tests/unit/common/ai/conftest.py` level that resets the collector for every test in this provider? Option (b) matches the "more durable fix" the issue calls out and would prevent the next `ObjectStoragePath`-using test from re-introducing the same leak.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org