You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "kacpermuda (via GitHub)" <gi...@apache.org> on 2026/02/16 11:35:33 UTC
[PR] feat: Make Hook Level Lineage limits configurable [airflow]
kacpermuda opened a new pull request, #62010:
URL: https://github.com/apache/airflow/pull/62010
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull request.
Write a good git commit message following this guide: http://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the documentation.
Feel free to ping (in general) for the review if you do not see reaction for a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
Currently `MAX_COLLECTED_ASSETS` and `MAX_COLLECTED_EXTRA` are hardcoded limits for hook lineage collectors. This PR makes them configurable with Airflow configuration. Users that want more detailed lineage can increase those limits and users that do not want any hook lineage can disable it by setting those limits to 0.
---
##### Was generative AI tooling used to co-author this PR?
No
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
-->
<!--
Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
* Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment: `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Make Hook Level Lineage limits configurable [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #62010:
URL: https://github.com/apache/airflow/pull/62010#discussion_r2812414698
##########
airflow-core/src/airflow/config_templates/config.yml:
##########
@@ -1929,6 +1929,22 @@ lineage:
type: string
example: ~
default: ""
+ hll_max_collected_assets:
+ description: |
+ Maximum number of asset inputs or outputs that can be collected by a single hook lineage collector
+ instance. Input assets and output assets are counted separately.
+ version_added: 3.2.0
+ type: integer
+ example: ~
+ default: "100"
+ hll_max_collected_extra:
Review Comment:
Sounds good, naming is hard!
##########
airflow-core/src/airflow/config_templates/config.yml:
##########
@@ -1929,6 +1929,22 @@ lineage:
type: string
example: ~
default: ""
+ hll_max_collected_assets:
Review Comment:
Naming is hard!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Make Hook Level Lineage limits configurable [airflow]
Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on PR #62010:
URL: https://github.com/apache/airflow/pull/62010#issuecomment-3915149863
I'll merge this :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Make Hook Level Lineage limits configurable [airflow]
Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski merged PR #62010:
URL: https://github.com/apache/airflow/pull/62010
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Make Hook Level Lineage limits configurable [airflow]
Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on code in PR #62010:
URL: https://github.com/apache/airflow/pull/62010#discussion_r2812364976
##########
airflow-core/src/airflow/config_templates/config.yml:
##########
@@ -1929,6 +1929,22 @@ lineage:
type: string
example: ~
default: ""
+ hll_max_collected_assets:
Review Comment:
Yes, that's better. I had troubles coming up with good name here. I don't expect any other collectors than hook ones, but maybe `max_assets_per_hook_collector`? WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Make Hook Level Lineage limits configurable [airflow]
Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on code in PR #62010:
URL: https://github.com/apache/airflow/pull/62010#discussion_r2812298475
##########
task-sdk/tests/task_sdk/test_lineage.py:
##########
@@ -565,6 +606,34 @@ def test_extra_limit(self, collector):
assert len(collector.collected_assets.extra) == 200
+ def test_configurable_max_collected_extra(self):
+ """MAX_COLLECTED_EXTRA is read from [lineage] conf and limits extra collection."""
+ import importlib
+
+ import airflow.sdk.lineage as lineage_mod
+ from airflow.sdk.configuration import conf
+
+ real_getint = conf.getint
+ new_max = 4
+
+ with patch.object(
+ conf,
+ "getint",
+ side_effect=lambda s, k, fallback=None: (
+ new_max if k == "hll_max_collected_extra" else real_getint(s, k, fallback=fallback)
+ ),
+ ):
+ importlib.reload(lineage_mod)
+ assert new_max == lineage_mod.MAX_COLLECTED_EXTRA
+
+ collector = lineage_mod.HookLineageCollector()
+ ctx = MagicMock(spec=BaseHook)
+ for i in range(new_max * 2):
+ collector.add_extra(ctx, f"key_{i}", f"value_{i}")
+ assert len(collector.collected_assets.extra) == new_max
+
+ importlib.reload(lineage_mod)
Review Comment:
Thanks, adjusted the code and fixed the failing tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Make Hook Level Lineage limits configurable [airflow]
Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on code in PR #62010:
URL: https://github.com/apache/airflow/pull/62010#discussion_r2812388971
##########
airflow-core/src/airflow/config_templates/config.yml:
##########
@@ -1929,6 +1929,22 @@ lineage:
type: string
example: ~
default: ""
+ hll_max_collected_assets:
+ description: |
+ Maximum number of asset inputs or outputs that can be collected by a single hook lineage collector
+ instance. Input assets and output assets are counted separately.
+ version_added: 3.2.0
+ type: integer
+ example: ~
+ default: "100"
+ hll_max_collected_extra:
Review Comment:
Pushed `max_extras_per_collector` for now, it's clear as well I hope. Thanks !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Make Hook Level Lineage limits configurable [airflow]
Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on code in PR #62010:
URL: https://github.com/apache/airflow/pull/62010#discussion_r2812387481
##########
airflow-core/src/airflow/config_templates/config.yml:
##########
@@ -1929,6 +1929,22 @@ lineage:
type: string
example: ~
default: ""
+ hll_max_collected_assets:
Review Comment:
Pushed your proposed name for now, it's good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Make Hook Level Lineage limits configurable [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #62010:
URL: https://github.com/apache/airflow/pull/62010#discussion_r2812002809
##########
task-sdk/tests/task_sdk/test_lineage.py:
##########
@@ -565,6 +606,34 @@ def test_extra_limit(self, collector):
assert len(collector.collected_assets.extra) == 200
+ def test_configurable_max_collected_extra(self):
+ """MAX_COLLECTED_EXTRA is read from [lineage] conf and limits extra collection."""
+ import importlib
+
+ import airflow.sdk.lineage as lineage_mod
+ from airflow.sdk.configuration import conf
+
+ real_getint = conf.getint
+ new_max = 4
+
+ with patch.object(
+ conf,
+ "getint",
+ side_effect=lambda s, k, fallback=None: (
+ new_max if k == "hll_max_collected_extra" else real_getint(s, k, fallback=fallback)
+ ),
+ ):
+ importlib.reload(lineage_mod)
+ assert new_max == lineage_mod.MAX_COLLECTED_EXTRA
+
+ collector = lineage_mod.HookLineageCollector()
+ ctx = MagicMock(spec=BaseHook)
+ for i in range(new_max * 2):
+ collector.add_extra(ctx, f"key_{i}", f"value_{i}")
+ assert len(collector.collected_assets.extra) == new_max
+
+ importlib.reload(lineage_mod)
Review Comment:
Do you need to patch `conf.getint`? Can't you use `@conf_vars` instead and reload the module like yuo are doing for it to catch the new values?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Make Hook Level Lineage limits configurable [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #62010:
URL: https://github.com/apache/airflow/pull/62010#discussion_r2812320732
##########
airflow-core/src/airflow/config_templates/config.yml:
##########
@@ -1929,6 +1929,22 @@ lineage:
type: string
example: ~
default: ""
+ hll_max_collected_assets:
Review Comment:
Since its a lineage config anways, do you think: `max_assets_per_collector` is more self explanatory?
##########
airflow-core/src/airflow/config_templates/config.yml:
##########
@@ -1929,6 +1929,22 @@ lineage:
type: string
example: ~
default: ""
+ hll_max_collected_assets:
+ description: |
+ Maximum number of asset inputs or outputs that can be collected by a single hook lineage collector
+ instance. Input assets and output assets are counted separately.
+ version_added: 3.2.0
+ type: integer
+ example: ~
+ default: "100"
+ hll_max_collected_extra:
Review Comment:
On a similar vein: `max_extra_metadata_per_collector`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Make Hook Level Lineage limits configurable [airflow]
Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on PR #62010:
URL: https://github.com/apache/airflow/pull/62010#issuecomment-3914601049
Ready to merge @amoghrajesh ? Do you think we should backport it to 3.1 as well ? Seems like a small feature that can be useful, not sure how tough it is to backport.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org