You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "mobuchowski (via GitHub)" <gi...@apache.org> on 2025/01/20 13:14:10 UTC
[PR] limit amount of assets collected by hook lineage collector [airflow]
mobuchowski opened a new pull request, #45798:
URL: https://github.com/apache/airflow/pull/45798
In some extreme cases collection of lineage can cause memory issues. This limits the collected assets now, with potential to do more reasonable optimizations later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] limit amount of assets collected by hook lineage collector [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #45798:
URL: https://github.com/apache/airflow/pull/45798#discussion_r1922603611
##########
airflow/lineage/hook.py:
##########
@@ -159,6 +160,9 @@ def add_input_asset(
asset_extra: dict | None = None,
):
"""Add the input asset and its corresponding hook execution context to the collector."""
+ if len(self._inputs) >= 100:
+ self.log.debug("Maximum number of asset inputs exceeded. Skipping.")
Review Comment:
Are you sure it's just debug? I guess this will be rather surprising for someone to see limited output without indication of what's happened?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] limit amount of assets collected by hook lineage collector [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #45798:
URL: https://github.com/apache/airflow/pull/45798#issuecomment-2604640575
### Backport failed to create: v2-10-test. View the failure log <a href='https://github.com/apache/airflow/actions/runs/12887243233'> Run details </a>
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>❌</td>
<td>v2-10-test</td>
<td><a href="https://github.com/apache/airflow/commit/5942c98ea7d2aa4bc34b999ff863a5ba55cc05db"><img src='https://img.shields.io/badge/Commit-5942c98-red' alt='Commit Link'></a></td>
</tr>
</table>
You can attempt to backport this manually by running:
```bash
cherry_picker 5942c98 v2-10-test
```
This should apply the commit to the v2-10-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.
After you have resolved the conflicts, you can continue the backport process by running:
```bash
cherry_picker --continue
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] limit amount of assets collected by hook lineage collector [airflow]
Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on code in PR #45798:
URL: https://github.com/apache/airflow/pull/45798#discussion_r1922624860
##########
airflow/lineage/hook.py:
##########
@@ -159,6 +160,9 @@ def add_input_asset(
asset_extra: dict | None = None,
):
"""Add the input asset and its corresponding hook execution context to the collector."""
+ if len(self._inputs) >= 100:
+ self.log.debug("Maximum number of asset inputs exceeded. Skipping.")
Review Comment:
I wanted to avoid flood of log messages when someone causes it to add 100s of thousands of inputs.
Changed it to have additional log for the first time output is exceeded, with log level warning - this way there will be an indication of that happening.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] limit amount of assets collected by hook lineage collector [airflow]
Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski merged PR #45798:
URL: https://github.com/apache/airflow/pull/45798
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] limit amount of assets collected by hook lineage collector [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #45798:
URL: https://github.com/apache/airflow/pull/45798#discussion_r1922971222
##########
airflow/lineage/hook.py:
##########
@@ -159,6 +160,9 @@ def add_input_asset(
asset_extra: dict | None = None,
):
"""Add the input asset and its corresponding hook execution context to the collector."""
+ if len(self._inputs) >= 100:
Review Comment:
Should we extract this `100` as a variable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] limit amount of assets collected by hook lineage collector [airflow]
Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on code in PR #45798:
URL: https://github.com/apache/airflow/pull/45798#discussion_r1923497251
##########
airflow/lineage/hook.py:
##########
@@ -159,6 +160,9 @@ def add_input_asset(
asset_extra: dict | None = None,
):
"""Add the input asset and its corresponding hook execution context to the collector."""
+ if len(self._inputs) >= 100:
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org