You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "mobuchowski (via GitHub)" <gi...@apache.org> on 2024/08/15 14:45:16 UTC
[I] `get_tree_view` can consume extreme amounts of memory. [airflow]
mobuchowski opened a new issue, #41505:
URL: https://github.com/apache/airflow/issues/41505
### Apache Airflow version
2.10.0rc1
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
`get_tree_view` in degenerated case can take a lot of memory.
For a DAG
```
with DAG("aaa_big_get_tree_view", schedule=None) as dag:
first_set = [LongEmptyOperator(task_id=f"hello_{i}_{'a' * 230}") for i in range(900)]
chain(*first_set)
last_task_in_first_set = first_set[-1]
chain(
last_task_in_first_set, [LongEmptyOperator(task_id=f"world_{i}_{'a' * 230}") for i in range(900)]
)
chain(
last_task_in_first_set, [LongEmptyOperator(task_id=f"this_{i}_{'a' * 230}") for i in range(900)]
)
chain(last_task_in_first_set, [LongEmptyOperator(task_id=f"is_{i}_{'a' * 230}") for i in range(900)])
chain(
last_task_in_first_set, [LongEmptyOperator(task_id=f"silly_{i}_{'a' * 230}") for i in range(900)]
)
chain(
last_task_in_first_set, [LongEmptyOperator(task_id=f"stuff_{i}_{'a' * 230}") for i in range(900)]
)
```
serializing it can take 2.7GB
```
root@a24bae3584cb:/opt/airflow# pytest --memray tests/providers/openlineage/utils/test_utils.py::test_get_dag_tree_large_dag
=========================================================================================================================================================================== test session starts ============================================================================================================================================================================
platform linux -- Python 3.12.5, pytest-8.3.2, pluggy-1.5.0 -- /usr/local/bin/python
cachedir: .pytest_cache
rootdir: /opt/airflow
configfile: pyproject.toml
plugins: memray-1.7.0, timeouts-1.2.1, icdiff-0.9, mock-3.14.0, rerunfailures-14.0, requests-mock-1.12.1, xdist-3.6.1, asyncio-0.23.8, anyio-4.4.0, instafail-0.5.0, cov-5.0.0, time-machine-2.15.0, custom-exit-code-0.3.0
asyncio: mode=Mode.STRICT
setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
collected 1 item
tests/providers/openlineage/utils/test_utils.py::test_get_dag_tree_large_dag PASSED [100%]
============================================================================================================================================================================== MEMRAY REPORT ===============================================================================================================================================================================
Allocation results for tests/providers/openlineage/utils/test_utils.py::test_get_dag_tree_large_dag at the high watermark
📦 Total memory allocated: 5.4GiB
📏 Total allocations: 23
📊 Histogram of allocation sizes: |▁▁█ |
🥇 Biggest allocating functions:
- _safe_get_dag_tree_view:/opt/airflow/airflow/providers/openlineage/utils/utils.py:446 -> 2.7GiB
- get_tree_view:/opt/airflow/airflow/models/dag.py:2445 -> 2.7GiB
- __setattr__:/opt/airflow/airflow/models/baseoperator.py:1191 -> 1.3MiB
- __setattr__:/opt/airflow/airflow/models/baseoperator.py:1191 -> 1.3MiB
- __setattr__:/opt/airflow/airflow/models/baseoperator.py:1191 -> 1.3MiB
=================================================================================================================================================================== Warning summary. Total: 3, Unique: 3 ===================================================================================================================================================================
airflow: total 1, unique 1
collect: total 1, unique 1
other: total 2, unique 2
collect: total 2, unique 2
Warnings saved into /opt/airflow/tests/warnings.txt file.
============================================================================================================================================================================ 1 passed in 8.60s =============================================================================================================================================================================
```
https://github.com/apache/airflow/pull/41494
### What you think should happen instead?
I think tree_view format should be changed to one that does not require extraordinary amount of whitespace in deeply nested cases.
Would be good to know in which cases it's being used though.
### How to reproduce
You can use above dag.
### Operating System
Docker/breeze on MacOS
### Versions of Apache Airflow Providers
_No response_
### Deployment
Other
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `get_tree_view` can consume extreme amounts of memory. [airflow]
Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on issue #41505:
URL: https://github.com/apache/airflow/issues/41505#issuecomment-2325453811
I've marked the whole "tree" concept - the `--tree` flag and the helper functions - as deprecated in 2.10 and removed them in 3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `get_tree_view` can consume extreme amounts of memory. [airflow]
Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham closed issue #41505: `get_tree_view` can consume extreme amounts of memory.
URL: https://github.com/apache/airflow/issues/41505
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `get_tree_view` can consume extreme amounts of memory. [airflow]
Posted by "mallman-experian (via GitHub)" <gi...@apache.org>.
mallman-experian commented on issue #41505:
URL: https://github.com/apache/airflow/issues/41505#issuecomment-3137536108
What's the replacement for the `DAG.tree_view()` method in Airflow 3?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `get_tree_view` can consume extreme amounts of memory. [airflow]
Posted by "mallman-experian (via GitHub)" <gi...@apache.org>.
mallman-experian commented on issue #41505:
URL: https://github.com/apache/airflow/issues/41505#issuecomment-3144792919
> > What's the replacement for the `DAG.tree_view()` method in Airflow 3?
>
> I don't think there is a direct replacement, because the whole philosphy of what gets shown has changed. The UI is rebuild from scratch. If you think you miss something there - describe the actual use case you want to do (after trying to adapt realisation of the use case to the new UI) and describe what you wanted to do and how you imagine your use case might be implemented in the new philosophy.
Hi @potiuk. I'm not referring to a UI element. I use the `tree_view` method to inspect a dag as part of its development. So I include a main guard like
```python
if __name__ == "__main__":
dag.tree_view()
```
in my dag file, and when I run `python my_new_dag.py` it shows a tree view of the dag. This does not require deployment of the dag, whereas viewing a dag in the UI (obviously) requires deploying it.
This is my use case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `get_tree_view` can consume extreme amounts of memory. [airflow]
Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on issue #41505:
URL: https://github.com/apache/airflow/issues/41505#issuecomment-2293050972
This is a better reproduction DAG:
```
with DAG("aaa_repro", schedule=None):
start = EmptyOperator(task_id="start")
a = [
start
>> EmptyOperator(task_id=f"a_1_{i}")
>> EmptyOperator(task_id=f"a_2_{i}")
>> EmptyOperator(task_id=f"a_3_{i}")
for i in range(200)
]
middle = EmptyOperator(task_id="middle")
b = [
middle
>> EmptyOperator(task_id=f"b_1_{i}")
>> EmptyOperator(task_id=f"b_2_{i}")
>> EmptyOperator(task_id=f"b_3_{i}")
for i in range(200)
]
middle2 = EmptyOperator(task_id="middle2")
c = [
middle2
>> EmptyOperator(task_id=f"c_1_{i}")
>> EmptyOperator(task_id=f"c_2_{i}")
>> EmptyOperator(task_id=f"c_3_{i}")
for i in range(200)
]
end = EmptyOperator(task_id="end")
start >> a >> middle >> b >> middle2 >> c >> end
```
It uses 5+GB and takes just under 8 minutes to generate on my machine.
And I believe it's less about the whitespace and more that we duplicate tasks in the output. For example, this DAG:
```
with DAG("aaa_runaway", schedule=None):
start = EmptyOperator(task_id="start")
x = [EmptyOperator(task_id=f"x_{i}") for i in range(3)]
middle = EmptyOperator(task_id="middle")
y = [EmptyOperator(task_id=f"y_{i}") for i in range(3)]
end = EmptyOperator(task_id="end")
start >> x >> middle >> y >> end
```
which results in this output:
```
<Task(EmptyOperator): start>
<Task(EmptyOperator): x_0>
<Task(EmptyOperator): middle>
<Task(EmptyOperator): y_0>
<Task(EmptyOperator): end>
<Task(EmptyOperator): y_1>
<Task(EmptyOperator): end>
<Task(EmptyOperator): y_2>
<Task(EmptyOperator): end>
<Task(EmptyOperator): x_1>
<Task(EmptyOperator): middle>
<Task(EmptyOperator): y_0>
<Task(EmptyOperator): end>
<Task(EmptyOperator): y_1>
<Task(EmptyOperator): end>
<Task(EmptyOperator): y_2>
<Task(EmptyOperator): end>
<Task(EmptyOperator): x_2>
<Task(EmptyOperator): middle>
<Task(EmptyOperator): y_0>
<Task(EmptyOperator): end>
<Task(EmptyOperator): y_1>
<Task(EmptyOperator): end>
<Task(EmptyOperator): y_2>
<Task(EmptyOperator): end>
```
Note how we get 9 `ends`. If you have a sufficiently complex DAG, this format becomes really problematic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `get_tree_view` can consume extreme amounts of memory. [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #41505:
URL: https://github.com/apache/airflow/issues/41505#issuecomment-3144301384
> What's the replacement for the `DAG.tree_view()` method in Airflow 3?
I don't think there is a direct replacement, because the whole philosphy of what gets shown has changed. The UI is rebuild from scratch. If you think you miss something there - describe the actual use case you want to do (after trying to adapt realisation of the use case to the new UI) and describe what you wanted to do and how you imagine your use case might be implemented in the new philosophy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `get_tree_view` can consume extreme amounts of memory. [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #41505:
URL: https://github.com/apache/airflow/issues/41505#issuecomment-3145274359
> Hi @potiuk. I'm not referring to a UI element. I use the tree_view method to inspect a dag as part of its development. So I include a main guard like
As mentioned above - it's been removed. I am not aware of any replacement. But if you look at the linked PR, the tree_view method was not overly complicated and you can write yourself a replacement method (just be aware that it's something that you might have to update in the future versions - this is not a part of Public Airflow Interface https://airflow.apache.org/docs/apache-airflow/stable/public-airflow-interface.html -> if you are trying to use `dag` model directly. There is Dag in task.sdk that is public interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `get_tree_view` can consume extreme amounts of memory. [airflow]
Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on issue #41505:
URL: https://github.com/apache/airflow/issues/41505#issuecomment-2293069069
It looks like `get_tree_view` is only used in a test and is relatively new from #37162. Probably should deprecate it in 2 so we can remove it in 3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org