You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/21 20:51:39 UTC

[GitHub] [airflow] iainwo opened a new issue #17148: DAGs with many Nodes Cannot be Rendered due to Python Recursion Limit

iainwo opened a new issue #17148:
URL: https://github.com/apache/airflow/issues/17148


   # Context
   
   - Helm Chart
   - Airflow 2.0.0
   
   # Problem
   
   When the depth of a DAG tree is greater than the python recursion limit the webserver will fail to render the DAG profile page.
   One example of this, is when the DAG is a linked list.
   
   See [view.py](https://github.com/apache/airflow/blob/7842de0ff124dd6a2696ec82bf6423455164df5b/airflow/www/views.py#L2046).
   
   # Solution
   
   1. maybe theres a DAG design pattern that circumvents recursion limits and a linked list of task instances
   2. use an iterative tree search instead of recursive


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] bbovenzi closed issue #17148: DAGs with many Nodes Cannot be Rendered due to Python Recursion Limit

Posted by GitBox <gi...@apache.org>.
bbovenzi closed issue #17148:
URL: https://github.com/apache/airflow/issues/17148


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] bbovenzi commented on issue #17148: DAGs with many Nodes Cannot be Rendered due to Python Recursion Limit

Posted by GitBox <gi...@apache.org>.
bbovenzi commented on issue #17148:
URL: https://github.com/apache/airflow/issues/17148#issuecomment-937889743


   @eladkal Yep! I just tested Jed's example dag and had 0 problem in the new tree view.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] bbovenzi commented on issue #17148: DAGs with many Nodes Cannot be Rendered due to Python Recursion Limit

Posted by GitBox <gi...@apache.org>.
bbovenzi commented on issue #17148:
URL: https://github.com/apache/airflow/issues/17148#issuecomment-937889743


   @eladkal Yep! I just tested Jed's example dag and had 0 problem in the new tree view.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #17148: DAGs with many Nodes Cannot be Rendered due to Python Recursion Limit

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #17148:
URL: https://github.com/apache/airflow/issues/17148#issuecomment-936861828


   @bbovenzi is this fixed in the new tree view https://github.com/apache/airflow/pull/18675 ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jedcunningham commented on issue #17148: DAGs with many Nodes Cannot be Rendered due to Python Recursion Limit

Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #17148:
URL: https://github.com/apache/airflow/issues/17148#issuecomment-885058106


   I was able to reproduce this. This happens on the tree view with the following DAG:
   
   ```
   from airflow import DAG
   from airflow.operators.dummy import DummyOperator
   from airflow.utils.dates import days_ago
   
   with DAG(dag_id="deep", schedule_interval=None, start_date=days_ago(100)) as dag:
       prev = None
       for i in range(0, 500):
           t = DummyOperator(task_id=f"t{i}")
           if prev:
               prev >> t
           prev = t
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org