You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Shubham Gupta <y2...@gmail.com> on 2019/12/24 05:10:08 UTC
Airflow DAG explodes with RecursionError when triggered
Hello maintainers
A programmatically generated DAG of mine is exploding (the bomb
text-picture) when triggered with following stack-trace
File "/home/admin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/airflow/models/__init__.py",
line 2392, in has_dag
return getattr(self, '_dag', None) is not None
RecursionError: maximum recursion depth exceeded while calling a Python object
At this stage, I start getting the following stacktrace at
/logs/scheduler/latest/<path/to/my/dag_script.py.log>
File "/home/admin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/airflow/jobs.py",
line 930, in _process_task_instances
if len(active_dag_runs) >= dag.max_active_runs:
TypeError: '>=' not supported between instances of 'int' and 'NoneType'
I've tried several things without success
- Ascertained that my Airflow LocalExecutor deployment is still intact
(older workflow's DAGs are still running fine)
- Killing and restarting Airflow's scheduler & webserver several times
- airflow initdb
- Checked airflow.cfg file for any discrepancies
------------------------------
I'm using
- Python 3.7.3 (installed via PyEnv <https://github.com/pyenv/pyenv>)
- Airflow 1.10.3 with LocalExecutor
- Linux ip-XXX-XX-XX-XX 4.9.0-8-amd64 #1 SMP Debian 4.9.130-2
(2018-10-27) x86_64 GNU/Linux
Here's the link <https://stackoverflow.com/q/59462957/3679900> to my
original (elaborated) question on StackOverflow
*Shubham Gupta*
Software Engineer
zomato
Re: Airflow DAG explodes with RecursionError when triggered
Posted by Daniel Standish <dp...@gmail.com>.
Did you check the max_active_runs param in your dag?
On Mon, Dec 23, 2019, 11:47 PM Shubham Gupta <y2...@gmail.com>
wrote:
> Hello maintainers
>
> A programmatically generated DAG of mine is exploding (the bomb
> text-picture) when triggered with following stack-trace
>
> File
> "/home/admin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/airflow/models/__init__.py",
> line 2392, in has_dag
> return getattr(self, '_dag', None) is not None
> RecursionError: maximum recursion depth exceeded while calling a Python
> object
>
> At this stage, I start getting the following stacktrace at
> /logs/scheduler/latest/<path/to/my/dag_script.py.log>
>
> File
> "/home/admin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/airflow/jobs.py",
> line 930, in _process_task_instances
> if len(active_dag_runs) >= dag.max_active_runs:
> TypeError: '>=' not supported between instances of 'int' and 'NoneType'
>
> I've tried several things without success
>
> - Ascertained that my Airflow LocalExecutor deployment is still intact
> (older workflow's DAGs are still running fine)
> - Killing and restarting Airflow's scheduler & webserver several times
> - airflow initdb
> - Checked airflow.cfg file for any discrepancies
>
> ------------------------------
>
> I'm using
>
> - Python 3.7.3 (installed via PyEnv <https://github.com/pyenv/pyenv>)
> - Airflow 1.10.3 with LocalExecutor
> - Linux ip-XXX-XX-XX-XX 4.9.0-8-amd64 #1 SMP Debian 4.9.130-2
> (2018-10-27) x86_64 GNU/Linux
>
> Here's the link <https://stackoverflow.com/q/59462957/3679900> to my
> original (elaborated) question on StackOverflow
>
> *Shubham Gupta*
> Software Engineer
> zomato
>