[GitHub] [airflow] jens-scheffler-bosch commented on issue #14338: subdag tasks don't get picked up when dag concurrency set to 1

[GitBox <gi...@apache.org>]

jens-scheffler-bosch commented on issue #14338:
URL: https://github.com/apache/airflow/issues/14338#issuecomment-1251456102

   I believe this is rather a technical limitation where documentation might need to be improved or we need to have a warning raised in logs. Each task that is being executed requires a worker slot. A call to a subdag also allocates one worker slot until the execution of the subdag is completed.
   
   I tried to re-produce the issue with the `example_subdag_operator.py` that is shipped with the example DAGs in Airflow (2.4.0) and was not able to re-produce when setting `concurrency=1` in the DAG. Alternatively also with the non-deprecated option `max_active_tasks=1`.
   
   My setup consisted out of LocalExecutor with a development setup of Airflow. What I was able to re-produce was forcing the Airflow scheduler to run `SequentialExecutor` which limits task processing to a single task. In this case (irrespective of setting of `concurrency`) only one task slot is provided as executor. In this case SubDagOperator can not be used because the run of the subdag allocates one task slot and there is no worker slot possible to process tasks from within the subdag.
   
   To be able to work on this issue, can you please:
   
   - Do a regression on a current Airflow release
   - If you can re-produce also try it with the `example_subdag_operator.py`
   - If you can re-produce in your environment using a current version, please describe:
       - Which Airflow-Version
       - Which Environment
       - Which Executor is used
       - A copy of the DAG (or a neutralized DAG) which can be used to re-produce in an external environment


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org