You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "aridavis (via GitHub)" <gi...@apache.org> on 2025/05/03 11:41:21 UTC

[I] TriggerDagRunOperator target DAG stuck in queue if wait_for_completion is True [airflow]

aridavis opened a new issue, #50147:
URL: https://github.com/apache/airflow/issues/50147

   ### Apache Airflow version
   
   2.10.5
   
   ### If "Other Airflow 2 version" selected, which one?
   
   _No response_
   
   ### What happened?
   
   I have 2 DAG in different files.
   
   Parent 
   ```
   with DAG(
       dag_id="parent_dag_id",
       default_args=default_args,
       description="PARENT DAG",
       start_date=datetime(2025, 1, 1),
       schedule_interval="@hourly",
       catchup=False,
   ) as dag:
       trigger_child = TriggerDagRunOperator(
           task_id="trigger_child",
           trigger_dag_id="child_dag_id",
           reset_dag_run=True,
           wait_for_completion=True,
       )
   
     trigger_child
   ```
   
   Child
   ```
   def call_func():
     print("Hello")
   
   with DAG(
       dag_id="child_dag_id",
       default_args=default_args,
       description="CHILD DAG",
       start_date=datetime(2025, 1, 1),
       catchup=False,
   ) as dag:
     task = PythonOperator(
           task_id="call_function",
           python_callable=call_func,
       )
     
     task
   ```
   
   Everytime I trigger parent_dag, it will trigger the child_dag also, but the child_dag will stuck in queue
   
   ![Image](https://github.com/user-attachments/assets/faad3cf0-5775-4baa-94f0-9aa264477cd4)
   
   ### What you think should happen instead?
   
   _No response_
   
   ### How to reproduce
   
   Trigger parent_dag_id
   
   ### Operating System
   
   MacOS and Ubuntu 24.04
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] TriggerDagRunOperator target DAG stuck in queue if wait_for_completion is True [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #50147:
URL: https://github.com/apache/airflow/issues/50147#issuecomment-2928182926

   This issue has been closed because it has not received response from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] TriggerDagRunOperator target DAG stuck in queue if wait_for_completion is True [airflow]

Posted by "aridavis (via GitHub)" <gi...@apache.org>.
aridavis commented on issue #50147:
URL: https://github.com/apache/airflow/issues/50147#issuecomment-2848917089

   Hello @mohamedmeqlad99 I don't think it is working. There are only 2 DAG, but only 1 is running. I have followed your suggestion.
   
   ```
   with DAG(
       dag_id="child_dag_id",
       default_args=default_args,
       description="CHILD DAG",
       start_date=datetime(2025, 1, 1),
       catchup=False,
       max_active_runs=100,
       concurrency=100,
   ) as dag:
   ```
   
   
   and I have check the airflow.cfg, and the configuration has been set like yours. I forgot to mention, everytime the Child DAG is triggered (which is stuck in queue). The webserver can't get the scheduler heart beat
   
   ![Image](https://github.com/user-attachments/assets/f0a30e38-c1aa-4f65-96e5-d5f35562256e)
   
   And everytime I restart the scheduler, the Child DAG will be run but the parent DAG will fail because of the SIGTERM
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] TriggerDagRunOperator target DAG stuck in queue if wait_for_completion is True [airflow]

Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on issue #50147:
URL: https://github.com/apache/airflow/issues/50147#issuecomment-2869296260

   Is this only happening locally? This is the **expected** behavior for the SequentialExecutor.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] TriggerDagRunOperator target DAG stuck in queue if wait_for_completion is True [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #50147: TriggerDagRunOperator target DAG stuck in queue if wait_for_completion is True
URL: https://github.com/apache/airflow/issues/50147


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] TriggerDagRunOperator target DAG stuck in queue if wait_for_completion is True [airflow]

Posted by "mohamedmeqlad99 (via GitHub)" <gi...@apache.org>.
mohamedmeqlad99 commented on issue #50147:
URL: https://github.com/apache/airflow/issues/50147#issuecomment-2848720950

   did you try to increase concurrent DAG runs and task executions If limits are too low, the child_dag may get stuck in the queued state.
   In your DAG definition, set higher limits:
   
   ```python
   
   Copy
   Edit
   with DAG(
       dag_id="child_dag_id",
       max_active_runs=3,  # Increase if needed
       concurrency=16,      # Controls how many tasks can run in parallel
       ...
   ) as dag:
   
   ```
   In airflow.cfg, check and increase the following parameters:
   
   ```
   
   parallelism = 32              # Total number of task instances that can run in parallel
   dag_concurrency = 16          # Tasks per DAG that can run at once
   max_active_runs_per_dag = 16  # DAG runs per DAG allowed at the same time
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] TriggerDagRunOperator target DAG stuck in queue if wait_for_completion is True [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #50147:
URL: https://github.com/apache/airflow/issues/50147#issuecomment-2848584795

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] TriggerDagRunOperator target DAG stuck in queue if wait_for_completion is True [airflow]

Posted by "aridavis (via GitHub)" <gi...@apache.org>.
aridavis commented on issue #50147:
URL: https://github.com/apache/airflow/issues/50147#issuecomment-2869295240

   sorry for the late response, @jroachgolf84 @mchoccac . In my local, I'm using SequentialExecutor. In the Kubernetes I'm using CeleryExecutor (default by helm I think). 
   Both default pool is 128


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] TriggerDagRunOperator target DAG stuck in queue if wait_for_completion is True [airflow]

Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on issue #50147:
URL: https://github.com/apache/airflow/issues/50147#issuecomment-2849337396

   Which executor are you using? What is the size of your default pool?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] TriggerDagRunOperator target DAG stuck in queue if wait_for_completion is True [airflow]

Posted by "mchoccac (via GitHub)" <gi...@apache.org>.
mchoccac commented on issue #50147:
URL: https://github.com/apache/airflow/issues/50147#issuecomment-2868239325

   Check the type of executor you are using or use the Docker image.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] TriggerDagRunOperator target DAG stuck in queue if wait_for_completion is True [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #50147:
URL: https://github.com/apache/airflow/issues/50147#issuecomment-2908190770

   This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org