You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "vashisht33 (via GitHub)" <gi...@apache.org> on 2024/05/29 08:56:22 UTC

[I] wait_for_completion not working when set to True while triggering child dag from parent dag [airflow]

vashisht33 opened a new issue, #39910:
URL: https://github.com/apache/airflow/issues/39910

   ### Apache Airflow version
   
   2.9.1
   
   ### If "Other Airflow 2 version" selected, which one?
   
   _No response_
   
   ### What happened?
   
   wait_for_completion not working when set to True while triggering child dag from parent dag.Below i have shared two files,one is parent_dag and other is child_dag.All i am trying to do is to trigger child dag from parent dag and wait for completion of child dag before i proceed further in parent_dag(to next task in line).But my child_dag gets in queued state and then scheduler starts skipping heartbeat,although there is no issue in the code as i was able to trigger child dag manually.Also,when i set wait_for_completion = False,it ran fine.
   
   ### What you think should happen instead?
   
   _No response_
   
   ### How to reproduce
   
   parent_dag.py
   from airflow import DAG
   from airflow.operators.dagrun_operator import TriggerDagRunOperator
   from airflow.operators.python_operator import PythonOperator
   from datetime import datetime, timedelta
   
   default_args = {
       'owner': 'airflow',
       'depends_on_past': False,
       'start_date': datetime(2024, 5, 27),
       'email_on_failure': False,
       'email_on_retry': False,
       'retries': 1,
   }
   
   parent_dag = DAG(
       'parent_dag',
       default_args=default_args,
       schedule_interval='@daily',
       catchup=False
   )
   
   def c():
       print("c")
   
   trigger_target = TriggerDagRunOperator(
           task_id='trigger_target',
           trigger_dag_id='child_dag',
           reset_dag_run=True,
           wait_for_completion=True,
           poke_interval=10,
           dag = parent_dag
       )
   task_after_child_dag_completion = PythonOperator(
       task_id = "task_after_child_dag_completion",
       python_callable = c,
       dag  = parent_dag
   )
   
   trigger_target >> task_after_child_dag_completion
   
   child_dag.py
   from airflow import DAG
   from airflow.operators.python_operator import PythonOperator
   from datetime import datetime, timedelta
   import time
   default_args = {
       'owner': 'airflow',
       'depends_on_past': False,
       'start_date': datetime(2024, 5, 27),
       'email_on_failure': False,
       'email_on_retry': False,
       'retries': 1,
   }
   
   child_dag = DAG(
       'child_dag',
       default_args=default_args,
       schedule_interval=None,
       catchup=False
   )
   
   def child_task():
       time.sleep(15)
       print("Executing child task")
   
   child_task = PythonOperator(
       task_id='child_task',
       python_callable=child_task,
       dag=child_dag
   )
   
   child_task
   
   
   ### Operating System
   
   Ubuntu 22.04.4 LTS
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Other
   
   ### Deployment details
   
   local env
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] wait_for_completion not working when set to True while triggering child dag from parent dag [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #39910:
URL: https://github.com/apache/airflow/issues/39910#issuecomment-2137133593

   > OK,can you suggest some other solution if any @potiuk .I tried using externaltasksensor as well and they also gave same issue.
   
   No i Have no suggestions. I have not looked into it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] wait_for_completion not working when set to True while triggering child dag from parent dag [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #39910:
URL: https://github.com/apache/airflow/issues/39910#issuecomment-2138758853

   > can you tag someone else who might know the alternate solution? @potiuk
   
   Nope. I have no idea whom I can tag. But what I suggest you is ypu can ask in Slack, open A girhub discussion about it, maybe even look in stack overflow, also you can check Ask Astro.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] wait_for_completion not working when set to True while triggering child dag from parent dag [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #39910:
URL: https://github.com/apache/airflow/issues/39910#issuecomment-2136896390

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] wait_for_completion not working when set to True while triggering child dag from parent dag [airflow]

Posted by "vashisht33 (via GitHub)" <gi...@apache.org>.
vashisht33 commented on issue #39910:
URL: https://github.com/apache/airflow/issues/39910#issuecomment-2137117060

   OK,can you suggest some other solution if any @potiuk .I tried using externaltasksensor as well and they also gave same issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] wait_for_completion not working when set to True while triggering child dag from parent dag [airflow]

Posted by "vashisht33 (via GitHub)" <gi...@apache.org>.
vashisht33 commented on issue #39910:
URL: https://github.com/apache/airflow/issues/39910#issuecomment-2138648440

   can you tag someone else who might know the alternate solution? @potiuk 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] wait_for_completion not working when set to True while triggering child dag from parent dag [airflow]

Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on issue #39910:
URL: https://github.com/apache/airflow/issues/39910#issuecomment-2137011977

   same than https://github.com/apache/airflow/issues/38353
   
   if you agree, you can close this issue, thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] wait_for_completion not working when set to True while triggering child dag from parent dag [airflow]

Posted by "vashisht33 (via GitHub)" <gi...@apache.org>.
vashisht33 commented on issue #39910:
URL: https://github.com/apache/airflow/issues/39910#issuecomment-2137036998

   What is the solution here? or expected timeline for bug fix?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] wait_for_completion not working when set to True while triggering child dag from parent dag [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #39910: wait_for_completion not working when set to True while triggering child dag from parent dag
URL: https://github.com/apache/airflow/issues/39910


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] wait_for_completion not working when set to True while triggering child dag from parent dag [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #39910:
URL: https://github.com/apache/airflow/issues/39910#issuecomment-2137048840

   No - it's just duplicate issue. And solution will be there when someone fixes it. This is open-source project and most certain way to be sure on "expected" timeline is to prepare a PR fixing it. Otherwise it will have to be picked by someone who will decide to fix it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org