[GitHub] [airflow] em-eman commented on issue #25438: Dags stucks on the "running" state

[GitBox <gi...@apache.org>]

em-eman commented on issue #25438:
URL: https://github.com/apache/airflow/issues/25438#issuecomment-1201005449

   Just to add more insights for current issue which is created by my colleague :
   
   There are two types of dags where dag one act as a internal scheduler which runs with 1 min heartbeat to check for files in s3 once the file is arrived in s3 then it will trigger the second dag to do data processing and other types of pythpn operator in our pipeline .
   Dag One  (scheduler) -> to check file in s3 and trigger dag two 
   <img width="879" alt="Screenshot 2022-08-01 at 12 11 24" src="https://user-images.githubusercontent.com/56021073/182126369-e7250b91-ede8-4af6-8158-b4a01469d887.png">
   Dag Two (upload dag)
   <img width="879" alt="Screenshot 2022-08-01 at 12 15 10" src="https://user-images.githubusercontent.com/56021073/182126988-76edf23b-3784-4427-95c1-d1b2711691c4.png">
   
   
   Problem happens in second dag where dag is stuck on success state for any of the above task while in flower (celery worker) task is active/running state forever. Seems like issue is between scheduler to worker communication for task state


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org