[GitHub] [airflow] cjac opened a new issue #22064: Airflow jobs are sporadically re-queuing days after dags are completed

[GitBox <gi...@apache.org>]

cjac opened a new issue #22064:
URL: https://github.com/apache/airflow/issues/22064


   ### Apache Airflow version
   
   2.2.3
   
   ### What happened
   
   The queuing process in Airflow2 continues to access
   already-completed job, effectively corrupting start times/end
   times. This ruins the ability for job dependencies to chain together.
   
   After a long period of no change to completed tasks, while under heavy load, airflow re-schedules all historical jobs just to have them do no operation and complete.  When the job is re-queued, the start time is re-set, and nothing happens, and the end time is immediately re-set as well.
   
   The updating of these dates to NOW() is problematic, since there are listeners monitoring the values of these fields.  When the completion time is updated, other tasks are fired off.
   
   Why did the jobs get re-queued?
   
   We are able to reproduce the problem and provide a video call where questions may be asked.
   
   ### What you expected to happen
   
   Once tasks are complete, they should never be re-queued and re-executed.
   
   If tasks must be re-queued and re-executed, the start and end times of the already completed tasks must not be updated.
   
   
   ### How to reproduce
   
   Put environment under heavy load for a short period
   monitor the start/end times of tasks
   note that the start/end times are updated all concurrently
   
   ### Operating System
   
   composer-2.0.3-airflow-2.2.3
   
   ### Versions of Apache Airflow Providers
   
   composer-2.0.3-airflow-2.2.3
   
   
   ### Deployment
   
   Composer
   
   ### Deployment details
   
   Available on request from customer.  Please be discrete with these sensitive information.
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #22064: Airflow jobs are sporadically re-queuing days after dags are completed

[GitBox <gi...@apache.org>]

boring-cyborg[bot] commented on issue #22064:
URL: https://github.com/apache/airflow/issues/22064#issuecomment-1061242203


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org