You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/02 23:00:50 UTC
[GitHub] [airflow] jedcunningham opened a new pull request #19375: Only mark SchedulerJobs as failed, not any jobs
jedcunningham opened a new pull request #19375:
URL: https://github.com/apache/airflow/pull/19375
In `adopt_or_reset_orphaned_tasks`, we set any SchedulerJobs that have
failed `scheduler_health_check_threshold` to failed, however a missing
condition was allowing that timeout to apply to all jobs, not just SchedulerJobs.
This is because polymorphic identity isn't included for `update()`:
https://docs.sqlalchemy.org/en/13/orm/query.html#sqlalchemy.orm.query.Query.update
So if we had any running LocalTaskJobs that, for whatever reason, aren't
heartbeating faster than `scheduler_health_check_threshold`, their state
gets set to failed and they subsequently exit with a log line similar to:
State of this instance has been externally set to scheduled. Terminating instance.
Note that the state it is set to can be different (e.g. queued or
up_for_retry) simply depending on how quickly the scheduler has
progressed that task_instance again.
Closes: #16881
Closes: #16573
Related: https://github.com/apache/airflow/issues/16023#issuecomment-856087959
Might also fix #19277
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil edited a comment on pull request #19375: Only mark SchedulerJobs as failed, not any jobs
Posted by GitBox <gi...@apache.org>.
kaxil edited a comment on pull request #19375:
URL: https://github.com/apache/airflow/pull/19375#issuecomment-958437079
Good find - I think @ephraimbuddy and I had stumbled upon it sometime back but weren't sure about that Polymorphic identity didn't apply to update statements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on pull request #19375: Only mark SchedulerJobs as failed, not any jobs
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on pull request #19375:
URL: https://github.com/apache/airflow/pull/19375#issuecomment-959190436
Yeah, shoutout to @ephraimbuddy for spotting this 🎉
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #19375: Only mark SchedulerJobs as failed, not any jobs
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #19375:
URL: https://github.com/apache/airflow/pull/19375#issuecomment-958688556
> Good find - I think @ephraimbuddy and I had stumbled upon it sometime back but weren't sure that Polymorphic identity didn't apply to update statements - TIL.
Yeah! Yesterday during our debugging session with Collin, it came up again. Good that it's now being fixed. Likely related to the many sigterms everyone is complaining about
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed pull request #19375: Only mark SchedulerJobs as failed, not any jobs
Posted by GitBox <gi...@apache.org>.
kaxil closed pull request #19375:
URL: https://github.com/apache/airflow/pull/19375
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #19375: Only mark SchedulerJobs as failed, not any jobs
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #19375:
URL: https://github.com/apache/airflow/pull/19375#issuecomment-959201643
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #19375: Only mark SchedulerJobs as failed, not any jobs
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #19375:
URL: https://github.com/apache/airflow/pull/19375#issuecomment-958688556
> Good find - I think @ephraimbuddy and I had stumbled upon it sometime back but weren't sure that Polymorphic identity didn't apply to update statements - TIL.
Yeah! Yesterday during our debugging session with Collin, it came up again. Good that it's now being fixed. Likely related to the many sigterms everyone is complaining about
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] collinmcnulty edited a comment on pull request #19375: Only mark SchedulerJobs as failed, not any jobs
Posted by GitBox <gi...@apache.org>.
collinmcnulty edited a comment on pull request #19375:
URL: https://github.com/apache/airflow/pull/19375#issuecomment-959131533
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] collinmcnulty commented on pull request #19375: Only mark SchedulerJobs as failed, not any jobs
Posted by GitBox <gi...@apache.org>.
collinmcnulty commented on pull request #19375:
URL: https://github.com/apache/airflow/pull/19375#issuecomment-959131533
Hallelujah
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19375: Only mark SchedulerJobs as failed, not any jobs
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #19375:
URL: https://github.com/apache/airflow/pull/19375#issuecomment-958432930
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil edited a comment on pull request #19375: Only mark SchedulerJobs as failed, not any jobs
Posted by GitBox <gi...@apache.org>.
kaxil edited a comment on pull request #19375:
URL: https://github.com/apache/airflow/pull/19375#issuecomment-958437079
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on pull request #19375: Only mark SchedulerJobs as failed, not any jobs
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on pull request #19375:
URL: https://github.com/apache/airflow/pull/19375#issuecomment-959190436
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed pull request #19375: Only mark SchedulerJobs as failed, not any jobs
Posted by GitBox <gi...@apache.org>.
kaxil closed pull request #19375:
URL: https://github.com/apache/airflow/pull/19375
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil edited a comment on pull request #19375: Only mark SchedulerJobs as failed, not any jobs
Posted by GitBox <gi...@apache.org>.
kaxil edited a comment on pull request #19375:
URL: https://github.com/apache/airflow/pull/19375#issuecomment-958437079
Good find - I think @ephraimbuddy and I had stumbled upon it sometime back but weren't sure that Polymorphic identity didn't apply to update statements - TIL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] collinmcnulty edited a comment on pull request #19375: Only mark SchedulerJobs as failed, not any jobs
Posted by GitBox <gi...@apache.org>.
collinmcnulty edited a comment on pull request #19375:
URL: https://github.com/apache/airflow/pull/19375#issuecomment-959131533
Hallelujah, great fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #19375: Only mark SchedulerJobs as failed, not any jobs
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #19375:
URL: https://github.com/apache/airflow/pull/19375#issuecomment-958437079
Good find - I think @ephraimbuddy and I had stumbled upon it but weren't sure about that Polymorphic identity didn't apply to update statements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy merged pull request #19375: Only mark SchedulerJobs as failed, not any jobs
Posted by GitBox <gi...@apache.org>.
ephraimbuddy merged pull request #19375:
URL: https://github.com/apache/airflow/pull/19375
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] collinmcnulty commented on pull request #19375: Only mark SchedulerJobs as failed, not any jobs
Posted by GitBox <gi...@apache.org>.
collinmcnulty commented on pull request #19375:
URL: https://github.com/apache/airflow/pull/19375#issuecomment-959131533
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19375: Only mark SchedulerJobs as failed, not any jobs
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #19375:
URL: https://github.com/apache/airflow/pull/19375#issuecomment-958432930
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #19375: Only mark SchedulerJobs as failed, not any jobs
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #19375:
URL: https://github.com/apache/airflow/pull/19375#issuecomment-959201643
:facepalm:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #19375: Only mark SchedulerJobs as failed, not any jobs
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #19375:
URL: https://github.com/apache/airflow/pull/19375#issuecomment-958688556
> Good find - I think @ephraimbuddy and I had stumbled upon it sometime back but weren't sure that Polymorphic identity didn't apply to update statements - TIL.
Yeah! Yesterday during our debugging session with Collin, it came up again. Good that it's now being fixed. Likely related to the many sigterms everyone is complaining about
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] collinmcnulty commented on pull request #19375: Only mark SchedulerJobs as failed, not any jobs
Posted by GitBox <gi...@apache.org>.
collinmcnulty commented on pull request #19375:
URL: https://github.com/apache/airflow/pull/19375#issuecomment-959131533
Hallelujah
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy merged pull request #19375: Only mark SchedulerJobs as failed, not any jobs
Posted by GitBox <gi...@apache.org>.
ephraimbuddy merged pull request #19375:
URL: https://github.com/apache/airflow/pull/19375
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #19375: Only mark SchedulerJobs as failed, not any jobs
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #19375:
URL: https://github.com/apache/airflow/pull/19375#issuecomment-958437079
Good find - I think @ephraimbuddy and I had stumbled upon it but weren't sure about that Polymorphic identity didn't apply to update statements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed pull request #19375: Only mark SchedulerJobs as failed, not any jobs
Posted by GitBox <gi...@apache.org>.
kaxil closed pull request #19375:
URL: https://github.com/apache/airflow/pull/19375
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #19375: Only mark SchedulerJobs as failed, not any jobs
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #19375:
URL: https://github.com/apache/airflow/pull/19375#issuecomment-958688556
> Good find - I think @ephraimbuddy and I had stumbled upon it sometime back but weren't sure that Polymorphic identity didn't apply to update statements - TIL.
Yeah! Yesterday during our debugging session with Collin, it came up again. Good that it's now being fixed. Likely related to the many sigterms everyone is complaining about
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy merged pull request #19375: Only mark SchedulerJobs as failed, not any jobs
Posted by GitBox <gi...@apache.org>.
ephraimbuddy merged pull request #19375:
URL: https://github.com/apache/airflow/pull/19375
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #19375: Only mark SchedulerJobs as failed, not any jobs
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #19375:
URL: https://github.com/apache/airflow/pull/19375#issuecomment-959201643
:facepalm:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] collinmcnulty edited a comment on pull request #19375: Only mark SchedulerJobs as failed, not any jobs
Posted by GitBox <gi...@apache.org>.
collinmcnulty edited a comment on pull request #19375:
URL: https://github.com/apache/airflow/pull/19375#issuecomment-959131533
Hallelujah, great fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on pull request #19375: Only mark SchedulerJobs as failed, not any jobs
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on pull request #19375:
URL: https://github.com/apache/airflow/pull/19375#issuecomment-959190436
Yeah, shoutout to @ephraimbuddy for spotting this 🎉
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org