You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jscheffl (via GitHub)" <gi...@apache.org> on 2024/11/18 19:55:12 UTC
[PR] [v2-10-test] Re-queue tassk when they are stuck in queued (#43520) [airflow]
jscheffl opened a new pull request, #44158:
URL: https://github.com/apache/airflow/pull/44158
Backport of #43520.
Note: Cherry-pick is w/o K8s provider files as these are always taken from main during test and release.
The old "stuck in queued" logic just failed the tasks. Now we requeue them. We accomplish this by revoking the task from executor and setting state to scheduled. We'll re-queue it up to 2 times. Number of times is configurable by hidden config.
We added a method to base executor revoke_task because, it's a discrete operation that is required for this feature, and it might be useful in other cases e.g. when detecting as zombies etc. We set state to failed or scheduled directly from scheduler (rather than sending through the event buffer) because event buffer makes more sense for handling external events -- why round trip through the executor and back to scheduler when scheduler is initiating the action? Anyway this avoids having to deal with "state mismatch" issues when processing events.
---------
(cherry picked from commit a41feeb5aedad842be2b0f954e0be30c767dbc5e)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [v2-10-test] Re-queue tassk when they are stuck in queued (#43520) [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl merged PR #44158:
URL: https://github.com/apache/airflow/pull/44158
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [v2-10-test] Re-queue tassk when they are stuck in queued (#43520) [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on PR #44158:
URL: https://github.com/apache/airflow/pull/44158#issuecomment-2484192416
might need this as well @jscheffl https://github.com/apache/airflow/pull/44093
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [v2-10-test] Re-queue tassk when they are stuck in queued (#43520) [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #44158:
URL: https://github.com/apache/airflow/pull/44158#issuecomment-2484198081
> might need this as well @jscheffl #44093
Yeeah, figured out the same commit right at the same time :-D Added to the PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org