You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/05/06 10:11:12 UTC

[GitHub] [airflow] ecerulm opened a new pull request, #23521: Prevent KubernetesJobWatcher getting stuck on resource too old

ecerulm opened a new pull request, #23521:
URL: https://github.com/apache/airflow/pull/23521

   If the watch fails because "resource too old" the
   KubernetesJobWatcher should not retry with the same resource version
   as that will end up in loop where there is no progress.
   
   In this PR I take the approach of resetting the `resource_version` to `'0'` on any unexpected exception. It's possible to try to narrow it down to `ApiException` but only if `.status==410`, etc but I feel like it's it's just safer to start watching fresh again. 
   
   Related  to #21087
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragement file, named `{pr_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ephraimbuddy commented on pull request #23521: Prevent KubernetesJobWatcher getting stuck on resource too old

Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on PR #23521:
URL: https://github.com/apache/airflow/pull/23521#issuecomment-1119558969

   You may want to look at this comment https://github.com/apache/airflow/pull/15500#issuecomment-883294131
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ncapeta commented on pull request #23521: Prevent KubernetesJobWatcher getting stuck on resource too old

Posted by GitBox <gi...@apache.org>.
ncapeta commented on PR #23521:
URL: https://github.com/apache/airflow/pull/23521#issuecomment-1123137391

   any potential release date ? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ncapeta commented on pull request #23521: Prevent KubernetesJobWatcher getting stuck on resource too old

Posted by GitBox <gi...@apache.org>.
ncapeta commented on PR #23521:
URL: https://github.com/apache/airflow/pull/23521#issuecomment-1120254400

   Any possibility to get a release with this ? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #23521: Prevent KubernetesJobWatcher getting stuck on resource too old

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #23521:
URL: https://github.com/apache/airflow/pull/23521#issuecomment-1120257206

   > Any possibility to get a release with this ?
   
   I am about to prepare next release of providers very soon. If it gets merged before that it will go to that release - if not, then it will have to wait for another one.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ecerulm commented on pull request #23521: Prevent KubernetesJobWatcher getting stuck on resource too old

Posted by GitBox <gi...@apache.org>.
ecerulm commented on PR #23521:
URL: https://github.com/apache/airflow/pull/23521#issuecomment-1127719478

   Yes, you potentially miss old event just like before. The only difference is that the scheduler does not get into an infinite loop because of it.
   
   If airflow gets a 410 from kubernetes api that already is a pretty good indicator that some event has been lost but there is nothing airflow can do about it at that point, right?
   
   the question remains if kubernetes offers the right API with the right guarantees to implement this kind of scheduling reliably. Or if other approach would be better. This PR was only about preventing the scheduler from entering an infinite loop in which progress was impossible. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #23521: Prevent KubernetesJobWatcher getting stuck on resource too old

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #23521:
URL: https://github.com/apache/airflow/pull/23521#issuecomment-1120273380

   > Tks, this is making in some instances 2.3 not usable at all as the schedulers after 1h need to be deleted
   
   Yeah. 2.3.1 with some teething problems og 2.3.0  solved is coming soon.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk merged pull request #23521: Prevent KubernetesJobWatcher getting stuck on resource too old

Posted by GitBox <gi...@apache.org>.
potiuk merged PR #23521:
URL: https://github.com/apache/airflow/pull/23521


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ashb commented on pull request #23521: Prevent KubernetesJobWatcher getting stuck on resource too old

Posted by GitBox <gi...@apache.org>.
ashb commented on PR #23521:
URL: https://github.com/apache/airflow/pull/23521#issuecomment-1127528439

   Does this change mean that we potentially miss some events?
   
   i.e. if the version _n_ is too old, and we reset it to 0, and the next event is _m_, then any pods state changes in _n_ to _m_ won't ever be seen?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #23521: Prevent KubernetesJobWatcher getting stuck on resource too old

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #23521:
URL: https://github.com/apache/airflow/pull/23521#issuecomment-1123238138

   > any potential release date ?
   
   Whatever we tell you we might not hold to because we might not be able to find/fix all important issues by then. And if you base any "business" decisions of yours on what we tell, then you will have complaints. So better not to give you false hopes for OSS project.
   
   Just watch how many issues are left here (some might get added though still):
   
   https://github.com/apache/airflow/milestone/54
   
   However being an OSS project it has an advantage, that you can just take the change and apply it yourself in your environment while waiting. It's a few line change that you can easily apply on installed Airflow - just find the installed sources and update them in the way that they will persist in your environment. This way you can handle your "business" and you cannot tell "I am stuck becasue they have not released yet"..
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ncapeta commented on pull request #23521: Prevent KubernetesJobWatcher getting stuck on resource too old

Posted by GitBox <gi...@apache.org>.
ncapeta commented on PR #23521:
URL: https://github.com/apache/airflow/pull/23521#issuecomment-1120270601

   Tks, this is making in some instances 2.3 not usable at all as the schedulers after 1h need to be deleted  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #23521: Prevent KubernetesJobWatcher getting stuck on resource too old

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #23521:
URL: https://github.com/apache/airflow/pull/23521#issuecomment-1120263457

   > The files modified by this PR are of Airflow core so it doesn't matter for provider release :)
   
   Ah. Right. The Provider vs Core Kubernetes strikes back :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] eladkal commented on pull request #23521: Prevent KubernetesJobWatcher getting stuck on resource too old

Posted by GitBox <gi...@apache.org>.
eladkal commented on PR #23521:
URL: https://github.com/apache/airflow/pull/23521#issuecomment-1120258472

   > I am about to prepare next release of providers very soon. If it gets merged before that it will go to that release - if not, then it will have to wait for the next one.
   
   The files modified by this PR are of Airflow core so it doesn't matter for provider release :)
   I'll mark it for 2.3.1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org