You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "NotYuki (via GitHub)" <gi...@apache.org> on 2024/05/05 09:27:43 UTC

[PR] FIX: do not reattach to finished pod on restart of k8s pod operator [airflow]

NotYuki opened a new pull request, #39414:
URL: https://github.com/apache/airflow/pull/39414

   For now, if KubernetesPodOperator.reattach_on_restart is enabled, the Operator tries to find a pod regardless of its status. This might lead to an unhealthy deadlock situation if on_finish_action is set to KEEP_POD or DELETE_SUCCEEDED_POD.
   
   For instance, if a task has on_finish_action = OnFinishAction.DELETE_SUCCEEDED_POD and it fails on the first run, any subsequent attempts of the same task will fail immediately with a Kubernetes API error, as Airflow will try to attach to a finished pod.
   
   This PR introduces changes to KPO, allowing it to reattach only to running pods.
   
   <!--
    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at
   
      http://www.apache.org/licenses/LICENSE-2.0
   
    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.
    -->
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   
   
   <!-- Please keep an empty line above the dashes. -->
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] FIX: do not reattach to finished pod on restart of k8s pod operator [airflow]

Posted by "NotYuki (via GitHub)" <gi...@apache.org>.
NotYuki commented on PR #39414:
URL: https://github.com/apache/airflow/pull/39414#issuecomment-2094795348

   Thanks for the comment. Actually, after delving deeper into the Kubernetes tests and KPO code, I've noticed similar logic already implemented with patch_already_checked() method. I want to spend some time investigating why it doesn't work as expected on my k8s environment.
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] FIX: do not reattach to finished pod on restart of k8s pod operator [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #39414:
URL: https://github.com/apache/airflow/pull/39414#issuecomment-2094704206

   Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
   Here are some useful points:
   - Pay attention to the quality of your code (ruff, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst#prerequisites-for-pre-commit-hooks) will help you with that.
   - In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
   - Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
   - Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
   - Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
   - Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#coding-style-and-best-practices).
   - Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
   Apache Airflow is a community-driven project and together we are making it better 🚀.
   In case of doubts contact the developers at:
   Mailing List: dev@airflow.apache.org
   Slack: https://s.apache.org/airflow-slack
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] FIX: do not reattach to finished pod on restart of k8s pod operator [airflow]

Posted by "dirrao (via GitHub)" <gi...@apache.org>.
dirrao commented on code in PR #39414:
URL: https://github.com/apache/airflow/pull/39414#discussion_r1590531270


##########
airflow/providers/cncf/kubernetes/operators/pod.py:
##########
@@ -544,8 +544,14 @@ def find_pod(self, namespace: str, context: Context, *, exclude_checked: bool =
 
     def get_or_create_pod(self, pod_request_obj: k8s.V1Pod, context: Context) -> k8s.V1Pod:
         if self.reattach_on_restart:
-            pod = self.find_pod(self.namespace or pod_request_obj.metadata.namespace, context=context)
-            if pod:
+            pod_request_obj_namespace = (
+                pod_request_obj.metadata.namespace if pod_request_obj.metadata else None
+            )
+            namespace = self.namespace or pod_request_obj_namespace
+            if not namespace:
+                raise ValueError("Pod namespace is empty")
+            pod = self.find_pod(namespace, context=context)
+            if pod and pod.status and pod.status.phase == "Running":

Review Comment:
   When the next retry attempt comes back, by that time the task might move from running to completed state. Why do you want to rerun the same? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org