You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "sunank200 (via GitHub)" <gi...@apache.org> on 2024/05/06 20:05:49 UTC

[PR] Workaround for asyncio cancelled error for DataprocCreateClusterOperator [airflow]

sunank200 opened a new pull request, #39446:
URL: https://github.com/apache/airflow/pull/39446

   <!--
    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at
   
      http://www.apache.org/licenses/LICENSE-2.0
   
    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.
    -->
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   [PR #39130](https://github.com/apache/airflow/pull/39130) introduces a method for handling asyncio.CancelledError in a try/except block. However, this method is deemed unsafe, and it affects `DataprocCreateClusterOperator` operators, which enables external job cancellation if the triggerer restarts or crashes. This can cause weird behaviour like rescheduling deferred operators, as Airflow remains unaware of job cancellations. 
   
   As a workaround, capturing `asyncio.CancelledError` cancels the job only if the TaskInstance is not in a running or deferred state. This prevents premature external job termination.
   
   More details at: https://github.com/apache/airflow/issues/36090#issuecomment-2094972855
   <!-- Please keep an empty line above the dashes. -->
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix logic to cancel the external job if the TaskInstance is not in a running or deferred state for DataprocCreateClusterOperator [airflow]

Posted by "sunank200 (via GitHub)" <gi...@apache.org>.
sunank200 commented on code in PR #39446:
URL: https://github.com/apache/airflow/pull/39446#discussion_r1592674019


##########
airflow/providers/google/cloud/triggers/dataproc.py:
##########
@@ -178,6 +184,36 @@ def serialize(self) -> tuple[str, dict[str, Any]]:
             },
         )
 
+    @provide_session
+    def get_task_instance(self, session: Session) -> TaskInstance:
+        query = session.query(TaskInstance).filter(
+            TaskInstance.dag_id == self.task_instance.dag_id,
+            TaskInstance.task_id == self.task_instance.task_id,
+            TaskInstance.run_id == self.task_instance.run_id,
+            TaskInstance.map_index == self.task_instance.map_index,
+        )
+        task_instance = query.one_or_none()
+        if task_instance is None:
+            raise AirflowException(
+                "TaskInstance dag_id: %s,task_id: %s, run_id: %s and map_index: %s not found",
+                self.task_instance.dag_id,
+                self.task_instance.task_id,
+                self.task_instance.run_id,
+                self.task_instance.map_index,
+            )
+        return task_instance
+
+    def safe_to_cancel(self) -> bool:
+        """
+        Whether it is safe to cancel the external job which is being executed by this trigger.
+
+        This is to avoid the case that `asyncio.CancelledError` is called because the trigger itself is stopped.
+        Because in those cases, we should NOT cancel the external job.
+        """
+        # Database query is needed to get the latest state of the task instance.
+        task_instance = self.get_task_instance()  # type: ignore[call-arg]
+        return task_instance.state != TaskInstanceState.DEFERRED

Review Comment:
   I can create a separate PR for the optimization



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix logic to cancel the external job if the TaskInstance is not in a running or deferred state for DataprocCreateClusterOperator [airflow]

Posted by "sunank200 (via GitHub)" <gi...@apache.org>.
sunank200 commented on PR #39446:
URL: https://github.com/apache/airflow/pull/39446#issuecomment-2097994226

   > Just have a quick discussion with @sunank200. We'll yield the event if CancelError was raised , and handle the TaskInstance state check and cancelation in `execute_complete`
   
   That won't work as the `execute_on_complete` won't be called when the task is cancelled. I have used task_instance from `BaseTrigger` instead.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix logic to cancel the external job if the TaskInstance is not in a running or deferred state for DataprocCreateClusterOperator [airflow]

Posted by "sunank200 (via GitHub)" <gi...@apache.org>.
sunank200 commented on PR #39446:
URL: https://github.com/apache/airflow/pull/39446#issuecomment-2098855321

   > Can you add the test case for this change?
   
   Added the test.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix logic to cancel the external job if the TaskInstance is not in a running or deferred state for DataprocCreateClusterOperator [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #39446:
URL: https://github.com/apache/airflow/pull/39446#discussion_r1592627819


##########
airflow/providers/google/cloud/triggers/dataproc.py:
##########
@@ -178,6 +184,36 @@ def serialize(self) -> tuple[str, dict[str, Any]]:
             },
         )
 
+    @provide_session
+    def get_task_instance(self, session: Session) -> TaskInstance:
+        query = session.query(TaskInstance).filter(
+            TaskInstance.dag_id == self.task_instance.dag_id,
+            TaskInstance.task_id == self.task_instance.task_id,
+            TaskInstance.run_id == self.task_instance.run_id,
+            TaskInstance.map_index == self.task_instance.map_index,
+        )
+        task_instance = query.one_or_none()
+        if task_instance is None:
+            raise AirflowException(
+                "TaskInstance dag_id: %s,task_id: %s, run_id: %s and map_index: %s not found",
+                self.task_instance.dag_id,
+                self.task_instance.task_id,
+                self.task_instance.run_id,
+                self.task_instance.map_index,
+            )
+        return task_instance
+
+    def safe_to_cancel(self) -> bool:
+        """
+        Whether it is safe to cancel the external job which is being executed by this trigger.
+
+        This is to avoid the case that `asyncio.CancelledError` is called because the trigger itself is stopped.
+        Because in those cases, we should NOT cancel the external job.
+        """
+        # Database query is needed to get the latest state of the task instance.
+        task_instance = self.get_task_instance()  # type: ignore[call-arg]
+        return task_instance.state != TaskInstanceState.DEFERRED

Review Comment:
   not really a blocker, but is an optimization we should do eventually



##########
airflow/providers/google/cloud/triggers/dataproc.py:
##########
@@ -178,6 +184,36 @@ def serialize(self) -> tuple[str, dict[str, Any]]:
             },
         )
 
+    @provide_session
+    def get_task_instance(self, session: Session) -> TaskInstance:
+        query = session.query(TaskInstance).filter(
+            TaskInstance.dag_id == self.task_instance.dag_id,
+            TaskInstance.task_id == self.task_instance.task_id,
+            TaskInstance.run_id == self.task_instance.run_id,
+            TaskInstance.map_index == self.task_instance.map_index,
+        )
+        task_instance = query.one_or_none()
+        if task_instance is None:
+            raise AirflowException(
+                "TaskInstance dag_id: %s,task_id: %s, run_id: %s and map_index: %s not found",
+                self.task_instance.dag_id,
+                self.task_instance.task_id,
+                self.task_instance.run_id,
+                self.task_instance.map_index,
+            )
+        return task_instance
+
+    def safe_to_cancel(self) -> bool:
+        """
+        Whether it is safe to cancel the external job which is being executed by this trigger.
+
+        This is to avoid the case that `asyncio.CancelledError` is called because the trigger itself is stopped.
+        Because in those cases, we should NOT cancel the external job.
+        """
+        # Database query is needed to get the latest state of the task instance.
+        task_instance = self.get_task_instance()  # type: ignore[call-arg]
+        return task_instance.state != TaskInstanceState.DEFERRED

Review Comment:
   I feel like we probably could make this check through SQL 🤔  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix logic to cancel the external job if the TaskInstance is not in a running or deferred state for DataprocCreateClusterOperator [airflow]

Posted by "sunank200 (via GitHub)" <gi...@apache.org>.
sunank200 commented on code in PR #39446:
URL: https://github.com/apache/airflow/pull/39446#discussion_r1592481410


##########
airflow/providers/google/cloud/triggers/dataproc.py:
##########
@@ -178,6 +179,18 @@ def serialize(self) -> tuple[str, dict[str, Any]]:
             },
         )
 
+    def safe_to_cancel(self) -> bool:

Review Comment:
   It attaches to an existing job that is triggered in the instance when it reattempts



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix logic to cancel the external job if the TaskInstance is not in a running or deferred state for DataprocCreateClusterOperator [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W merged PR #39446:
URL: https://github.com/apache/airflow/pull/39446


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix logic to cancel the external job if the TaskInstance is not in a running or deferred state for DataprocCreateClusterOperator [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on PR #39446:
URL: https://github.com/apache/airflow/pull/39446#issuecomment-2097776602

   Just have a quick discussion with @sunank200. We'll yield the event if CancelError was raised , and handle the TaskInstance state check and cancelation in `execute_complete`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix logic to cancel the external job if the TaskInstance is not in a running or deferred state for DataprocCreateClusterOperator [airflow]

Posted by "dirrao (via GitHub)" <gi...@apache.org>.
dirrao commented on code in PR #39446:
URL: https://github.com/apache/airflow/pull/39446#discussion_r1592248734


##########
airflow/providers/google/cloud/triggers/dataproc.py:
##########
@@ -178,6 +179,18 @@ def serialize(self) -> tuple[str, dict[str, Any]]:
             },
         )
 
+    def safe_to_cancel(self) -> bool:

Review Comment:
   When it is reattempted, does this attached to the existing external job or will it create new job?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org