You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "kristopherkane (via GitHub)" <gi...@apache.org> on 2023/08/25 16:27:16 UTC

[GitHub] [airflow] kristopherkane commented on a diff in pull request #33668: Early delete a Dataproc cluster if started in the ERROR state.

kristopherkane commented on code in PR #33668:
URL: https://github.com/apache/airflow/pull/33668#discussion_r1305884663


##########
airflow/providers/google/cloud/operators/dataproc.py:
##########
@@ -594,12 +594,16 @@ def _handle_error_state(self, hook: DataprocHook, cluster: Cluster) -> None:
         if cluster.status.state != cluster.status.State.ERROR:
             return
         self.log.info("Cluster is in ERROR state")
+        self.log.info("Gathering diagnostic information.")
         gcs_uri = hook.diagnose_cluster(
             region=self.region, cluster_name=self.cluster_name, project_id=self.project_id
         )
         self.log.info("Diagnostic information for cluster %s available at: %s", self.cluster_name, gcs_uri)
         if self.delete_on_error:
             self._delete_cluster(hook)
+            # The delete op is asynchronous and can cause further failure if the cluster finishes
+            # deleting between catching AlreadyExists and checking state
+            self._wait_for_cluster_in_deleting_state(hook)
             raise AirflowException("Cluster was created but was in ERROR state.")
         raise AirflowException("Cluster was created but is in ERROR state")

Review Comment:
   I'll remove it. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org