[jira] [Commented] (AIRFLOW-3149) GCP dataproc cluster creation should have the option to delete an ERROR cluster

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16931285#comment-16931285 ] 

ASF GitHub Bot commented on AIRFLOW-3149:
-----------------------------------------

mik-laj commented on pull request #4064: AIRFLOW-3149 Support dataproc cluster deletion on ERROR
URL: https://github.com/apache/airflow/pull/4064
 
 
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> GCP dataproc cluster creation should have the option to delete an ERROR cluster
> -------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-3149
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3149
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: gcp
>    Affects Versions: 1.10.0
>            Reporter: Aaron Dossett
>            Assignee: Aaron Dossett
>            Priority: Minor
>
> We sometimes encounter issues where a dataproc cluster creation ends up in ERROR state. That is, the cluster “exists” but in the state of ERROR[1] (not just that the cluster creation API call failed). This makes retries impossible since the cluster name already exists subsequent retried creations are guaranteed to fail. 
> A `delete_cluster_on_error` parameter should be added to the `DataprocClusterCreateOperator` operator that controls whether or not an attempt to delete an ERROR cluster is made.
>  
> [1] - I’ve seen that happen in two ways 1) a purely transient error from GCP `Internal server error` or the like 2) when the request is rejected because it would exceed the project quota.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)