[jira] [Resolved] (AIRFLOW-3149) GCP dataproc cluster creation should have the option to delete an ERROR cluster

["Jarek Potiuk (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jarek Potiuk resolved AIRFLOW-3149.
-----------------------------------
    Fix Version/s: 2.0.0
       Resolution: Fixed

> GCP dataproc cluster creation should have the option to delete an ERROR cluster
> -------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-3149
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3149
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: gcp
>    Affects Versions: 1.10.0
>            Reporter: Aaron Blake Niskode-Dossett
>            Assignee: Aaron Blake Niskode-Dossett
>            Priority: Minor
>             Fix For: 2.0.0
>
>
> We sometimes encounter issues where a dataproc cluster creation ends up in ERROR state. That is, the cluster “exists” but in the state of ERROR[1] (not just that the cluster creation API call failed). This makes retries impossible since the cluster name already exists subsequent retried creations are guaranteed to fail. 
> A `delete_cluster_on_error` parameter should be added to the `DataprocClusterCreateOperator` operator that controls whether or not an attempt to delete an ERROR cluster is made.
>  
> [1] - I’ve seen that happen in two ways 1) a purely transient error from GCP `Internal server error` or the like 2) when the request is rejected because it would exceed the project quota.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)