You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Kamil Bregula (Jira)" <ji...@apache.org> on 2019/09/17 14:30:00 UTC
[jira] [Closed] (AIRFLOW-3149) GCP dataproc cluster creation should
have the option to delete an ERROR cluster
[ https://issues.apache.org/jira/browse/AIRFLOW-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kamil Bregula closed AIRFLOW-3149.
----------------------------------
Resolution: Fixed
> GCP dataproc cluster creation should have the option to delete an ERROR cluster
> -------------------------------------------------------------------------------
>
> Key: AIRFLOW-3149
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3149
> Project: Apache Airflow
> Issue Type: Improvement
> Components: gcp
> Affects Versions: 1.10.0
> Reporter: Aaron Dossett
> Assignee: Aaron Dossett
> Priority: Minor
>
> We sometimes encounter issues where a dataproc cluster creation ends up in ERROR state. That is, the cluster “exists” but in the state of ERROR[1] (not just that the cluster creation API call failed). This makes retries impossible since the cluster name already exists subsequent retried creations are guaranteed to fail.
> A `delete_cluster_on_error` parameter should be added to the `DataprocClusterCreateOperator` operator that controls whether or not an attempt to delete an ERROR cluster is made.
>
> [1] - I’ve seen that happen in two ways 1) a purely transient error from GCP `Internal server error` or the like 2) when the request is rejected because it would exceed the project quota.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)