You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "Maulik Soneji (JIRA)" <ji...@apache.org> on 2019/08/12 05:05:00 UTC

[jira] [Created] (AIRFLOW-5165) Make Dataproc highly available

Maulik Soneji created AIRFLOW-5165:
--------------------------------------

             Summary: Make Dataproc highly available
                 Key: AIRFLOW-5165
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5165
             Project: Apache Airflow
          Issue Type: Bug
          Components: plugins
    Affects Versions: 1.10.4, 1.10.3, 1.10.2, 1.10.1
            Reporter: Maulik Soneji


Currently the number of master nodes in dataproc create operator is set to 1.
There is no way to increase the number of master nodes in dataproc.

Inorder to have HA dataproc, Google Cloud suggests to have 3 master nodes
[https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/high-availability|https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/high-availability]

The number of masters should be a config in DataprocCreateOperator so that the created dataproc is HA



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)