[jira] [Created] (GOBBLIN-480) Allow job distribution cluster to be separated from cluster manager cluster

["Kuai Yu (JIRA)" <ji...@apache.org>]

Kuai Yu created GOBBLIN-480:
-------------------------------

             Summary: Allow job distribution cluster to be separated from cluster manager cluster
                 Key: GOBBLIN-480
                 URL: https://issues.apache.org/jira/browse/GOBBLIN-480
             Project: Apache Gobblin
          Issue Type: Improvement
            Reporter: Kuai Yu
            Assignee: Kuai Yu


Today GobblinClusterManager leverages single Helix cluster responsible for both job distribution and cluster manager HA. This all-in-one mode cannot works with Helix super controller, because GobblinClusterManager will create its own dedicated controller for HA handling, which is internal to Gobblin framework. This architect works fine but gradually we find it's hard to monitor Helix behavior and debug Helix related issues due to the lack of Helix task framework metrics, which is enabled for free, but only available when using a dedicated controllers under Helix super controller's supervision.

To allow the migration, we separated existing cluster into two clusters:

1. Our existing cluster will remain the same, called "job distribution cluster". In unit test or local deployment mode, we will create a dedicated controller for this cluster. In production mode, we assume Helix will provide this dedicated controller for us.

2. A new cluster will be created, called 'manager cluster', which is responsible for cluster manager leadership change. This will leadership change callback just like we did earlier in all-in-one mode.

Two cluster mode can be turned on/off by user configuration. Similarly to whether a controller for job distribution should be created.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)