You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "Kuai Yu (JIRA)" <ji...@apache.org> on 2018/04/29 23:33:00 UTC
[jira] [Updated] (GOBBLIN-480) Allow job distribution cluster to be
separated from cluster manager cluster
[ https://issues.apache.org/jira/browse/GOBBLIN-480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kuai Yu updated GOBBLIN-480:
----------------------------
Description:
Today GobblinClusterManager leverages single Helix cluster responsible for both job distribution and cluster manager HA. This all-in-one mode cannot works with Helix super controller, because GobblinClusterManager will create its own dedicated controller for HA handling, which is internal to Gobblin framework. This architect works fine but gradually we find it's hard to monitor Helix behavior and debug Helix related issues due to the lack of Helix task framework metrics, which is enabled for free, but only available when using a dedicated controllers under Helix super controller's supervision.
To allow the migration, we separated existing cluster into two clusters:
1. Our existing cluster will remain the same, but called as "job distribution cluster" in the separation mode. In unit test or local deployment mode, we will create a dedicated controller for this cluster. In production mode, we can assume Helix will provide a dedicated controller for us.
2. A new cluster will be created, now called 'manager cluster', which is responsible for cluster manager leadership change. This will provide leadership change callback just like we did earlier in all-in-one mode.
The new 'two cluster mode' can be turned on/off by user configuration. Similarly user can configure whether a controller for job distribution should be created.
was:
Today GobblinClusterManager leverages single Helix cluster responsible for both job distribution and cluster manager HA. This all-in-one mode cannot works with Helix super controller, because GobblinClusterManager will create its own dedicated controller for HA handling, which is internal to Gobblin framework. This architect works fine but gradually we find it's hard to monitor Helix behavior and debug Helix related issues due to the lack of Helix task framework metrics, which is enabled for free, but only available when using a dedicated controllers under Helix super controller's supervision.
To allow the migration, we separated existing cluster into two clusters:
1. Our existing cluster will remain the same, called "job distribution cluster". In unit test or local deployment mode, we will create a dedicated controller for this cluster. In production mode, we assume Helix will provide this dedicated controller for us.
2. A new cluster will be created, called 'manager cluster', which is responsible for cluster manager leadership change. This will leadership change callback just like we did earlier in all-in-one mode.
Two cluster mode can be turned on/off by user configuration. Similarly to whether a controller for job distribution should be created.
> Allow job distribution cluster to be separated from cluster manager cluster
> ---------------------------------------------------------------------------
>
> Key: GOBBLIN-480
> URL: https://issues.apache.org/jira/browse/GOBBLIN-480
> Project: Apache Gobblin
> Issue Type: Improvement
> Reporter: Kuai Yu
> Assignee: Kuai Yu
> Priority: Major
>
> Today GobblinClusterManager leverages single Helix cluster responsible for both job distribution and cluster manager HA. This all-in-one mode cannot works with Helix super controller, because GobblinClusterManager will create its own dedicated controller for HA handling, which is internal to Gobblin framework. This architect works fine but gradually we find it's hard to monitor Helix behavior and debug Helix related issues due to the lack of Helix task framework metrics, which is enabled for free, but only available when using a dedicated controllers under Helix super controller's supervision.
> To allow the migration, we separated existing cluster into two clusters:
> 1. Our existing cluster will remain the same, but called as "job distribution cluster" in the separation mode. In unit test or local deployment mode, we will create a dedicated controller for this cluster. In production mode, we can assume Helix will provide a dedicated controller for us.
> 2. A new cluster will be created, now called 'manager cluster', which is responsible for cluster manager leadership change. This will provide leadership change callback just like we did earlier in all-in-one mode.
> The new 'two cluster mode' can be turned on/off by user configuration. Similarly user can configure whether a controller for job distribution should be created.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)