You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@gobblin.apache.org by "Kuai Yu (JIRA)" <ji...@apache.org> on 2018/05/10 21:55:00 UTC

[jira] [Updated] (GOBBLIN-490) Allow job to be distributed by Helix

     [ https://issues.apache.org/jira/browse/GOBBLIN-490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kuai Yu updated GOBBLIN-490:
----------------------------
    Description: This new job launcher will submit original job to Helix instead of the generated work units. This will allow Helix to re-distribute the original jobs to different worker nodes. Each worker then process the original jobs and launch its own GobblinHelixJobLauncher. By doing this, we will relieve manager node because the task driver logic (mainly generate work units) is now distributed to worker nodes.

> Allow job to be distributed by Helix
> ------------------------------------
>
>                 Key: GOBBLIN-490
>                 URL: https://issues.apache.org/jira/browse/GOBBLIN-490
>             Project: Apache Gobblin
>          Issue Type: New Feature
>            Reporter: Kuai Yu
>            Assignee: Kuai Yu
>            Priority: Major
>
> This new job launcher will submit original job to Helix instead of the generated work units. This will allow Helix to re-distribute the original jobs to different worker nodes. Each worker then process the original jobs and launch its own GobblinHelixJobLauncher. By doing this, we will relieve manager node because the task driver logic (mainly generate work units) is now distributed to worker nodes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)