You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@gobblin.apache.org by "Zihan Li (JIRA)" <ji...@apache.org> on 2019/03/29 20:48:00 UTC

[jira] [Created] (GOBBLIN-717) Filter Out Empty MultiWorkUnits

Zihan Li created GOBBLIN-717:
--------------------------------

             Summary: Filter Out Empty MultiWorkUnits
                 Key: GOBBLIN-717
                 URL: https://issues.apache.org/jira/browse/GOBBLIN-717
             Project: Apache Gobblin
          Issue Type: Improvement
            Reporter: Zihan Li


Now when we run a job, Gobblin use the value of max mappers or the target size of a mapper to determine the number of mappers. But since one partition cannot be divided into several WorkUnits, work cannot be evenly distributed, there are many mappers(MultiWorkUnits) have no work to do. This will waste a lot of resources. So we need to filter out MultiWorkUnits which contains no WorkUnit when we determine the number of mappers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)