You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@pig.apache.org by "Satish Subhashrao Saley (JIRA)" <ji...@apache.org> on 2018/10/12 19:23:00 UTC

[jira] [Created] (PIG-5365) Add support for PARALLEL clause in LOAD statement

Satish Subhashrao Saley created PIG-5365:
--------------------------------------------

             Summary: Add support for PARALLEL clause in LOAD statement
                 Key: PIG-5365
                 URL: https://issues.apache.org/jira/browse/PIG-5365
             Project: Pig
          Issue Type: New Feature
            Reporter: Satish Subhashrao Saley
            Assignee: Satish Subhashrao Saley


It is tiresome to keep telling users to increase pig.maxCombinedSplitSize to 512MB or 1G when they are reading TBs of data to avoid launching too many map tasks (50-100K) for loading data. It has unnecessary overhead in terms of container launch and wastes lot of resources. 

Would be good to have a new settings to configure the max number of tasks which will override pig.maxCombinedSplitSize and combine more splits into one task. For eg: pig.max.input.splits=30000 and data size is 2TB, it will combine more than 128MB (default pig.maxCombinedSplitSize) per task to have maximum of 30K tasks. That will go as default into pig-default.properties and apply to all users.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)