You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Rohini Palaniswamy (JIRA)" <ji...@apache.org> on 2014/04/17 21:20:15 UTC

[jira] [Created] (PIG-3903) Configure mapred.min.split.size to be same as pig.maxCombinedSplitSize

Rohini Palaniswamy created PIG-3903:
---------------------------------------

             Summary: Configure mapred.min.split.size to be same as pig.maxCombinedSplitSize
                 Key: PIG-3903
                 URL: https://issues.apache.org/jira/browse/PIG-3903
             Project: Pig
          Issue Type: Bug
            Reporter: Rohini Palaniswamy


FileInputFormat calculates the split size as 
Math.max(minSize, Math.min(maxSize, blockSize));

By default pig.maxCombinedSplitSize is 128MB if pig.noSplitCombinaton is not specifically turned off. We should set the mapred.min.split.size (if not already set by the user) to same as pig.maxCombinedSplitSize, so the underlying FileInputFormat itself gives us bigger splits when possible instead of pig combining smaller splits.



--
This message was sent by Atlassian JIRA
(v6.2#6252)