You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Rohini Palaniswamy (JIRA)" <ji...@apache.org> on 2014/04/17 21:20:15 UTC
[jira] [Created] (PIG-3903) Configure mapred.min.split.size to be
same as pig.maxCombinedSplitSize
Rohini Palaniswamy created PIG-3903:
---------------------------------------
Summary: Configure mapred.min.split.size to be same as pig.maxCombinedSplitSize
Key: PIG-3903
URL: https://issues.apache.org/jira/browse/PIG-3903
Project: Pig
Issue Type: Bug
Reporter: Rohini Palaniswamy
FileInputFormat calculates the split size as
Math.max(minSize, Math.min(maxSize, blockSize));
By default pig.maxCombinedSplitSize is 128MB if pig.noSplitCombinaton is not specifically turned off. We should set the mapred.min.split.size (if not already set by the user) to same as pig.maxCombinedSplitSize, so the underlying FileInputFormat itself gives us bigger splits when possible instead of pig combining smaller splits.
--
This message was sent by Atlassian JIRA
(v6.2#6252)