You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-issues@hadoop.apache.org by "william zhu (JIRA)" <ji...@apache.org> on 2015/12/31 02:59:49 UTC

[jira] [Created] (HADOOP-12684) CombineFileInputFormat may cause splits uneveness

william zhu created HADOOP-12684:
------------------------------------

             Summary: CombineFileInputFormat may cause splits uneveness
                 Key: HADOOP-12684
                 URL: https://issues.apache.org/jira/browse/HADOOP-12684
             Project: Hadoop Common
          Issue Type: Wish
         Environment: Hadoop 2.6
            Reporter: william zhu


CombieFileInputFormat add blocks into one split use below codes : 
```
 if (maxSize != 0 && curSplitSize >= maxSize) {
            // create an input split and add it to the splits array
            addCreatedSplit(splits, nodes, validBlocks);
            curSplitSize = 0;
            validBlocks.clear();
          }
```
but , in some cases , the file is not splitable. Then files may contains both large and small . In this case,  one split will be added a large file and then reach the  threshold  . 
I think some algorithm such as sort first will make the splits more even . 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)