You are viewing a plain text version of this content. The canonical link for it is here.

Posted to mapreduce-dev@hadoop.apache.org by "Siddharth Seth (Created) (JIRA)" <ji...@apache.org> on 2012/01/23 20:13:41 UTC

[jira] [Created] (MAPREDUCE-3710) last split generated by FileInputFormat.getSplits may not have the best locality

last split generated by FileInputFormat.getSplits may not have the best locality
--------------------------------------------------------------------------------

                 Key: MAPREDUCE-3710
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3710
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: mrv1, mrv2
    Affects Versions: 1.0.0, 0.23.0
            Reporter: Siddharth Seth
            Assignee: Siddharth Seth


The last split generated by FileInputFormat.getSplits considers {{blkLocations.length-1}} to be the hosts for the split.
The last split may be larger than the rest (SPLIT_SLOP=1.1 by default) - in which case locality is picked up from a smaller block.
e.g. 1027MB file with a 128MB split size. The last split ends up being 131MB. The hosts for locality end up being the nodes containing the 3MB block instead of the 128MB block.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira