You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Siddharth Seth (Created) (JIRA)" <ji...@apache.org> on 2012/01/23 20:13:41 UTC
[jira] [Created] (MAPREDUCE-3710) last split generated by
FileInputFormat.getSplits may not have the best locality
last split generated by FileInputFormat.getSplits may not have the best locality
--------------------------------------------------------------------------------
Key: MAPREDUCE-3710
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3710
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1, mrv2
Affects Versions: 1.0.0, 0.23.0
Reporter: Siddharth Seth
Assignee: Siddharth Seth
The last split generated by FileInputFormat.getSplits considers {{blkLocations.length-1}} to be the hosts for the split.
The last split may be larger than the rest (SPLIT_SLOP=1.1 by default) - in which case locality is picked up from a smaller block.
e.g. 1027MB file with a 128MB split size. The last split ends up being 131MB. The hosts for locality end up being the nodes containing the 3MB block instead of the 128MB block.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3710) last split generated by
FileInputFormat.getSplits may not have the best locality
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192559#comment-13192559 ]
Hudson commented on MAPREDUCE-3710:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1579 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1579/])
MAPREDUCE-3710. Improved FileInputFormat to return better locality for the last split. Contributed by Siddarth Seth.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235510
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
> last split generated by FileInputFormat.getSplits may not have the best locality
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3710
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3710
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
> Affects Versions: 0.23.0, 1.0.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Fix For: 0.23.1
>
> Attachments: MR-3710_v1.txt
>
>
> The last split generated by FileInputFormat.getSplits considers {{blkLocations.length-1}} to be the hosts for the split.
> The last split may be larger than the rest (SPLIT_SLOP=1.1 by default) - in which case locality is picked up from a smaller block.
> e.g. 1027MB file with a 128MB split size. The last split ends up being 131MB. The hosts for locality end up being the nodes containing the 3MB block instead of the 128MB block.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3710) last split generated by
FileInputFormat.getSplits may not have the best locality
Posted by "Siddharth Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3710:
--------------------------------------
Status: Patch Available (was: Open)
> last split generated by FileInputFormat.getSplits may not have the best locality
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3710
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3710
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
> Affects Versions: 1.0.0, 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Attachments: MR-3710_v1.txt
>
>
> The last split generated by FileInputFormat.getSplits considers {{blkLocations.length-1}} to be the hosts for the split.
> The last split may be larger than the rest (SPLIT_SLOP=1.1 by default) - in which case locality is picked up from a smaller block.
> e.g. 1027MB file with a 128MB split size. The last split ends up being 131MB. The hosts for locality end up being the nodes containing the 3MB block instead of the 128MB block.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3710) last split generated by
FileInputFormat.getSplits may not have the best locality
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192560#comment-13192560 ]
Hudson commented on MAPREDUCE-3710:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #413 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/413/])
MAPREDUCE-3710. Improved FileInputFormat to return better locality for the last split. Contributed by Siddarth Seth.
svn merge --ignore-ancestry -c 1235510 ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235511
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
> last split generated by FileInputFormat.getSplits may not have the best locality
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3710
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3710
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
> Affects Versions: 0.23.0, 1.0.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Fix For: 0.23.1
>
> Attachments: MR-3710_v1.txt
>
>
> The last split generated by FileInputFormat.getSplits considers {{blkLocations.length-1}} to be the hosts for the split.
> The last split may be larger than the rest (SPLIT_SLOP=1.1 by default) - in which case locality is picked up from a smaller block.
> e.g. 1027MB file with a 128MB split size. The last split ends up being 131MB. The hosts for locality end up being the nodes containing the 3MB block instead of the 128MB block.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3710) last split generated by
FileInputFormat.getSplits may not have the best locality
Posted by "Siddharth Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3710:
--------------------------------------
Attachment: MR-3710_v1.txt
Patch for the last split.
Mapred.FileInputFormat seems to have additional optimizations for rack locality which don't exist in mapreduce.FileInputFormat. The patch does not include a fix for this.
> last split generated by FileInputFormat.getSplits may not have the best locality
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3710
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3710
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
> Affects Versions: 0.23.0, 1.0.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Attachments: MR-3710_v1.txt
>
>
> The last split generated by FileInputFormat.getSplits considers {{blkLocations.length-1}} to be the hosts for the split.
> The last split may be larger than the rest (SPLIT_SLOP=1.1 by default) - in which case locality is picked up from a smaller block.
> e.g. 1027MB file with a 128MB split size. The last split ends up being 131MB. The hosts for locality end up being the nodes containing the 3MB block instead of the 128MB block.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3710) last split generated by
FileInputFormat.getSplits may not have the best locality
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193079#comment-13193079 ]
Hudson commented on MAPREDUCE-3710:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #969 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/969/])
MAPREDUCE-3710. Improved FileInputFormat to return better locality for the last split. Contributed by Siddarth Seth.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235510
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
> last split generated by FileInputFormat.getSplits may not have the best locality
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3710
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3710
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
> Affects Versions: 0.23.0, 1.0.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Fix For: 0.23.1
>
> Attachments: MR-3710_v1.txt
>
>
> The last split generated by FileInputFormat.getSplits considers {{blkLocations.length-1}} to be the hosts for the split.
> The last split may be larger than the rest (SPLIT_SLOP=1.1 by default) - in which case locality is picked up from a smaller block.
> e.g. 1027MB file with a 128MB split size. The last split ends up being 131MB. The hosts for locality end up being the nodes containing the 3MB block instead of the 128MB block.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3710) last split generated by
FileInputFormat.getSplits may not have the best locality
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192550#comment-13192550 ]
Hudson commented on MAPREDUCE-3710:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1652 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1652/])
MAPREDUCE-3710. Improved FileInputFormat to return better locality for the last split. Contributed by Siddarth Seth.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235510
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
> last split generated by FileInputFormat.getSplits may not have the best locality
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3710
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3710
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
> Affects Versions: 0.23.0, 1.0.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Fix For: 0.23.1
>
> Attachments: MR-3710_v1.txt
>
>
> The last split generated by FileInputFormat.getSplits considers {{blkLocations.length-1}} to be the hosts for the split.
> The last split may be larger than the rest (SPLIT_SLOP=1.1 by default) - in which case locality is picked up from a smaller block.
> e.g. 1027MB file with a 128MB split size. The last split ends up being 131MB. The hosts for locality end up being the nodes containing the 3MB block instead of the 128MB block.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3710) last split generated by
FileInputFormat.getSplits may not have the best locality
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192530#comment-13192530 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3710:
----------------------------------------------------
+1. This looks good.
> last split generated by FileInputFormat.getSplits may not have the best locality
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3710
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3710
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
> Affects Versions: 0.23.0, 1.0.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Attachments: MR-3710_v1.txt
>
>
> The last split generated by FileInputFormat.getSplits considers {{blkLocations.length-1}} to be the hosts for the split.
> The last split may be larger than the rest (SPLIT_SLOP=1.1 by default) - in which case locality is picked up from a smaller block.
> e.g. 1027MB file with a 128MB split size. The last split ends up being 131MB. The hosts for locality end up being the nodes containing the 3MB block instead of the 128MB block.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3710) last split generated by
FileInputFormat.getSplits may not have the best locality
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192571#comment-13192571 ]
Hudson commented on MAPREDUCE-3710:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #429 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/429/])
MAPREDUCE-3710. Improved FileInputFormat to return better locality for the last split. Contributed by Siddarth Seth.
svn merge --ignore-ancestry -c 1235510 ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235511
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
> last split generated by FileInputFormat.getSplits may not have the best locality
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3710
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3710
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
> Affects Versions: 0.23.0, 1.0.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Fix For: 0.23.1
>
> Attachments: MR-3710_v1.txt
>
>
> The last split generated by FileInputFormat.getSplits considers {{blkLocations.length-1}} to be the hosts for the split.
> The last split may be larger than the rest (SPLIT_SLOP=1.1 by default) - in which case locality is picked up from a smaller block.
> e.g. 1027MB file with a 128MB split size. The last split ends up being 131MB. The hosts for locality end up being the nodes containing the 3MB block instead of the 128MB block.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3710) last split generated by
FileInputFormat.getSplits may not have the best locality
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191427#comment-13191427 ]
Hadoop QA commented on MAPREDUCE-3710:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511536/MR-3710_v1.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1652//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1652//console
This message is automatically generated.
> last split generated by FileInputFormat.getSplits may not have the best locality
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3710
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3710
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
> Affects Versions: 0.23.0, 1.0.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Attachments: MR-3710_v1.txt
>
>
> The last split generated by FileInputFormat.getSplits considers {{blkLocations.length-1}} to be the hosts for the split.
> The last split may be larger than the rest (SPLIT_SLOP=1.1 by default) - in which case locality is picked up from a smaller block.
> e.g. 1027MB file with a 128MB split size. The last split ends up being 131MB. The hosts for locality end up being the nodes containing the 3MB block instead of the 128MB block.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3710) last split generated by
FileInputFormat.getSplits may not have the best locality
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193033#comment-13193033 ]
Hudson commented on MAPREDUCE-3710:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #171 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/171/])
MAPREDUCE-3710. Improved FileInputFormat to return better locality for the last split. Contributed by Siddarth Seth.
svn merge --ignore-ancestry -c 1235510 ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235511
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
> last split generated by FileInputFormat.getSplits may not have the best locality
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3710
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3710
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
> Affects Versions: 0.23.0, 1.0.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Fix For: 0.23.1
>
> Attachments: MR-3710_v1.txt
>
>
> The last split generated by FileInputFormat.getSplits considers {{blkLocations.length-1}} to be the hosts for the split.
> The last split may be larger than the rest (SPLIT_SLOP=1.1 by default) - in which case locality is picked up from a smaller block.
> e.g. 1027MB file with a 128MB split size. The last split ends up being 131MB. The hosts for locality end up being the nodes containing the 3MB block instead of the 128MB block.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3710) last split generated by
FileInputFormat.getSplits may not have the best locality
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3710:
-----------------------------------------------
Resolution: Fixed
Fix Version/s: 0.23.1
Release Note: Improved FileInputFormat to return better locality for the last split.
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I just committed this to trunk and branch-0.23. Thanks Sid!
We can merge this into 1.0 on demand.
> last split generated by FileInputFormat.getSplits may not have the best locality
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3710
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3710
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
> Affects Versions: 0.23.0, 1.0.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Fix For: 0.23.1
>
> Attachments: MR-3710_v1.txt
>
>
> The last split generated by FileInputFormat.getSplits considers {{blkLocations.length-1}} to be the hosts for the split.
> The last split may be larger than the rest (SPLIT_SLOP=1.1 by default) - in which case locality is picked up from a smaller block.
> e.g. 1027MB file with a 128MB split size. The last split ends up being 131MB. The hosts for locality end up being the nodes containing the 3MB block instead of the 128MB block.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3710) last split generated by
FileInputFormat.getSplits may not have the best locality
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192437#comment-13192437 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3710:
----------------------------------------------------
Looking through the patch for review/commit.
> last split generated by FileInputFormat.getSplits may not have the best locality
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3710
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3710
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
> Affects Versions: 0.23.0, 1.0.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Attachments: MR-3710_v1.txt
>
>
> The last split generated by FileInputFormat.getSplits considers {{blkLocations.length-1}} to be the hosts for the split.
> The last split may be larger than the rest (SPLIT_SLOP=1.1 by default) - in which case locality is picked up from a smaller block.
> e.g. 1027MB file with a 128MB split size. The last split ends up being 131MB. The hosts for locality end up being the nodes containing the 3MB block instead of the 128MB block.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3710) last split generated by
FileInputFormat.getSplits may not have the best locality
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192551#comment-13192551 ]
Hudson commented on MAPREDUCE-3710:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #404 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/404/])
MAPREDUCE-3710. Improved FileInputFormat to return better locality for the last split. Contributed by Siddarth Seth.
svn merge --ignore-ancestry -c 1235510 ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235511
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
> last split generated by FileInputFormat.getSplits may not have the best locality
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3710
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3710
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
> Affects Versions: 0.23.0, 1.0.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Fix For: 0.23.1
>
> Attachments: MR-3710_v1.txt
>
>
> The last split generated by FileInputFormat.getSplits considers {{blkLocations.length-1}} to be the hosts for the split.
> The last split may be larger than the rest (SPLIT_SLOP=1.1 by default) - in which case locality is picked up from a smaller block.
> e.g. 1027MB file with a 128MB split size. The last split ends up being 131MB. The hosts for locality end up being the nodes containing the 3MB block instead of the 128MB block.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3710) last split generated by
FileInputFormat.getSplits may not have the best locality
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193057#comment-13193057 ]
Hudson commented on MAPREDUCE-3710:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #149 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/149/])
MAPREDUCE-3710. Improved FileInputFormat to return better locality for the last split. Contributed by Siddarth Seth.
svn merge --ignore-ancestry -c 1235510 ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235511
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
> last split generated by FileInputFormat.getSplits may not have the best locality
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3710
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3710
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
> Affects Versions: 0.23.0, 1.0.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Fix For: 0.23.1
>
> Attachments: MR-3710_v1.txt
>
>
> The last split generated by FileInputFormat.getSplits considers {{blkLocations.length-1}} to be the hosts for the split.
> The last split may be larger than the rest (SPLIT_SLOP=1.1 by default) - in which case locality is picked up from a smaller block.
> e.g. 1027MB file with a 128MB split size. The last split ends up being 131MB. The hosts for locality end up being the nodes containing the 3MB block instead of the 128MB block.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3710) last split generated by
FileInputFormat.getSplits may not have the best locality
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193013#comment-13193013 ]
Hudson commented on MAPREDUCE-3710:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #936 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/936/])
MAPREDUCE-3710. Improved FileInputFormat to return better locality for the last split. Contributed by Siddarth Seth.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235510
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
> last split generated by FileInputFormat.getSplits may not have the best locality
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3710
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3710
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
> Affects Versions: 0.23.0, 1.0.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Fix For: 0.23.1
>
> Attachments: MR-3710_v1.txt
>
>
> The last split generated by FileInputFormat.getSplits considers {{blkLocations.length-1}} to be the hosts for the split.
> The last split may be larger than the rest (SPLIT_SLOP=1.1 by default) - in which case locality is picked up from a smaller block.
> e.g. 1027MB file with a 128MB split size. The last split ends up being 131MB. The hosts for locality end up being the nodes containing the 3MB block instead of the 128MB block.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3710) last split generated by
FileInputFormat.getSplits may not have the best locality
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192572#comment-13192572 ]
Hudson commented on MAPREDUCE-3710:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1596 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1596/])
MAPREDUCE-3710. Improved FileInputFormat to return better locality for the last split. Contributed by Siddarth Seth.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235510
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
> last split generated by FileInputFormat.getSplits may not have the best locality
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3710
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3710
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
> Affects Versions: 0.23.0, 1.0.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Fix For: 0.23.1
>
> Attachments: MR-3710_v1.txt
>
>
> The last split generated by FileInputFormat.getSplits considers {{blkLocations.length-1}} to be the hosts for the split.
> The last split may be larger than the rest (SPLIT_SLOP=1.1 by default) - in which case locality is picked up from a smaller block.
> e.g. 1027MB file with a 128MB split size. The last split ends up being 131MB. The hosts for locality end up being the nodes containing the 3MB block instead of the 128MB block.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira