You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org> on 2010/08/20 06:22:16 UTC
[jira] Created: (MAPREDUCE-2021) CombineFileInputFormat returns
duplicate hostnames in split locations
CombineFileInputFormat returns duplicate hostnames in split locations
----------------------------------------------------------------------
Key: MAPREDUCE-2021
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2021
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 0.20.2
Reporter: Amareshwari Sriramadasu
Assignee: Amareshwari Sriramadasu
Fix For: 0.22.0
CombineFileInputFormat.getSplits creates splits with duplicate locations. It adds locations of the files in the split to an ArrayList; if all the files are on same location, the location is added again and again. Instead, it should add it to a Set instead of List to avoid duplicates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2021) CombineFileInputFormat returns
duplicate hostnames in split locations
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12901301#action_12901301 ]
Amareshwari Sriramadasu commented on MAPREDUCE-2021:
----------------------------------------------------
All the core and contrib tests passed with the patch.
> CombineFileInputFormat returns duplicate hostnames in split locations
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-2021
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2021
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 0.20.2
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.22.0
>
> Attachments: patch-2021.txt
>
>
> CombineFileInputFormat.getSplits creates splits with duplicate locations. It adds locations of the files in the split to an ArrayList; if all the files are on same location, the location is added again and again. Instead, it should add it to a Set instead of List to avoid duplicates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2021) CombineFileInputFormat returns
duplicate hostnames in split locations
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926078#action_12926078 ]
Hudson commented on MAPREDUCE-2021:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/])
> CombineFileInputFormat returns duplicate hostnames in split locations
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-2021
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2021
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 0.20.2
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.22.0
>
> Attachments: patch-2021-1.txt, patch-2021-ydist.txt, patch-2021-ydist.txt, patch-2021.txt
>
>
> CombineFileInputFormat.getSplits creates splits with duplicate locations. It adds locations of the files in the split to an ArrayList; if all the files are on same location, the location is added again and again. Instead, it should add it to a Set instead of List to avoid duplicates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2021) CombineFileInputFormat returns
duplicate hostnames in split locations
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated MAPREDUCE-2021:
-----------------------------------------------
Status: Patch Available (was: Open)
test-patch result:
{noformat}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
[exec]
{noformat}
> CombineFileInputFormat returns duplicate hostnames in split locations
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-2021
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2021
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 0.20.2
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.22.0
>
> Attachments: patch-2021.txt
>
>
> CombineFileInputFormat.getSplits creates splits with duplicate locations. It adds locations of the files in the split to an ArrayList; if all the files are on same location, the location is added again and again. Instead, it should add it to a Set instead of List to avoid duplicates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2021) CombineFileInputFormat returns
duplicate hostnames in split locations
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated MAPREDUCE-2021:
-----------------------------------------------
Status: Resolved (was: Patch Available)
Hadoop Flags: [Reviewed]
Resolution: Fixed
Patch has been tested on our cluster. Assuming there are no more comments from Scott, I just committed this.
> CombineFileInputFormat returns duplicate hostnames in split locations
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-2021
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2021
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 0.20.2
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.22.0
>
> Attachments: patch-2021-1.txt, patch-2021-ydist.txt, patch-2021.txt
>
>
> CombineFileInputFormat.getSplits creates splits with duplicate locations. It adds locations of the files in the split to an ArrayList; if all the files are on same location, the location is added again and again. Instead, it should add it to a Set instead of List to avoid duplicates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2021) CombineFileInputFormat returns
duplicate hostnames in split locations
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated MAPREDUCE-2021:
-----------------------------------------------
Attachment: patch-2021-ydist.txt
Patch for Yahoo! distribution. The patch is on top of HADOOP-5759.
> CombineFileInputFormat returns duplicate hostnames in split locations
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-2021
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2021
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 0.20.2
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.22.0
>
> Attachments: patch-2021-ydist.txt, patch-2021.txt
>
>
> CombineFileInputFormat.getSplits creates splits with duplicate locations. It adds locations of the files in the split to an ArrayList; if all the files are on same location, the location is added again and again. Instead, it should add it to a Set instead of List to avoid duplicates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2021) CombineFileInputFormat returns
duplicate hostnames in split locations
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated MAPREDUCE-2021:
-----------------------------------------------
Attachment: patch-2021.txt
Patch with the fix and regression test.
> CombineFileInputFormat returns duplicate hostnames in split locations
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-2021
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2021
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 0.20.2
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.22.0
>
> Attachments: patch-2021.txt
>
>
> CombineFileInputFormat.getSplits creates splits with duplicate locations. It adds locations of the files in the split to an ArrayList; if all the files are on same location, the location is added again and again. Instead, it should add it to a Set instead of List to avoid duplicates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2021) CombineFileInputFormat returns
duplicate hostnames in split locations
Posted by "Greg Roelofs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900839#action_12900839 ]
Greg Roelofs commented on MAPREDUCE-2021:
-----------------------------------------
Note the similarity to MAPREDUCE-1974. (Not the same issue, though.)
> CombineFileInputFormat returns duplicate hostnames in split locations
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-2021
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2021
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 0.20.2
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.22.0
>
> Attachments: patch-2021.txt
>
>
> CombineFileInputFormat.getSplits creates splits with duplicate locations. It adds locations of the files in the split to an ArrayList; if all the files are on same location, the location is added again and again. Instead, it should add it to a Set instead of List to avoid duplicates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2021) CombineFileInputFormat returns
duplicate hostnames in split locations
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12906430#action_12906430 ]
Scott Chen commented on MAPREDUCE-2021:
---------------------------------------
Sorry for the late reply. I just got back from China for the Hadoop China conference.
The patch looks good. Thanks for the fix!
> CombineFileInputFormat returns duplicate hostnames in split locations
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-2021
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2021
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 0.20.2
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.22.0
>
> Attachments: patch-2021-1.txt, patch-2021-ydist.txt, patch-2021.txt
>
>
> CombineFileInputFormat.getSplits creates splits with duplicate locations. It adds locations of the files in the split to an ArrayList; if all the files are on same location, the location is added again and again. Instead, it should add it to a Set instead of List to avoid duplicates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2021) CombineFileInputFormat returns
duplicate hostnames in split locations
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated MAPREDUCE-2021:
-----------------------------------------------
Attachment: patch-2021-1.txt
Patch changes the comments for testSplitPlacement().
test-patch and ant test ran successfully. All core and contrib tests passed except TestTaskLauncher and TestTaskTrackerLocalization (due to MAPREDUCE-2031) and TestJobOutputCommitter(MAPREDUCE-2032).
> CombineFileInputFormat returns duplicate hostnames in split locations
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-2021
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2021
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 0.20.2
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.22.0
>
> Attachments: patch-2021-1.txt, patch-2021-ydist.txt, patch-2021.txt
>
>
> CombineFileInputFormat.getSplits creates splits with duplicate locations. It adds locations of the files in the split to an ArrayList; if all the files are on same location, the location is added again and again. Instead, it should add it to a Set instead of List to avoid duplicates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2021) CombineFileInputFormat returns
duplicate hostnames in split locations
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904939#action_12904939 ]
Amareshwari Sriramadasu commented on MAPREDUCE-2021:
----------------------------------------------------
Scott, Can you have a look at the latest patch? Would like to check this in if there are no further comments. Thanks.
> CombineFileInputFormat returns duplicate hostnames in split locations
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-2021
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2021
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 0.20.2
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.22.0
>
> Attachments: patch-2021-1.txt, patch-2021-ydist.txt, patch-2021.txt
>
>
> CombineFileInputFormat.getSplits creates splits with duplicate locations. It adds locations of the files in the split to an ArrayList; if all the files are on same location, the location is added again and again. Instead, it should add it to a Set instead of List to avoid duplicates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2021) CombineFileInputFormat returns
duplicate hostnames in split locations
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902096#action_12902096 ]
Scott Chen commented on MAPREDUCE-2021:
---------------------------------------
Looks good. We will need this change too.
One nitpick: Can you also change the comments in the beginning of testSplitPlacement() which describe how files are created?
> CombineFileInputFormat returns duplicate hostnames in split locations
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-2021
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2021
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 0.20.2
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.22.0
>
> Attachments: patch-2021-ydist.txt, patch-2021.txt
>
>
> CombineFileInputFormat.getSplits creates splits with duplicate locations. It adds locations of the files in the split to an ArrayList; if all the files are on same location, the location is added again and again. Instead, it should add it to a Set instead of List to avoid duplicates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2021) CombineFileInputFormat returns
duplicate hostnames in split locations
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated MAPREDUCE-2021:
-----------------------------------------------
Attachment: patch-2021-ydist.txt
Patch for Yahoo! distribution with comment for test-case fixed.
> CombineFileInputFormat returns duplicate hostnames in split locations
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-2021
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2021
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 0.20.2
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.22.0
>
> Attachments: patch-2021-1.txt, patch-2021-ydist.txt, patch-2021-ydist.txt, patch-2021.txt
>
>
> CombineFileInputFormat.getSplits creates splits with duplicate locations. It adds locations of the files in the split to an ArrayList; if all the files are on same location, the location is added again and again. Instead, it should add it to a Set instead of List to avoid duplicates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.