You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2011/02/15 02:41:58 UTC
[jira] Created: (MAPREDUCE-2327) MapTask doesn't need to put
username information in SpillRecord
MapTask doesn't need to put username information in SpillRecord
---------------------------------------------------------------
Key: MAPREDUCE-2327
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2327
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.22.0
Reporter: Todd Lipcon
This is an amendment to MAPREDUCE-2096 that's found in Yahoo's 0.20.100 branch.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2327) MapTask doesn't need to put
username information in SpillRecord
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-2327:
-----------------------------------
Attachment: mapreduce-2327.txt
Sure, here's an updated patch that does that.
> MapTask doesn't need to put username information in SpillRecord
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2327
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2327
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Fix For: 0.22.0
>
> Attachments: mapreduce-2327.txt, mapreduce-2327.txt, mapreduce-2327.txt, mr-2327.txt
>
>
> This is an amendment to MAPREDUCE-2096 that's found in Yahoo's 0.20.100 branch.
> This bug causes task failures in the following case:
> - Cluster is not set up with LinuxTaskController (ie not secured cluster)
> - Job submitter is not the same as the user running the TT
> - Map output is more than one spill's worth
> The issue is that UserGroupInformation's view of the current user is the job submitter, but on disk the spill files will be owned by the TT user. SecureIO will then fail when constructing the spill record.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2327) MapTask doesn't need to put
username information in SpillRecord
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033644#comment-13033644 ]
Tom White commented on MAPREDUCE-2327:
--------------------------------------
+1
> MapTask doesn't need to put username information in SpillRecord
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2327
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2327
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Fix For: 0.22.0
>
> Attachments: mapreduce-2327.txt, mapreduce-2327.txt, mapreduce-2327.txt, mr-2327.txt
>
>
> This is an amendment to MAPREDUCE-2096 that's found in Yahoo's 0.20.100 branch.
> This bug causes task failures in the following case:
> - Cluster is not set up with LinuxTaskController (ie not secured cluster)
> - Job submitter is not the same as the user running the TT
> - Map output is more than one spill's worth
> The issue is that UserGroupInformation's view of the current user is the job submitter, but on disk the spill files will be owned by the TT user. SecureIO will then fail when constructing the spill record.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2327) MapTask doesn't need to put
username information in SpillRecord
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034748#comment-13034748 ]
Hudson commented on MAPREDUCE-2327:
-----------------------------------
Integrated in Hadoop-Mapreduce-22-branch #52 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/52/])
Merge -r 1103989:1103990 from trunk to branch-0.22. Fixes: MAPREDUCE-2327
> MapTask doesn't need to put username information in SpillRecord
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2327
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2327
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Fix For: 0.22.0
>
> Attachments: mapreduce-2327.txt, mapreduce-2327.txt, mapreduce-2327.txt, mr-2327.txt
>
>
> This is an amendment to MAPREDUCE-2096 that's found in Yahoo's 0.20.100 branch.
> This bug causes task failures in the following case:
> - Cluster is not set up with LinuxTaskController (ie not secured cluster)
> - Job submitter is not the same as the user running the TT
> - Map output is more than one spill's worth
> The issue is that UserGroupInformation's view of the current user is the job submitter, but on disk the spill files will be owned by the TT user. SecureIO will then fail when constructing the spill record.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2327) MapTask doesn't need to put
username information in SpillRecord
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034499#comment-13034499 ]
Hadoop QA commented on MAPREDUCE-2327:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12479210/mapreduce-2327.txt
against trunk revision 1103921.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/247//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/247//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/247//console
This message is automatically generated.
> MapTask doesn't need to put username information in SpillRecord
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2327
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2327
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Fix For: 0.22.0
>
> Attachments: mapreduce-2327.txt, mapreduce-2327.txt, mapreduce-2327.txt, mr-2327.txt
>
>
> This is an amendment to MAPREDUCE-2096 that's found in Yahoo's 0.20.100 branch.
> This bug causes task failures in the following case:
> - Cluster is not set up with LinuxTaskController (ie not secured cluster)
> - Job submitter is not the same as the user running the TT
> - Map output is more than one spill's worth
> The issue is that UserGroupInformation's view of the current user is the job submitter, but on disk the spill files will be owned by the TT user. SecureIO will then fail when constructing the spill record.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2327) MapTask doesn't need to put
username information in SpillRecord
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034558#comment-13034558 ]
Hadoop QA commented on MAPREDUCE-2327:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12479210/mapreduce-2327.txt
against trunk revision 1103921.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/254//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/254//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/254//console
This message is automatically generated.
> MapTask doesn't need to put username information in SpillRecord
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2327
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2327
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Fix For: 0.22.0
>
> Attachments: mapreduce-2327.txt, mapreduce-2327.txt, mapreduce-2327.txt, mr-2327.txt
>
>
> This is an amendment to MAPREDUCE-2096 that's found in Yahoo's 0.20.100 branch.
> This bug causes task failures in the following case:
> - Cluster is not set up with LinuxTaskController (ie not secured cluster)
> - Job submitter is not the same as the user running the TT
> - Map output is more than one spill's worth
> The issue is that UserGroupInformation's view of the current user is the job submitter, but on disk the spill files will be owned by the TT user. SecureIO will then fail when constructing the spill record.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2327) MapTask doesn't need to put
username information in SpillRecord
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026139#comment-13026139 ]
Hadoop QA commented on MAPREDUCE-2327:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12477073/mapreduce-2327.txt
against trunk revision 1097305.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/189//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/189//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/189//console
This message is automatically generated.
> MapTask doesn't need to put username information in SpillRecord
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2327
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2327
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Fix For: 0.22.0
>
> Attachments: mapreduce-2327.txt, mapreduce-2327.txt, mr-2327.txt
>
>
> This is an amendment to MAPREDUCE-2096 that's found in Yahoo's 0.20.100 branch.
> This bug causes task failures in the following case:
> - Cluster is not set up with LinuxTaskController (ie not secured cluster)
> - Job submitter is not the same as the user running the TT
> - Map output is more than one spill's worth
> The issue is that UserGroupInformation's view of the current user is the job submitter, but on disk the spill files will be owned by the TT user. SecureIO will then fail when constructing the spill record.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2327) MapTask doesn't need to put
username information in SpillRecord
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026106#comment-13026106 ]
Tom White commented on MAPREDUCE-2327:
--------------------------------------
If security isn't enabled then won't the SecureIOUtils.openForRead() call ignore the username that's passed in?
The new configuration parameter name should begin "mapreduce", not "mapred", for consistency.
> MapTask doesn't need to put username information in SpillRecord
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2327
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2327
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Fix For: 0.22.0
>
> Attachments: mapreduce-2327.txt, mr-2327.txt
>
>
> This is an amendment to MAPREDUCE-2096 that's found in Yahoo's 0.20.100 branch.
> This bug causes task failures in the following case:
> - Cluster is not set up with LinuxTaskController (ie not secured cluster)
> - Job submitter is not the same as the user running the TT
> - Map output is more than one spill's worth
> The issue is that UserGroupInformation's view of the current user is the job submitter, but on disk the spill files will be owned by the TT user. SecureIO will then fail when constructing the spill record.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2327) MapTask doesn't need to put
username information in SpillRecord
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018009#comment-13018009 ]
Nigel Daley commented on MAPREDUCE-2327:
----------------------------------------
Todd, any update on this for 0.22?
> MapTask doesn't need to put username information in SpillRecord
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2327
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2327
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Fix For: 0.22.0
>
> Attachments: mr-2327.txt
>
>
> This is an amendment to MAPREDUCE-2096 that's found in Yahoo's 0.20.100 branch.
> This bug causes task failures in the following case:
> - Cluster is not set up with LinuxTaskController (ie not secured cluster)
> - Job submitter is not the same as the user running the TT
> - Map output is more than one spill's worth
> The issue is that UserGroupInformation's view of the current user is the job submitter, but on disk the spill files will be owned by the TT user. SecureIO will then fail when constructing the spill record.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2327) MapTask doesn't need to put
username information in SpillRecord
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-2327:
-----------------------------------
Attachment: mr-2327.txt
> MapTask doesn't need to put username information in SpillRecord
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2327
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2327
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Attachments: mr-2327.txt
>
>
> This is an amendment to MAPREDUCE-2096 that's found in Yahoo's 0.20.100 branch.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2327) MapTask doesn't need to put
username information in SpillRecord
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999126#comment-12999126 ]
Hadoop QA commented on MAPREDUCE-2327:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12471252/mr-2327.txt
against trunk revision 1074251.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/26//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/26//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/26//console
This message is automatically generated.
> MapTask doesn't need to put username information in SpillRecord
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2327
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2327
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Fix For: 0.22.0
>
> Attachments: mr-2327.txt
>
>
> This is an amendment to MAPREDUCE-2096 that's found in Yahoo's 0.20.100 branch.
> This bug causes task failures in the following case:
> - Cluster is not set up with LinuxTaskController (ie not secured cluster)
> - Job submitter is not the same as the user running the TT
> - Map output is more than one spill's worth
> The issue is that UserGroupInformation's view of the current user is the job submitter, but on disk the spill files will be owned by the TT user. SecureIO will then fail when constructing the spill record.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2327) MapTask doesn't need to put
username information in SpillRecord
Posted by "Raj Vishwanathan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995830#comment-12995830 ]
Raj Vishwanathan commented on MAPREDUCE-2327:
---------------------------------------------
Todd
This looks like the bug I hit. Can I work around it by running the job as the user mapred? From the bug report, that should work.
Raj
> MapTask doesn't need to put username information in SpillRecord
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2327
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2327
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Fix For: 0.22.0
>
> Attachments: mr-2327.txt
>
>
> This is an amendment to MAPREDUCE-2096 that's found in Yahoo's 0.20.100 branch.
> This bug causes task failures in the following case:
> - Cluster is not set up with LinuxTaskController (ie not secured cluster)
> - Job submitter is not the same as the user running the TT
> - Map output is more than one spill's worth
> The issue is that UserGroupInformation's view of the current user is the job submitter, but on disk the spill files will be owned by the TT user. SecureIO will then fail when constructing the spill record.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2327) MapTask doesn't need to put
username information in SpillRecord
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-2327:
-----------------------------------
Status: Open (was: Patch Available)
Hudson is right, I should write a unit test for this.
> MapTask doesn't need to put username information in SpillRecord
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2327
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2327
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Fix For: 0.22.0
>
> Attachments: mr-2327.txt
>
>
> This is an amendment to MAPREDUCE-2096 that's found in Yahoo's 0.20.100 branch.
> This bug causes task failures in the following case:
> - Cluster is not set up with LinuxTaskController (ie not secured cluster)
> - Job submitter is not the same as the user running the TT
> - Map output is more than one spill's worth
> The issue is that UserGroupInformation's view of the current user is the job submitter, but on disk the spill files will be owned by the TT user. SecureIO will then fail when constructing the spill record.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2327) MapTask doesn't need to put
username information in SpillRecord
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034591#comment-13034591 ]
Hudson commented on MAPREDUCE-2327:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #678 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/678/])
MAPREDUCE-2327. MapTask doesn't need to put username information in SpillRecord. Contributed by Todd Lipcon.
> MapTask doesn't need to put username information in SpillRecord
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2327
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2327
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Fix For: 0.22.0
>
> Attachments: mapreduce-2327.txt, mapreduce-2327.txt, mapreduce-2327.txt, mr-2327.txt
>
>
> This is an amendment to MAPREDUCE-2096 that's found in Yahoo's 0.20.100 branch.
> This bug causes task failures in the following case:
> - Cluster is not set up with LinuxTaskController (ie not secured cluster)
> - Job submitter is not the same as the user running the TT
> - Map output is more than one spill's worth
> The issue is that UserGroupInformation's view of the current user is the job submitter, but on disk the spill files will be owned by the TT user. SecureIO will then fail when constructing the spill record.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2327) MapTask doesn't need to put
username information in SpillRecord
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034824#comment-13034824 ]
Hudson commented on MAPREDUCE-2327:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #682 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/682/])
MAPREDUCE-2327. MapTask doesn't need to put username information in SpillRecord. Contributed by Todd Lipcon.
> MapTask doesn't need to put username information in SpillRecord
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2327
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2327
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Fix For: 0.22.0
>
> Attachments: mapreduce-2327.txt, mapreduce-2327.txt, mapreduce-2327.txt, mr-2327.txt
>
>
> This is an amendment to MAPREDUCE-2096 that's found in Yahoo's 0.20.100 branch.
> This bug causes task failures in the following case:
> - Cluster is not set up with LinuxTaskController (ie not secured cluster)
> - Job submitter is not the same as the user running the TT
> - Map output is more than one spill's worth
> The issue is that UserGroupInformation's view of the current user is the job submitter, but on disk the spill files will be owned by the TT user. SecureIO will then fail when constructing the spill record.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2327) MapTask doesn't need to put
username information in SpillRecord
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026679#comment-13026679 ]
Tom White commented on MAPREDUCE-2327:
--------------------------------------
Rather than passing in null, how about adding a new constructor to SpillRecord that doesn't take an expected owner?
> MapTask doesn't need to put username information in SpillRecord
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2327
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2327
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Fix For: 0.22.0
>
> Attachments: mapreduce-2327.txt, mapreduce-2327.txt, mr-2327.txt
>
>
> This is an amendment to MAPREDUCE-2096 that's found in Yahoo's 0.20.100 branch.
> This bug causes task failures in the following case:
> - Cluster is not set up with LinuxTaskController (ie not secured cluster)
> - Job submitter is not the same as the user running the TT
> - Map output is more than one spill's worth
> The issue is that UserGroupInformation's view of the current user is the job submitter, but on disk the spill files will be owned by the TT user. SecureIO will then fail when constructing the spill record.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2327) MapTask doesn't need to put
username information in SpillRecord
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-2327:
-----------------------------------
Status: Patch Available (was: Open)
> MapTask doesn't need to put username information in SpillRecord
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2327
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2327
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Fix For: 0.22.0
>
> Attachments: mr-2327.txt
>
>
> This is an amendment to MAPREDUCE-2096 that's found in Yahoo's 0.20.100 branch.
> This bug causes task failures in the following case:
> - Cluster is not set up with LinuxTaskController (ie not secured cluster)
> - Job submitter is not the same as the user running the TT
> - Map output is more than one spill's worth
> The issue is that UserGroupInformation's view of the current user is the job submitter, but on disk the spill files will be owned by the TT user. SecureIO will then fail when constructing the spill record.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2327) MapTask doesn't need to put
username information in SpillRecord
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-2327:
-----------------------------------
Attachment: mapreduce-2327.txt
Good point on the config name. Fixed.
As for whether this is actually a real bug -- now that HADOOP-7172 has gone in, you're right that this won't cause a problem. But, as a matter of principle, we might as well put the right expected owner in the data structure :)
> MapTask doesn't need to put username information in SpillRecord
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2327
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2327
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Fix For: 0.22.0
>
> Attachments: mapreduce-2327.txt, mapreduce-2327.txt, mr-2327.txt
>
>
> This is an amendment to MAPREDUCE-2096 that's found in Yahoo's 0.20.100 branch.
> This bug causes task failures in the following case:
> - Cluster is not set up with LinuxTaskController (ie not secured cluster)
> - Job submitter is not the same as the user running the TT
> - Map output is more than one spill's worth
> The issue is that UserGroupInformation's view of the current user is the job submitter, but on disk the spill files will be owned by the TT user. SecureIO will then fail when constructing the spill record.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2327) MapTask doesn't need to put
username information in SpillRecord
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-2327:
---------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I've just committed this. Thanks, Todd!
> MapTask doesn't need to put username information in SpillRecord
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2327
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2327
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Fix For: 0.22.0
>
> Attachments: mapreduce-2327.txt, mapreduce-2327.txt, mapreduce-2327.txt, mr-2327.txt
>
>
> This is an amendment to MAPREDUCE-2096 that's found in Yahoo's 0.20.100 branch.
> This bug causes task failures in the following case:
> - Cluster is not set up with LinuxTaskController (ie not secured cluster)
> - Job submitter is not the same as the user running the TT
> - Map output is more than one spill's worth
> The issue is that UserGroupInformation's view of the current user is the job submitter, but on disk the spill files will be owned by the TT user. SecureIO will then fail when constructing the spill record.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2327) MapTask doesn't need to put
username information in SpillRecord
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-2327:
-----------------------------------
Attachment: mapreduce-2327.txt
Here's a patch on top of MAPREDUCE-2445 including a unit test (which fails without this bug fix)
Will mark patch avail after 2445 is in.
> MapTask doesn't need to put username information in SpillRecord
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2327
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2327
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Fix For: 0.22.0
>
> Attachments: mapreduce-2327.txt, mr-2327.txt
>
>
> This is an amendment to MAPREDUCE-2096 that's found in Yahoo's 0.20.100 branch.
> This bug causes task failures in the following case:
> - Cluster is not set up with LinuxTaskController (ie not secured cluster)
> - Job submitter is not the same as the user running the TT
> - Map output is more than one spill's worth
> The issue is that UserGroupInformation's view of the current user is the job submitter, but on disk the spill files will be owned by the TT user. SecureIO will then fail when constructing the spill record.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2327) MapTask doesn't need to put
username information in SpillRecord
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-2327:
-----------------------------------
Status: Patch Available (was: Open)
> MapTask doesn't need to put username information in SpillRecord
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2327
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2327
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Fix For: 0.22.0
>
> Attachments: mapreduce-2327.txt, mr-2327.txt
>
>
> This is an amendment to MAPREDUCE-2096 that's found in Yahoo's 0.20.100 branch.
> This bug causes task failures in the following case:
> - Cluster is not set up with LinuxTaskController (ie not secured cluster)
> - Job submitter is not the same as the user running the TT
> - Map output is more than one spill's worth
> The issue is that UserGroupInformation's view of the current user is the job submitter, but on disk the spill files will be owned by the TT user. SecureIO will then fail when constructing the spill record.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2327) MapTask doesn't need to put
username information in SpillRecord
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-2327:
-----------------------------------
Description:
This is an amendment to MAPREDUCE-2096 that's found in Yahoo's 0.20.100 branch.
This bug causes task failures in the following case:
- Cluster is not set up with LinuxTaskController (ie not secured cluster)
- Job submitter is not the same as the user running the TT
- Map output is more than one spill's worth
The issue is that UserGroupInformation's view of the current user is the job submitter, but on disk the spill files will be owned by the TT user. SecureIO will then fail when constructing the spill record.
was:This is an amendment to MAPREDUCE-2096 that's found in Yahoo's 0.20.100 branch.
Priority: Blocker (was: Major)
Fix Version/s: 0.22.0
Assignee: Todd Lipcon
> MapTask doesn't need to put username information in SpillRecord
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2327
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2327
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Fix For: 0.22.0
>
> Attachments: mr-2327.txt
>
>
> This is an amendment to MAPREDUCE-2096 that's found in Yahoo's 0.20.100 branch.
> This bug causes task failures in the following case:
> - Cluster is not set up with LinuxTaskController (ie not secured cluster)
> - Job submitter is not the same as the user running the TT
> - Map output is more than one spill's worth
> The issue is that UserGroupInformation's view of the current user is the job submitter, but on disk the spill files will be owned by the TT user. SecureIO will then fail when constructing the spill record.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2327) MapTask doesn't need to put
username information in SpillRecord
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026192#comment-13026192 ]
Hadoop QA commented on MAPREDUCE-2327:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12477614/mapreduce-2327.txt
against trunk revision 1097315.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/191//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/191//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/191//console
This message is automatically generated.
> MapTask doesn't need to put username information in SpillRecord
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2327
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2327
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Fix For: 0.22.0
>
> Attachments: mapreduce-2327.txt, mapreduce-2327.txt, mr-2327.txt
>
>
> This is an amendment to MAPREDUCE-2096 that's found in Yahoo's 0.20.100 branch.
> This bug causes task failures in the following case:
> - Cluster is not set up with LinuxTaskController (ie not secured cluster)
> - Job submitter is not the same as the user running the TT
> - Map output is more than one spill's worth
> The issue is that UserGroupInformation's view of the current user is the job submitter, but on disk the spill files will be owned by the TT user. SecureIO will then fail when constructing the spill record.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira