You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Karam Singh (JIRA)" <ji...@apache.org> on 2009/08/19 14:14:14 UTC
[jira] Created: (MAPREDUCE-890) Afer 4491, the user who started
mapred system is not able to run job.
Afer 4491, the user who started mapred system is not able to run job.
---------------------------------------------------------------------
Key: MAPREDUCE-890
URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Karam Singh
Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
Directories are created as -:
[dr-xrws--- mapred hadoop ] job_200908190916_0002
mapred is not wrtie under this. Even manually I failed to touch file.
mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-890:
------------------------------------
Resolution: Fixed
Fix Version/s: (was: 0.21.0)
0.22.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Since this was already committed to trunk, I'm going to mark it as resolved purely for accounting purposes. I've also moved the entry from 0.21 to 0.22 in CHANGES.txt. Once the patch for 0.21 is committed, the entry can be moved back to 0.21.
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, MR890.20S.patch, MR890.patch, MR890.v1.1.patch, MR890.v1.2.patch, MR890.v1.patch
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod K V updated MAPREDUCE-890:
--------------------------------
Attachment: MAPREDUCE-890-20090904.txt
Attaching patch which should be applied over the latest patch at MAPREDUCE-856. It follows a slight modification of the second proposal above, which is
bq. Set drwxrws--- on $job_cache and $dist_cache in all cases. This means user's tasks CAN potentially create unwarranted files/dirs in the $job_cache or $dist-cache or delete their own files themselves.
So now, the user-directory taskTracker/$user will have "2570 user-owner task-tracker-group" and all the world under it will have "2570/0770 user-owner task-tracker-group".
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Vinod K V
> Attachments: MAPREDUCE-890-20090904.txt
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12776390#action_12776390 ]
Hadoop QA commented on MAPREDUCE-890:
-------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12419014/MAPREDUCE-890-20090909.txt
against trunk revision 834284.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/133/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/133/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/133/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/133/console
This message is automatically generated.
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Gummadi updated MAPREDUCE-890:
-----------------------------------
Attachment: MR890.v1.patch
Attaching patch with Vinod's 1st 2 review comments incorporated. I don't see value in making source code changes for 3rd comment and discussed it with Vinod offline and MAPREDUCE-1429 will take care of it, if needed.
Please review and provide your comments.
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, MR890.patch, MR890.v1.patch
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841747#action_12841747 ]
Vinod K V commented on MAPREDUCE-890:
-------------------------------------
The tests TestSymLink.testSymLink, TestUlimit, TestIO failed because of the test run by Hudson could not find some of the core class. Wierd. Anyone knows why?
Filed MAPREDUCE-1562 for the failure of TestBadRecords.testBadMapRed.
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, MR890.patch, MR890.v1.1.patch, MR890.v1.patch
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Gummadi updated MAPREDUCE-890:
-----------------------------------
Status: Patch Available (was: Open)
Allowing it to go through Hudson
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, MR890.patch, MR890.v1.1.patch, MR890.v1.2.patch, MR890.v1.patch
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12847729#action_12847729 ]
Hudson commented on MAPREDUCE-890:
----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #285 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/285/])
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, MR890.20S.patch, MR890.patch, MR890.v1.1.patch, MR890.v1.2.patch, MR890.v1.patch
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod K V updated MAPREDUCE-890:
--------------------------------
Component/s: tasktracker
Summary: After HADOOP-4491, the user who started mapred system is not able to run job. (was: Afer 4491, the user who started mapred system is not able to run job.)
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12828564#action_12828564 ]
Ravi Gummadi commented on MAPREDUCE-890:
----------------------------------------
I don't see much value in changing all LinuxTaskController based tests to take default value for taskcontroller-ugi when taskcontroller-path is specified(anyway user has to specify taskcontroller-path for the tests to be run) and taskcontroller-ugi is not specified.
Do you really think it is useful ?
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, MR890.patch
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Gummadi updated MAPREDUCE-890:
-----------------------------------
Attachment: MR890.v1.2.patch
Syncing with the latest trunk(as MAPREDUCE-1435 went in).
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, MR890.patch, MR890.v1.1.patch, MR890.v1.2.patch, MR890.v1.patch
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841102#action_12841102 ]
Vinod K V commented on MAPREDUCE-890:
-------------------------------------
Ravi, can you run tests and test-patch? Hudson is not picking patches up, so..
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, MR890.patch, MR890.v1.1.patch, MR890.v1.patch
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12847788#action_12847788 ]
Hudson commented on MAPREDUCE-890:
----------------------------------
Integrated in Hadoop-Mapreduce-trunk #264 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/264/])
Move from 0.21 to 0.22 in changelog to match commit
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, MR890.20S.patch, MR890.patch, MR890.v1.1.patch, MR890.v1.2.patch, MR890.v1.patch
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod K V updated MAPREDUCE-890:
--------------------------------
Attachment: MAPREDUCE-890-20090909.txt
Updated patch to be applied over the latest patch at MAPREDUCE-856.
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Vinod K V
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod K V updated MAPREDUCE-890:
--------------------------------
Status: Patch Available (was: Open)
MAPREDUCE-1421 is committed. Running this patch through Hudson.
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, MR890.patch, MR890.v1.1.patch, MR890.v1.patch
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Gummadi updated MAPREDUCE-890:
-----------------------------------
Status: Open (was: Patch Available)
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, MR890.patch, MR890.v1.1.patch, MR890.v1.patch
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod K V reassigned MAPREDUCE-890:
-----------------------------------
Assignee: Vinod K V
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Vinod K V
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod K V updated MAPREDUCE-890:
--------------------------------
Status: Patch Available (was: Open)
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12828056#action_12828056 ]
Amareshwari Sriramadasu commented on MAPREDUCE-890:
---------------------------------------------------
why don't we run LinuxTaskController tests as the test-launcher(thereby as TT), if nothing is passed in taskcontroller-ugi ?
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, MR890.patch
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12804375#action_12804375 ]
Ravi Gummadi commented on MAPREDUCE-890:
----------------------------------------
Another solution is to call DefaultTaskController's methods from LinuxTaskContrller's methods if user(i.e. jobOwner) is tt_user. This will keep 700 permissions for userDir, jobcache, distcache dirs and so are secure and are accessible to jobOwner.
As I don't see any issues with either of these 2 solutions, currently I am planning to implement 1st approach(i.e. LinuxTaskController setting 770 permissions for jobcache and distcache dirs if jobOwner is tt_user).
Thoughts ?
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840174#action_12840174 ]
Vinod K V commented on MAPREDUCE-890:
-------------------------------------
Patch looks good to me. +1, pending MAPREDUCE-1421.
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, MR890.patch, MR890.v1.1.patch, MR890.v1.patch
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758191#action_12758191 ]
Vinod K V commented on MAPREDUCE-890:
-------------------------------------
The same (latest) patch works post feature-freeze even with all the patches that went in.
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842597#action_12842597 ]
Vinod K V commented on MAPREDUCE-890:
-------------------------------------
I've just committed this to trunk. Thanks Ravi!
The trunk patch isn't applying to 0.21 branch. I think we'll need a new one, can you work on it?
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, MR890.patch, MR890.v1.1.patch, MR890.v1.2.patch, MR890.v1.patch
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802718#action_12802718 ]
Vinod K V commented on MAPREDUCE-890:
-------------------------------------
The patch above that implements the 2nd proposal needs more changes.
bq. Set drwxrws--- on $job-dir in all cases. This means user's tasks CAN potentially create unwarranted files/dirs in the $job_dir
With this, unwarranted files/dirs with improper permissions in job-dir will prevent TT from recursively deleting all the files - a manifestation of MAPREDUCE-896 in the job-dir. This should also be fixed in this patch itself.
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745783#action_12745783 ]
Vinod K V commented on MAPREDUCE-890:
-------------------------------------
As part of MAPREDUCE-842, job directories are set to have the following permissions:
|| permissions || user ownership || group ownership || file/dir name ||
| dr-xrws--- | $job-owner | $tt_group |job_200908190916_0002|
CASE I: the $job-owner is other than the $tt_user
$tt_user is part of $tt_group, and so can create attempt directories inside the job directory as part of the task-localization because job_directory is group writable.
CASE II: the $job-owner is same as the $tt_user
TT cannot create attempt directories inside the job directory!! Because Linux seems to check uid of the process with the fsuid of the directory and return error if the directory is not user writable!
CASE II is what is causing the current bug.
Alternative solutions at hand:
- Leave the code as is, and live with the fact that user's cannot submit jobs as mapred user
- Set drwxrws--- on $job-dir in all cases. This means user's tasks CAN potentially create unwarranted files/dirs in the $job_dir
- Set drwxrws-- if the $job-owner is same as the $tt_user, set dr-xrws--- otherwise. Handles both the cases, but complicates code very slightly.
Thoughts?
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Gummadi updated MAPREDUCE-890:
-----------------------------------
Release Note: Allows the user who started the mapreduce system(cluster) to run jobs
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, MR890.20S.patch, MR890.patch, MR890.v1.1.patch, MR890.v1.2.patch, MR890.v1.patch
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod K V updated MAPREDUCE-890:
--------------------------------
Release Note: Fixed a bug that failed jobs that are run by the same user who started the mapreduce system(cluster). (was: Allows the user who started the mapreduce system(cluster) to run jobs)
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, MR890.20S.patch, MR890.patch, MR890.v1.1.patch, MR890.v1.2.patch, MR890.v1.patch
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842579#action_12842579 ]
Vinod K V commented on MAPREDUCE-890:
-------------------------------------
For some reason, Hudson's couldn't report which tests failed on its web portal, even though the console output is still there. I've downloaded the console output and saw that all the tests passed except TestTaskTrackerMemoryManager. And this failure is because of the same wierd java.lang.NoClassDefFoundError on some core classes that's been plaguing other test runs, this time on org.apache.hadoop.http.HttpServer$QuotingInputFilter.
I could run TestTaskTrackerMemoryManager successfully. I am going to check this in.
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, MR890.patch, MR890.v1.1.patch, MR890.v1.2.patch, MR890.v1.patch
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839600#action_12839600 ]
Vinod K V commented on MAPREDUCE-890:
-------------------------------------
The patch is not applying over the latest patch at MAPREDUCE-1421 which is a blocker for this JIRA issue. Can you please upload an updated patch?
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, MR890.patch, MR890.v1.patch
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Gummadi updated MAPREDUCE-890:
-----------------------------------
Attachment: MR890.v1.1.patch
Syncing the patch with the current trunk.
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, MR890.patch, MR890.v1.1.patch, MR890.v1.patch
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod K V updated MAPREDUCE-890:
--------------------------------
Priority: Blocker (was: Major)
Fix Version/s: 0.21.0
I think this is a blocker - when running with LinuxTaskController, it prevents job-submission as the user running the mapred daemons.
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745784#action_12745784 ]
Vinod K V commented on MAPREDUCE-890:
-------------------------------------
In other words, the problem is the following:
{code}
$ mkdir testing
$ ls -ld testing/
drwxr-xr-x 2 vinodkv vinodkv 4096 2009-08-21 09:54 testing/
$ touch testing/t1.txt
$ echo $?
0
$ chmod 0570 testing/
$ ls -ld testing/
dr-xrwx--- 2 vinodkv vinodkv 4096 2009-08-21 09:54 testing/
$ touch testing/t2.txt
touch: cannot touch `testing/t2.txt': Permission denied
{code}
I searched and hunted this down in the Linux Kernel filesystem code: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=fs/namei.c;h=f3c5b278895a0d3e0f23fe6fd474e2728a1c6cb6;hb=6c30c53fd5ae6a99a23ad78e90c428d2c8ffb07f . Particularly the generic_permission() function where we check if current_fsuid() == inode->i_uid.
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Vinod K V
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841226#action_12841226 ]
Hadoop QA commented on MAPREDUCE-890:
-------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12437589/MR890.v1.1.patch
against trunk revision 918864.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 24 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/342/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/342/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/342/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/342/console
This message is automatically generated.
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, MR890.patch, MR890.v1.1.patch, MR890.v1.patch
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Gummadi updated MAPREDUCE-890:
-----------------------------------
Attachment: MR890.20S.patch
Attaching patch for earlier version of hadoop(includes the fix of MAPREDUCE-1573 also). Not for commit here.
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, MR890.20S.patch, MR890.patch, MR890.v1.1.patch, MR890.v1.2.patch, MR890.v1.patch
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Gummadi updated MAPREDUCE-890:
-----------------------------------
Assignee: Ravi Gummadi (was: Vinod K V)
Status: Open (was: Patch Available)
Setting drwxrws--- for jobcache and distcache directories allows users to create files in these directories and as these directories won't be cleaned up for each job(will be cleaned up only when TT is restarted), this can cause space issues.
So I propose considering tt_user running MR job as a special case and handle that in a special way. i.e. set drwxrwx--- for distcache and jobcache only for jobs of tt_user. This keeps the security of these dirs for all other users as it is there in current trunk.
Thoughts ?
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12804636#action_12804636 ]
Hemanth Yamijala commented on MAPREDUCE-890:
--------------------------------------------
I would like to give it some more thought. But for now, I think Approach 1 seems to be the right way to go (in spite of the special check). I am assuming the special check is in the task-controller ?
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841839#action_12841839 ]
Hadoop QA commented on MAPREDUCE-890:
-------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12437984/MR890.v1.2.patch
against trunk revision 919335.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 24 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/23/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/23/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/23/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/23/console
This message is automatically generated.
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, MR890.patch, MR890.v1.1.patch, MR890.v1.2.patch, MR890.v1.patch
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12804963#action_12804963 ]
Ravi Gummadi commented on MAPREDUCE-890:
----------------------------------------
Yes Hemanth. Check is in task-controller.
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Gummadi updated MAPREDUCE-890:
-----------------------------------
Attachment: MR890.patch
Attaching patch fixing the issue.
Patch does the following:
(1) When localizing userDir, in intialize_user(), we do "chmod 2770 -R taskTracker/$user" for tt_user(i.e. if jobOwner is same as tt_user) and "chmod 2570 -R taskTracker/$user" otherwise.
(2) Similarly,
(a) "chmod 2770 -R taskTracker/$user/jobcache/$jobid" for tt_user and "chmod 2570 -R taskTracker/$user/jobcache/$jobid" otherwise.
(b) "chmod 2770 -R taskTracker/$user/distcache/<randomdir>" for tt_user and "chmod 2570 -R taskTracker/$user/distcache/<randomdir>" otherwise.
(3) compare_ownership() check was done in secure_path() sothat setting of permissions for distributed-cache files(for which permissions are already set earlier) will not go through chown and chmod again and again. This check is not needed after MAPREDUCE-1186(which is already committed to trunk). So deleted this check from secure_path().
(4) Modified TestLocalizationWithLinuxTaskController sothat validation of permissions of different files/directories will work properly in the cases of (a) jobOwner is tt_user and (b) any other user.
Please review and provide your comments.
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, MR890.patch
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12828053#action_12828053 ]
Ravi Gummadi commented on MAPREDUCE-890:
----------------------------------------
As MAPREDUCE-1429 will take care of running the linux task controller based tests with both different user and tt_user(and also changing the source code of all tests is not maintainable --- as future linux task controller based testcases should also consciously follow this duplication of testcases for both users), I will drop this(Vinod's 3rd comment) from this patch. Incorporating first 2 comments.
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, MR890.patch
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who
started mapred system is not able to run job.
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805856#action_12805856 ]
Vinod K V commented on MAPREDUCE-890:
-------------------------------------
Looked at the patch. Got some comments:
- After patch, at +643, in {{initialize_user()}} function, the patch incorrectly passes 0 to {{secure_path()}}'s shouldCheckOwnership argument.
- TestTrackerDistributedCacheManager* tests will need modifications too, I guess.
- It's already a pain running the task-controller tests. With this patch, the expectation is to run these 5-odd tests with $tt_user and $$non_tt_user, that is two times. I think we should modify the tests themselves so that this double test-run is automatically done.
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Reporter: Karam Singh
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, MR890.patch
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.