You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "lohit vijayarenu (JIRA)" <ji...@apache.org> on 2008/04/05 00:38:25 UTC
[jira] Created: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
----------------------------------------------------------------------
Key: HADOOP-3182
URL: https://issues.apache.org/jira/browse/HADOOP-3182
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Affects Versions: 0.16.2
Reporter: lohit vijayarenu
JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12585876#action_12585876 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-3182:
------------------------------------------------
After changed the job-dir permission to 777. Tried the following:
* Start NN, DN as nn_account (superuser by definition)
* Start JT, TT as jt_account (non-superuser)
* Run wordcount as user_account (non-superuser)
Below is an observation of the files/dir creation and their permission setting.
h4. Step 0: During JobTracker startup
mapred-sys-dir (delete-mkdirs-setPerm as jt_account)
733
h4. Step 1: Client side job submission (JobClient .submitJob)
job-dir (aka submitJobDir or mapred-sys-dir/jobId, mkdirs-setPerm as user_account)
(was 733) 777
job-dir/job.jar (may not exists, create-setPerm by JobClient)
job-dir/job.split (create-setPerm by JobClient)
644
job-dir/job.xml (aka jobconf, create-setPerm by JobClient; Should it be visible in JobTracker webpage??? It is visible now.)
644
Miscellaneous dirs: Current using the default permission, i.e. umask. What is the correct permission for them???
- filesDir = new Path(submitJobDir, "files");
- archivesDir = new Path(submitJobDir, "archives");
- libjarsDir = new Path(submitJobDir, "libjars");
h4. Step 2: JobTracker side job submission (JobTracker.submitJob)
job-output-dir (aka mapred.output.dir)
job-history-dir
What are the correct permissions???
They may be created by anto-mkdir in JobTracker as user_account
- For example the first file created under job-output-dir in wordcount is job-output-dir/_logs/history/hostname_1207356629390_job_200804041750_0001_username_wordcount. Note that job-output-dir/_logs/history is the job-history-dir.
- This file is created as user_account
at org.apache.hadoop.mapred.JobHistory$JobInfo.logSubmitted(JobHistory.java:413)
at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:194)
at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1751)
Task does mkdirs job-output-dir as user_account
at org.apache.hadoop.mapred.Task.moveTaskOutputs(Task.java:594)
at org.apache.hadoop.mapred.Task.saveTaskOutput(Task.java:554)
at org.apache.hadoop.mapred.JobTracker$TaskCommitQueue.run(JobTracker.java:2208)
- In Task.moveTaskOutputs(...), is finalOutputPath a constant? If yes, it should not be computed again and again in Task.moveTaskOutputs(...).
job-temp-dir (aka _temporary, created in JobInProgress.<init> as user_account)
What is the correct permission???
Still need to look at the directories clean-up and the case in LocalJobRunner.
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587288#action_12587288 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-3182:
------------------------------------------------
Thank you for your answers, Amareshwari. Let's work on the miscellaneous dirs in another issue.
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
> Attachments: 3182_20080408.patch, 3182_20080408.patch, 3182_20080408_0.16.patch
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-3182:
----------------------------------
Status: Patch Available (was: Open)
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
> Attachments: 3182_20080408.patch, 3182_20080408.patch, 3182_20080408_0.16.patch, HADOOP-3182_2_20080410.patch, patch-3182.txt
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3182:
-------------------------------------------
Status: Open (was: Patch Available)
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
> Attachments: 3182_20080408.patch, 3182_20080408_0.16.patch
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12586561#action_12586561 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-3182:
------------------------------------------------
Below is the clean-up trace for wordcount :
h4. Step c1: Task.saveTaskOutput
- FileSystem.delete <job-output-dir>/_temporary/_task_200804071355_0001_r_000000_0 by JobTracker as user_account
at org.apache.hadoop.mapred.Task.saveTaskOutput(Task.java:557)
at org.apache.hadoop.mapred.JobTracker$TaskCommitQueue.run(JobTracker.java:2208)
h4. Step c2: JobInProgress.garbageCollect
- FileSystem.delete <job-dir> by JobTracker as user_account
at org.apache.hadoop.mapred.JobInProgress.garbageCollect(JobInProgress.java:1637)
at org.apache.hadoop.mapred.JobInProgress.isJobComplete(JobInProgress.java:1396)
at org.apache.hadoop.mapred.JobInProgress.completedTask(JobInProgress.java:1357)
at org.apache.hadoop.mapred.JobInProgress.updateTaskStatus(JobInProgress.java:565)
at org.apache.hadoop.mapred.JobTracker$TaskCommitQueue.run(JobTracker.java:2270)
<job-dir> is obtained from profile.getJobFile()).getParent()
- FileSystem.delete <job-dir> *again* by JobTracker as user_account
at org.apache.hadoop.mapred.JobInProgress.garbageCollect(JobInProgress.java:1642)
at org.apache.hadoop.mapred.JobInProgress.isJobComplete(JobInProgress.java:1396)
at org.apache.hadoop.mapred.JobInProgress.completedTask(JobInProgress.java:1357)
at org.apache.hadoop.mapred.JobInProgress.updateTaskStatus(JobInProgress.java:565)
at org.apache.hadoop.mapred.JobTracker$TaskCommitQueue.run(JobTracker.java:2270)
<job-dir> is obtained from new Path(conf.getSystemDir(), jobId)
*Question:* Are profile.getJobFile()).getParent() and new Path(conf.getSystemDir(), jobId) supposed to be different?
- FileUtil.fullyDelete <job-output-dir>/_temporary as user_account
at org.apache.hadoop.mapred.JobInProgress.garbageCollect(JobInProgress.java:1650)
at org.apache.hadoop.mapred.JobInProgress.isJobComplete(JobInProgress.java:1396)
at org.apache.hadoop.mapred.JobInProgress.completedTask(JobInProgress.java:1357)
at org.apache.hadoop.mapred.JobInProgress.updateTaskStatus(JobInProgress.java:565)
at org.apache.hadoop.mapred.JobTracker$TaskCommitQueue.run(JobTracker.java:2270)
*Question:* Why is the usage of FileUtil.fullyDelete? Is it the same as FileSystem.delete(path, recursive=true)?
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12585877#action_12585877 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-3182:
------------------------------------------------
Below are some codes found in TaskTracker:
{code}
private void localizeJob(TaskInProgress tip) throws IOException {
...
FileStatus status[] = fileSystem.listStatus(new Path(jobFile).getParent());
...
for(FileStatus stat : status) {
if (stat.getPath().toString().contains("job.xml")) {
...
}
if (stat.getPath().toString().contains("job.jar")) {
...
}
}
...
}
{code}
Why it first gets all FileStatus and then does a linear search for the path containing "job.xml" and "job.jar"? It seems to me that the paths can be constructed directly from <job-dir>/job.xml and <job-dir>/job.jar
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-3182:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this to trunk, branch-0.17 & branch-0.16. Thanks, Nicholas & Amareshwari; also Mahadev for helping to validate this!
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
> Attachments: 3182_20080408.patch, 3182_20080408.patch, 3182_20080408_0.16.patch, HADOOP-3182_2_20080410.patch, HADOOP-3182_2_20080410_0.16.patch, patch-3182.txt
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3182:
-------------------------------------------
Attachment: 3182_20080408.patch
3182_20080408.patch: use 777 for <job-dir>.
Also tried randomwriter, the behavior is similar.
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
> Attachments: 3182_20080408.patch
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3182:
-------------------------------------------
Status: Patch Available (was: Open)
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
> Attachments: 3182_20080408.patch, 3182_20080408.patch, 3182_20080408_0.16.patch
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3182:
-------------------------------------------
Attachment: 3182_20080408_0.16.patch
3182_20080408_0.16.patch: for 0.16
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
> Attachments: 3182_20080408.patch, 3182_20080408_0.16.patch
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587758#action_12587758 ]
Hadoop QA commented on HADOOP-3182:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12379870/HADOOP-3182_2_20080410.patch
against trunk revision 645773.
@author +1. The patch does not contain any @author tags.
tests included +1. The patch appears to include 3 new or modified tests.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2199/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2199/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2199/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2199/console
This message is automatically generated.
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
> Attachments: 3182_20080408.patch, 3182_20080408.patch, 3182_20080408_0.16.patch, HADOOP-3182_2_20080410.patch, HADOOP-3182_2_20080410_0.16.patch, patch-3182.txt
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587019#action_12587019 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-3182:
------------------------------------------------
3182_20080408.patch works for both 0.17 and 0.18
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
> Attachments: 3182_20080408.patch, 3182_20080408_0.16.patch
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587486#action_12587486 ]
Owen O'Malley commented on HADOOP-3182:
---------------------------------------
+1 for the 777 on the directory for now.
In the medium term we need to remove the job client doing anything to the system dir. The client should just submit using rpc and the job tracker should be the one writing it to hdfs.
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
> Attachments: 3182_20080408.patch, 3182_20080408.patch, 3182_20080408_0.16.patch
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3182:
-------------------------------------------
Release Note: Changed <job-dir> from 733 to 777, so that a shared JobTracker can be started by a non-superuser account.
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
> Attachments: 3182_20080408.patch, 3182_20080408.patch, 3182_20080408_0.16.patch, HADOOP-3182_2_20080410.patch, HADOOP-3182_2_20080410_0.16.patch, patch-3182.txt
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3182:
-------------------------------------------
Priority: Blocker (was: Major)
Assignee: Tsz Wo (Nicholas), SZE
Setting submitJobDir to 777 will work for this problem. The permission setting in the map/red files/dirs are not clear at this point. I will write down a specification of the permission requirement.
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nigel Daley updated HADOOP-3182:
--------------------------------
Fix Version/s: 0.17.0
0.16.3
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Fix For: 0.16.3, 0.17.0
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587130#action_12587130 ]
Amareshwari Sriramadasu commented on HADOOP-3182:
-------------------------------------------------
bq. In Task.moveTaskOutputs(...), is finalOutputPath a constant?
No, that is calculated from the parameters to moveTaskOutputs() and is called recursively.
bq. Miscellaneous dirs: Current using the default permission, i.e. umask. What is the correct permission for them???
The miscellaneous directories (submitJobdir/files, submitJobdir/archives, submitJobdir/libjars) should have permissions same as submitJobDir. Patch should address this one also.
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
> Attachments: 3182_20080408.patch, 3182_20080408_0.16.patch
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587042#action_12587042 ]
Amareshwari Sriramadasu commented on HADOOP-3182:
-------------------------------------------------
bq. What is the usage of the jobconf property "hadoop.job.history.user.location"? Cannot find any other codes using it or any description.about it.
You can find description in hadoop-default.xml.
{noformat}
<property>
<name>hadoop.job.history.user.location</name>
<value></value>
<description> User can specify a location to store the history files of
a particular job. If nothing is specified, the logs are stored in
output directory. The files are stored in "_logs/history/" in the directory.
User can stop logging by giving the value "none".
</description>
</property>
{noformat}
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
> Attachments: 3182_20080408.patch, 3182_20080408_0.16.patch
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587966#action_12587966 ]
Hudson commented on HADOOP-3182:
--------------------------------
Integrated in Hadoop-trunk #457 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/457/])
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
> Attachments: 3182_20080408.patch, 3182_20080408.patch, 3182_20080408_0.16.patch, HADOOP-3182_2_20080410.patch, HADOOP-3182_2_20080410_0.16.patch, patch-3182.txt
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-3182:
--------------------------------------------
Attachment: patch-3182.txt
Here is patch doing the permission changes for miscellaneous directories
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
> Attachments: 3182_20080408.patch, 3182_20080408.patch, 3182_20080408_0.16.patch, patch-3182.txt
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587500#action_12587500 ]
Arun C Murthy commented on HADOOP-3182:
---------------------------------------
Nicholas, have you tested with -files/-archives and made sure they work? Thanks!
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
> Attachments: 3182_20080408.patch, 3182_20080408.patch, 3182_20080408_0.16.patch, patch-3182.txt
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-3182:
----------------------------------
Attachment: HADOOP-3182_2_20080410.patch
Updated patch ...
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
> Attachments: 3182_20080408.patch, 3182_20080408.patch, 3182_20080408_0.16.patch, HADOOP-3182_2_20080410.patch, patch-3182.txt
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-3182) JobClient creates
submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587096#action_12587096 ]
amareshwari edited comment on HADOOP-3182 at 4/9/08 1:52 AM:
-------------------------------------------------------------------------
job-output-dir, job-history-dir and job-temp-dir can have the permissions as 755.
bq. Are profile.getJobFile()).getParent() and new Path(conf.getSystemDir(), jobId) supposed to be different?
Yes. It looks like they are the same directories. One of the deletes can be removed.
To be specific, fs.delete(new Path(profile.getJobFile()).getParent(), true); can be removed.
bq. What is the usage of FileUtil.fullyDelete? Is it the same as FileSystem.delete(path, recursive=true)?
Yes they are the same.
bq. job-dir/job.xml (aka jobconf, create-setPerm by JobClient; Should it be visible in JobTracker webpage??? It is visible now.)
Yes. It should be visible :)
was (Author: amareshwari):
job-output-dir, job-history-dir and job-temp-dir can have the permissions as 644.
bq. Are profile.getJobFile()).getParent() and new Path(conf.getSystemDir(), jobId) supposed to be different?
Yes. It looks like they are the same directories. One of the deletes can be removed.
To be specific, fs.delete(new Path(profile.getJobFile()).getParent(), true); can be removed.
bq. What is the usage of FileUtil.fullyDelete? Is it the same as FileSystem.delete(path, recursive=true)?
Yes they are the same.
bq. job-dir/job.xml (aka jobconf, create-setPerm by JobClient; Should it be visible in JobTracker webpage??? It is visible now.)
Yes. It should be visible :)
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
> Attachments: 3182_20080408.patch, 3182_20080408_0.16.patch
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12586627#action_12586627 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-3182:
------------------------------------------------
> Why it first gets all FileStatus and then does a linear search for the path containing "job.xml" and "job.jar"? It seems to me that the paths can be constructed directly from <job-dir>/job.xml and <job-dir>/job.jar
See also HADOOP-3203.
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12586952#action_12586952 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-3182:
------------------------------------------------
*Where should <job-history-dir> be?* Found some codes in org.apache.hadoop.mapred.JobHistory.JobInfo:
{code}
public static void logSubmitted(String jobId, JobConf jobConf,
String jobConfPath, long submitTime) {
...
// find user log directory
Path outputPath = FileOutputFormat.getOutputPath(jobConf);
userLogDir = jobConf.get("hadoop.job.history.user.location",
outputPath == null ? null : outputPath.toString());
if ("none".equals(userLogDir)) {
userLogDir = null;
}
if (userLogDir != null) {
userLogDir = userLogDir + "/_logs/history";
}
...
}
{code}
What is the usage of the jobconf property "hadoop.job.history.user.location"?
Cannot find any other codes using it or any description.about it.
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-3182:
----------------------------------
Status: Open (was: Patch Available)
Amareshwari's patch is almost there... we do need to change permissions of the directories inside jobSubmissionDir to be safe.
The only nit is that we need to move {SYSTEM_DIR_PERMISSION} to JobTracker since it's only used there...
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
> Attachments: 3182_20080408.patch, 3182_20080408.patch, 3182_20080408_0.16.patch, HADOOP-3182_2_20080410.patch, patch-3182.txt
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587339#action_12587339 ]
Hadoop QA commented on HADOOP-3182:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12379755/3182_20080408.patch
against trunk revision 645773.
@author +1. The patch does not contain any @author tags.
tests included +1. The patch appears to include 3 new or modified tests.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2190/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2190/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2190/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2190/console
This message is automatically generated.
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
> Attachments: 3182_20080408.patch, 3182_20080408.patch, 3182_20080408_0.16.patch
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587096#action_12587096 ]
Amareshwari Sriramadasu commented on HADOOP-3182:
-------------------------------------------------
job-output-dir, job-history-dir and job-temp-dir can have the permissions as 644.
bq. Are profile.getJobFile()).getParent() and new Path(conf.getSystemDir(), jobId) supposed to be different?
Yes. It looks like they are the same directories. One of the deletes can be removed.
To be specific, fs.delete(new Path(profile.getJobFile()).getParent(), true); can be removed.
bq. What is the usage of FileUtil.fullyDelete? Is it the same as FileSystem.delete(path, recursive=true)?
Yes they are the same.
bq. job-dir/job.xml (aka jobconf, create-setPerm by JobClient; Should it be visible in JobTracker webpage??? It is visible now.)
Yes. It should be visible :)
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
> Attachments: 3182_20080408.patch, 3182_20080408_0.16.patch
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-3182) JobClient creates
submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12586561#action_12586561 ]
szetszwo edited comment on HADOOP-3182 at 4/7/08 3:53 PM:
------------------------------------------------------------------------
Below is the clean-up trace for wordcount :
h4. Step c1: Task.saveTaskOutput
- FileSystem.delete <job-output-dir>/_temporary/_task_200804071355_0001_r_000000_0 by JobTracker as user_account
at org.apache.hadoop.mapred.Task.saveTaskOutput(Task.java:557)
at org.apache.hadoop.mapred.JobTracker$TaskCommitQueue.run(JobTracker.java:2208)
h4. Step c2: JobInProgress.garbageCollect
- FileSystem.delete <job-dir> by JobTracker as user_account
at org.apache.hadoop.mapred.JobInProgress.garbageCollect(JobInProgress.java:1637)
at org.apache.hadoop.mapred.JobInProgress.isJobComplete(JobInProgress.java:1396)
at org.apache.hadoop.mapred.JobInProgress.completedTask(JobInProgress.java:1357)
at org.apache.hadoop.mapred.JobInProgress.updateTaskStatus(JobInProgress.java:565)
at org.apache.hadoop.mapred.JobTracker$TaskCommitQueue.run(JobTracker.java:2270)
<job-dir> is obtained by profile.getJobFile()).getParent()
- FileSystem.delete <job-dir> *again* by JobTracker as user_account
at org.apache.hadoop.mapred.JobInProgress.garbageCollect(JobInProgress.java:1642)
at org.apache.hadoop.mapred.JobInProgress.isJobComplete(JobInProgress.java:1396)
at org.apache.hadoop.mapred.JobInProgress.completedTask(JobInProgress.java:1357)
at org.apache.hadoop.mapred.JobInProgress.updateTaskStatus(JobInProgress.java:565)
at org.apache.hadoop.mapred.JobTracker$TaskCommitQueue.run(JobTracker.java:2270)
<job-dir> is obtained by new Path(conf.getSystemDir(), jobId)
*Question:* Are profile.getJobFile()).getParent() and new Path(conf.getSystemDir(), jobId) supposed to be different?
- FileUtil.fullyDelete <job-output-dir>/_temporary as user_account
at org.apache.hadoop.mapred.JobInProgress.garbageCollect(JobInProgress.java:1650)
at org.apache.hadoop.mapred.JobInProgress.isJobComplete(JobInProgress.java:1396)
at org.apache.hadoop.mapred.JobInProgress.completedTask(JobInProgress.java:1357)
at org.apache.hadoop.mapred.JobInProgress.updateTaskStatus(JobInProgress.java:565)
at org.apache.hadoop.mapred.JobTracker$TaskCommitQueue.run(JobTracker.java:2270)
*Question:* What is the usage of FileUtil.fullyDelete? Is it the same as FileSystem.delete(path, recursive=true)?
was (Author: szetszwo):
Below is the clean-up trace for wordcount :
h4. Step c1: Task.saveTaskOutput
- FileSystem.delete <job-output-dir>/_temporary/_task_200804071355_0001_r_000000_0 by JobTracker as user_account
at org.apache.hadoop.mapred.Task.saveTaskOutput(Task.java:557)
at org.apache.hadoop.mapred.JobTracker$TaskCommitQueue.run(JobTracker.java:2208)
h4. Step c2: JobInProgress.garbageCollect
- FileSystem.delete <job-dir> by JobTracker as user_account
at org.apache.hadoop.mapred.JobInProgress.garbageCollect(JobInProgress.java:1637)
at org.apache.hadoop.mapred.JobInProgress.isJobComplete(JobInProgress.java:1396)
at org.apache.hadoop.mapred.JobInProgress.completedTask(JobInProgress.java:1357)
at org.apache.hadoop.mapred.JobInProgress.updateTaskStatus(JobInProgress.java:565)
at org.apache.hadoop.mapred.JobTracker$TaskCommitQueue.run(JobTracker.java:2270)
<job-dir> is obtained from profile.getJobFile()).getParent()
- FileSystem.delete <job-dir> *again* by JobTracker as user_account
at org.apache.hadoop.mapred.JobInProgress.garbageCollect(JobInProgress.java:1642)
at org.apache.hadoop.mapred.JobInProgress.isJobComplete(JobInProgress.java:1396)
at org.apache.hadoop.mapred.JobInProgress.completedTask(JobInProgress.java:1357)
at org.apache.hadoop.mapred.JobInProgress.updateTaskStatus(JobInProgress.java:565)
at org.apache.hadoop.mapred.JobTracker$TaskCommitQueue.run(JobTracker.java:2270)
<job-dir> is obtained from new Path(conf.getSystemDir(), jobId)
*Question:* Are profile.getJobFile()).getParent() and new Path(conf.getSystemDir(), jobId) supposed to be different?
- FileUtil.fullyDelete <job-output-dir>/_temporary as user_account
at org.apache.hadoop.mapred.JobInProgress.garbageCollect(JobInProgress.java:1650)
at org.apache.hadoop.mapred.JobInProgress.isJobComplete(JobInProgress.java:1396)
at org.apache.hadoop.mapred.JobInProgress.completedTask(JobInProgress.java:1357)
at org.apache.hadoop.mapred.JobInProgress.updateTaskStatus(JobInProgress.java:565)
at org.apache.hadoop.mapred.JobTracker$TaskCommitQueue.run(JobTracker.java:2270)
*Question:* Why is the usage of FileUtil.fullyDelete? Is it the same as FileSystem.delete(path, recursive=true)?
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12585871#action_12585871 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-3182:
------------------------------------------------
> Should the JobClient ask the JobTracker to create this directory for it, rather than creating it itself?
In the current implementation, job-dir is created by user (not JobTracker) and then user put the job related files like jobconf, job.jar in job-dir.
If JobTracker creates and owns this dir, we need to provide some way (may be RPC) for user sending jobconf, job.jar, etc. to JobTracker.
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3182:
-------------------------------------------
Attachment: 3182_20080408.patch
Forgot to do the trick for submitting patch for two branches. Upload the same file again.
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
> Attachments: 3182_20080408.patch, 3182_20080408.patch, 3182_20080408_0.16.patch
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-3182:
--------------------------------------------
Status: Open (was: Patch Available)
I think the same problem will be still there if the users use -files, -libsjars and -archives options
Cancelling patch to make miscellaneous directories have same permissions as submitJobDir.
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
> Attachments: 3182_20080408.patch, 3182_20080408.patch, 3182_20080408_0.16.patch, patch-3182.txt
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3182:
-------------------------------------------
Status: Patch Available (was: Open)
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
> Attachments: 3182_20080408.patch
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587284#action_12587284 ]
Hadoop QA commented on HADOOP-3182:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12379695/3182_20080408_0.16.patch
against trunk revision 645773.
@author +1. The patch does not contain any @author tags.
tests included +1. The patch appears to include 3 new or modified tests.
patch -1. The patch command could not apply the patch.
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2188/console
This message is automatically generated.
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
> Attachments: 3182_20080408.patch, 3182_20080408_0.16.patch
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3182:
-------------------------------------------
Attachment: HADOOP-3182_2_20080410_0.16.patch
HADOOP-3182_2_20080410_0.16.patch: for 0.16
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
> Attachments: 3182_20080408.patch, 3182_20080408.patch, 3182_20080408_0.16.patch, HADOOP-3182_2_20080410.patch, HADOOP-3182_2_20080410_0.16.patch, patch-3182.txt
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-3182:
--------------------------------------------
Status: Patch Available (was: Open)
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
> Attachments: 3182_20080408.patch, 3182_20080408.patch, 3182_20080408_0.16.patch, patch-3182.txt
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12586577#action_12586577 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-3182:
------------------------------------------------
> What is the usage of FileUtil.fullyDelete? Is it the same as FileSystem.delete(path, recursive=true)?
Created HADOOP-3202 for deprecating FileUtil.fullyDelete(FileSystem fs, Path dir).
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3182) JobClient creates submitJobDir with
SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12585838#action_12585838 ]
Doug Cutting commented on HADOOP-3182:
--------------------------------------
Should the JobClient ask the JobTracker to create this directory for it, rather than creating it itself?
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx)
> ----------------------------------------------------------------------
>
> Key: HADOOP-3182
> URL: https://issues.apache.org/jira/browse/HADOOP-3182
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.2
> Reporter: lohit vijayarenu
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.3, 0.17.0
>
>
> JobClient creates submitJobDir with SYSTEM_DIR_PERMISSION ( rwx-wx-wx ) which causes problem while sharing a cluster.
> Consider the case where userA starts jobtracker/tasktrackers and userB submits a job to this cluster. When userB creates submitJobDir it is created with rwx-wx-wx which cannot be read by tasktracker started by userA
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.