You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Hemanth Yamijala (JIRA)" <ji...@apache.org> on 2008/03/03 12:36:50 UTC
[jira] Created: (HADOOP-2925) [HOD] Create mapred system directory
using a naming convention that will avoid clashes in multi-user shared
cluster scenario.
[HOD] Create mapred system directory using a naming convention that will avoid clashes in multi-user shared cluster scenario.
-----------------------------------------------------------------------------------------------------------------------------
Key: HADOOP-2925
URL: https://issues.apache.org/jira/browse/HADOOP-2925
Project: Hadoop Core
Issue Type: Bug
Components: contrib/hod
Affects Versions: 0.16.0
Reporter: Hemanth Yamijala
Assignee: Hemanth Yamijala
Fix For: 0.16.1
Currently, HOD generates the name of the mapredsystem directory using the name /mapredsystem/hostname-of-jobtracker.
In HADOOP-2899, we ran into a scenario where this naming convention could lead to problems in case dfs permissions are enabled. While the bug should ideally be addressed in Hadoop M/R, it will be better that HOD does not generate names that can potentially clash across runs. One way to solve the problem is to do what HOD already does for local log and temp directories - name it using username.torque-job-id, which is going to be pretty unique mostly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2925) [HOD] Create mapred system
directory using a naming convention that will avoid clashes in multi-user
shared cluster scenario.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12575428#action_12575428 ]
Hadoop QA commented on HADOOP-2925:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12377149/2925.2.patch
against trunk revision 619744.
@author +1. The patch does not contain any @author tags.
tests included -1. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1895/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1895/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1895/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1895/console
This message is automatically generated.
> [HOD] Create mapred system directory using a naming convention that will avoid clashes in multi-user shared cluster scenario.
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-2925
> URL: https://issues.apache.org/jira/browse/HADOOP-2925
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Affects Versions: 0.16.0
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Fix For: 0.16.1
>
> Attachments: 2925.1.patch, 2925.2.patch, 2925.patch
>
>
> Currently, HOD generates the name of the mapredsystem directory using the name /mapredsystem/hostname-of-jobtracker.
> In HADOOP-2899, we ran into a scenario where this naming convention could lead to problems in case dfs permissions are enabled. While the bug should ideally be addressed in Hadoop M/R, it will be better that HOD does not generate names that can potentially clash across runs. One way to solve the problem is to do what HOD already does for local log and temp directories - name it using username.torque-job-id, which is going to be pretty unique mostly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2925) [HOD] Create mapred system directory
using a naming convention that will avoid clashes in multi-user shared
cluster scenario.
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-2925:
-------------------------------------
Attachment: 2925.1.patch
This new patch preserves the idea of the earlier patch, but parametrizes the root directory of the mapred system directory. This was earlier hardcoded to /mapredsystem. But it was decided it will be better to make it a config parameter, so administrators have a choice of setting it to the value they want according to security policies.
The current path will be hodring.mapred-system-dir-root/userid/mapredsystem/torque-jobid. Note that setting hodring.mapred-system-dir-root to /user will make these get written to the user's home directory in DFS and thus negate the need for any management in configuring permissions for the system directory.
Another change included in the patch is do a small change to the parameter of the hod logs directory. So, this will now be hodring.log-destination-uri/userid/hod-logs/torque-jobid/.. The change is for the same reason.
> [HOD] Create mapred system directory using a naming convention that will avoid clashes in multi-user shared cluster scenario.
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-2925
> URL: https://issues.apache.org/jira/browse/HADOOP-2925
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Affects Versions: 0.16.0
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Fix For: 0.16.1
>
> Attachments: 2925.1.patch, 2925.patch
>
>
> Currently, HOD generates the name of the mapredsystem directory using the name /mapredsystem/hostname-of-jobtracker.
> In HADOOP-2899, we ran into a scenario where this naming convention could lead to problems in case dfs permissions are enabled. While the bug should ideally be addressed in Hadoop M/R, it will be better that HOD does not generate names that can potentially clash across runs. One way to solve the problem is to do what HOD already does for local log and temp directories - name it using username.torque-job-id, which is going to be pretty unique mostly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2925) [HOD] Create mapred system
directory using a naming convention that will avoid clashes in multi-user
shared cluster scenario.
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12575347#action_12575347 ]
Vinod Kumar Vavilapalli commented on HADOOP-2925:
-------------------------------------------------
+1 Looks good.
> [HOD] Create mapred system directory using a naming convention that will avoid clashes in multi-user shared cluster scenario.
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-2925
> URL: https://issues.apache.org/jira/browse/HADOOP-2925
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Affects Versions: 0.16.0
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Fix For: 0.16.1
>
> Attachments: 2925.1.patch, 2925.2.patch, 2925.patch
>
>
> Currently, HOD generates the name of the mapredsystem directory using the name /mapredsystem/hostname-of-jobtracker.
> In HADOOP-2899, we ran into a scenario where this naming convention could lead to problems in case dfs permissions are enabled. While the bug should ideally be addressed in Hadoop M/R, it will be better that HOD does not generate names that can potentially clash across runs. One way to solve the problem is to do what HOD already does for local log and temp directories - name it using username.torque-job-id, which is going to be pretty unique mostly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2925) [HOD] Create mapred system
directory using a naming convention that will avoid clashes in multi-user
shared cluster scenario.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12575669#action_12575669 ]
Hudson commented on HADOOP-2925:
--------------------------------
Integrated in Hadoop-trunk #421 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/421/])
> [HOD] Create mapred system directory using a naming convention that will avoid clashes in multi-user shared cluster scenario.
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-2925
> URL: https://issues.apache.org/jira/browse/HADOOP-2925
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Affects Versions: 0.16.0
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Fix For: 0.16.1
>
> Attachments: 2925.1.patch, 2925.2.patch, 2925.patch
>
>
> Currently, HOD generates the name of the mapredsystem directory using the name /mapredsystem/hostname-of-jobtracker.
> In HADOOP-2899, we ran into a scenario where this naming convention could lead to problems in case dfs permissions are enabled. While the bug should ideally be addressed in Hadoop M/R, it will be better that HOD does not generate names that can potentially clash across runs. One way to solve the problem is to do what HOD already does for local log and temp directories - name it using username.torque-job-id, which is going to be pretty unique mostly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2925) [HOD] Create mapred system directory
using a naming convention that will avoid clashes in multi-user shared
cluster scenario.
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-2925:
-------------------------------------
Attachment: 2925.patch
Attached patch will name mapred system directories using <username>.<torque-jobid> format. This will prevent a clash from occuring in a multi-user shared cluster scenario. The patch also includes some changes that will help HADOOP-2899 in case the JobTracker implements mechanisms to clean up the mapred system directory at shutdown time.
> [HOD] Create mapred system directory using a naming convention that will avoid clashes in multi-user shared cluster scenario.
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-2925
> URL: https://issues.apache.org/jira/browse/HADOOP-2925
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Affects Versions: 0.16.0
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Fix For: 0.16.1
>
> Attachments: 2925.patch
>
>
> Currently, HOD generates the name of the mapredsystem directory using the name /mapredsystem/hostname-of-jobtracker.
> In HADOOP-2899, we ran into a scenario where this naming convention could lead to problems in case dfs permissions are enabled. While the bug should ideally be addressed in Hadoop M/R, it will be better that HOD does not generate names that can potentially clash across runs. One way to solve the problem is to do what HOD already does for local log and temp directories - name it using username.torque-job-id, which is going to be pretty unique mostly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2925) [HOD] Create mapred system directory
using a naming convention that will avoid clashes in multi-user shared
cluster scenario.
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nigel Daley updated HADOOP-2925:
--------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just commited this. Thanks Hemanth!
> [HOD] Create mapred system directory using a naming convention that will avoid clashes in multi-user shared cluster scenario.
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-2925
> URL: https://issues.apache.org/jira/browse/HADOOP-2925
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Affects Versions: 0.16.0
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Fix For: 0.16.1
>
> Attachments: 2925.1.patch, 2925.2.patch, 2925.patch
>
>
> Currently, HOD generates the name of the mapredsystem directory using the name /mapredsystem/hostname-of-jobtracker.
> In HADOOP-2899, we ran into a scenario where this naming convention could lead to problems in case dfs permissions are enabled. While the bug should ideally be addressed in Hadoop M/R, it will be better that HOD does not generate names that can potentially clash across runs. One way to solve the problem is to do what HOD already does for local log and temp directories - name it using username.torque-job-id, which is going to be pretty unique mostly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2925) [HOD] Create mapred system directory
using a naming convention that will avoid clashes in multi-user shared
cluster scenario.
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-2925:
-------------------------------------
Attachment: 2925.2.patch
Added documentation for the new config option mapred-system-dir-root and also on some other related options.
> [HOD] Create mapred system directory using a naming convention that will avoid clashes in multi-user shared cluster scenario.
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-2925
> URL: https://issues.apache.org/jira/browse/HADOOP-2925
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Affects Versions: 0.16.0
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Fix For: 0.16.1
>
> Attachments: 2925.1.patch, 2925.2.patch, 2925.patch
>
>
> Currently, HOD generates the name of the mapredsystem directory using the name /mapredsystem/hostname-of-jobtracker.
> In HADOOP-2899, we ran into a scenario where this naming convention could lead to problems in case dfs permissions are enabled. While the bug should ideally be addressed in Hadoop M/R, it will be better that HOD does not generate names that can potentially clash across runs. One way to solve the problem is to do what HOD already does for local log and temp directories - name it using username.torque-job-id, which is going to be pretty unique mostly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2925) [HOD] Create mapred system directory
using a naming convention that will avoid clashes in multi-user shared
cluster scenario.
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-2925:
-------------------------------------
Status: Patch Available (was: Open)
Run through Hudson
> [HOD] Create mapred system directory using a naming convention that will avoid clashes in multi-user shared cluster scenario.
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-2925
> URL: https://issues.apache.org/jira/browse/HADOOP-2925
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Affects Versions: 0.16.0
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Fix For: 0.16.1
>
> Attachments: 2925.1.patch, 2925.2.patch, 2925.patch
>
>
> Currently, HOD generates the name of the mapredsystem directory using the name /mapredsystem/hostname-of-jobtracker.
> In HADOOP-2899, we ran into a scenario where this naming convention could lead to problems in case dfs permissions are enabled. While the bug should ideally be addressed in Hadoop M/R, it will be better that HOD does not generate names that can potentially clash across runs. One way to solve the problem is to do what HOD already does for local log and temp directories - name it using username.torque-job-id, which is going to be pretty unique mostly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2925) [HOD] Create mapred system
directory using a naming convention that will avoid clashes in multi-user
shared cluster scenario.
Posted by "Luca Telloli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12575058#action_12575058 ]
Luca Telloli commented on HADOOP-2925:
--------------------------------------
I patched the version 0.4.0 of Hadoop 0.16.0 and it worked flawlessly. Directory names do not clash under /mapredsystem which is a very useful step waiting for HADOOP-2899.
Thanks,
Luca
> [HOD] Create mapred system directory using a naming convention that will avoid clashes in multi-user shared cluster scenario.
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-2925
> URL: https://issues.apache.org/jira/browse/HADOOP-2925
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Affects Versions: 0.16.0
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Fix For: 0.16.1
>
> Attachments: 2925.patch
>
>
> Currently, HOD generates the name of the mapredsystem directory using the name /mapredsystem/hostname-of-jobtracker.
> In HADOOP-2899, we ran into a scenario where this naming convention could lead to problems in case dfs permissions are enabled. While the bug should ideally be addressed in Hadoop M/R, it will be better that HOD does not generate names that can potentially clash across runs. One way to solve the problem is to do what HOD already does for local log and temp directories - name it using username.torque-job-id, which is going to be pretty unique mostly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.