You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2010/12/15 23:12:03 UTC
[jira] Created: (MAPREDUCE-2219) JT should not try to remove
mapred.system.dir during startup
JT should not try to remove mapred.system.dir during startup
------------------------------------------------------------
Key: MAPREDUCE-2219
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2219
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 0.22.0
Reporter: Todd Lipcon
During startup, the JT tries to clean up mapred.system.dir by recursively removing it and then recreating it. This requires that mapred.system.dir is inside a directory owned by the mapred user. For example, if set to /system/mapred then /system must be owned by the mapred account. This isn't documented properly and also seems unnecessary. Instead we can remove the *contents* of mapred.system.dir instead of the directory itself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2219) JT should not try to remove
mapred.system.dir during startup
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978002#action_12978002 ]
Tom White commented on MAPREDUCE-2219:
--------------------------------------
This looks like a good change. A couple of comments:
* Can you use FileUtil#fullyDeleteContents() rather than writing a new one?
* The comment in the test for 755 perms should say "rwxr-xr-x" not "rwx-rx-rx".
> JT should not try to remove mapred.system.dir during startup
> ------------------------------------------------------------
>
> Key: MAPREDUCE-2219
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2219
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: mapreduce-2219.txt
>
>
> During startup, the JT tries to clean up mapred.system.dir by recursively removing it and then recreating it. This requires that mapred.system.dir is inside a directory owned by the mapred user. For example, if set to /system/mapred then /system must be owned by the mapred account. This isn't documented properly and also seems unnecessary. Instead we can remove the *contents* of mapred.system.dir instead of the directory itself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2219) JT should not try to remove
mapred.system.dir during startup
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-2219:
-----------------------------------
Resolution: Fixed
Fix Version/s: 0.22.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
> JT should not try to remove mapred.system.dir during startup
> ------------------------------------------------------------
>
> Key: MAPREDUCE-2219
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2219
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.22.0
>
> Attachments: mapreduce-2219.2.txt, mapreduce-2219.txt
>
>
> During startup, the JT tries to clean up mapred.system.dir by recursively removing it and then recreating it. This requires that mapred.system.dir is inside a directory owned by the mapred user. For example, if set to /system/mapred then /system must be owned by the mapred account. This isn't documented properly and also seems unnecessary. Instead we can remove the *contents* of mapred.system.dir instead of the directory itself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-2219) JT should not try to remove
mapred.system.dir during startup
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon reassigned MAPREDUCE-2219:
--------------------------------------
Assignee: Todd Lipcon
> JT should not try to remove mapred.system.dir during startup
> ------------------------------------------------------------
>
> Key: MAPREDUCE-2219
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2219
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: mapreduce-2219.txt
>
>
> During startup, the JT tries to clean up mapred.system.dir by recursively removing it and then recreating it. This requires that mapred.system.dir is inside a directory owned by the mapred user. For example, if set to /system/mapred then /system must be owned by the mapred account. This isn't documented properly and also seems unnecessary. Instead we can remove the *contents* of mapred.system.dir instead of the directory itself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2219) JT should not try to remove
mapred.system.dir during startup
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978424#action_12978424 ]
Tom White commented on MAPREDUCE-2219:
--------------------------------------
> Unfortunately I couldn't use FileUtil.fullyDeleteContents because that method takes File and not Path/FileSystem.
I see. The patch has tests for this code, so +1 (pending Hudson).
> JT should not try to remove mapred.system.dir during startup
> ------------------------------------------------------------
>
> Key: MAPREDUCE-2219
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2219
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: mapreduce-2219.2.txt, mapreduce-2219.txt
>
>
> During startup, the JT tries to clean up mapred.system.dir by recursively removing it and then recreating it. This requires that mapred.system.dir is inside a directory owned by the mapred user. For example, if set to /system/mapred then /system must be owned by the mapred account. This isn't documented properly and also seems unnecessary. Instead we can remove the *contents* of mapred.system.dir instead of the directory itself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2219) JT should not try to remove
mapred.system.dir during startup
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977484#action_12977484 ]
Todd Lipcon commented on MAPREDUCE-2219:
----------------------------------------
https://reviews.apache.org/r/206/
> JT should not try to remove mapred.system.dir during startup
> ------------------------------------------------------------
>
> Key: MAPREDUCE-2219
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2219
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: mapreduce-2219.txt
>
>
> During startup, the JT tries to clean up mapred.system.dir by recursively removing it and then recreating it. This requires that mapred.system.dir is inside a directory owned by the mapred user. For example, if set to /system/mapred then /system must be owned by the mapred account. This isn't documented properly and also seems unnecessary. Instead we can remove the *contents* of mapred.system.dir instead of the directory itself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2219) JT should not try to remove
mapred.system.dir during startup
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-2219:
-----------------------------------
Status: Patch Available (was: Open)
> JT should not try to remove mapred.system.dir during startup
> ------------------------------------------------------------
>
> Key: MAPREDUCE-2219
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2219
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: mapreduce-2219.txt
>
>
> During startup, the JT tries to clean up mapred.system.dir by recursively removing it and then recreating it. This requires that mapred.system.dir is inside a directory owned by the mapred user. For example, if set to /system/mapred then /system must be owned by the mapred account. This isn't documented properly and also seems unnecessary. Instead we can remove the *contents* of mapred.system.dir instead of the directory itself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2219) JT should not try to remove
mapred.system.dir during startup
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-2219:
-----------------------------------
Attachment: mapreduce-2219.txt
> JT should not try to remove mapred.system.dir during startup
> ------------------------------------------------------------
>
> Key: MAPREDUCE-2219
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2219
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Attachments: mapreduce-2219.txt
>
>
> During startup, the JT tries to clean up mapred.system.dir by recursively removing it and then recreating it. This requires that mapred.system.dir is inside a directory owned by the mapred user. For example, if set to /system/mapred then /system must be owned by the mapred account. This isn't documented properly and also seems unnecessary. Instead we can remove the *contents* of mapred.system.dir instead of the directory itself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2219) JT should not try to remove
mapred.system.dir during startup
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-2219:
-----------------------------------
Attachment: mapreduce-2219.2.txt
This patch fixes the permissions string in the comment, nice catch.
Unfortunately I couldn't use FileUtil.fullyDeleteContents because that method takes File and not Path/FileSystem. Rather than filing a second JIRA to add that method to common, I just implemented it inline since it's pretty straightforward.
> JT should not try to remove mapred.system.dir during startup
> ------------------------------------------------------------
>
> Key: MAPREDUCE-2219
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2219
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: mapreduce-2219.2.txt, mapreduce-2219.txt
>
>
> During startup, the JT tries to clean up mapred.system.dir by recursively removing it and then recreating it. This requires that mapred.system.dir is inside a directory owned by the mapred user. For example, if set to /system/mapred then /system must be owned by the mapred account. This isn't documented properly and also seems unnecessary. Instead we can remove the *contents* of mapred.system.dir instead of the directory itself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2219) JT should not try to remove
mapred.system.dir during startup
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978614#action_12978614 ]
Todd Lipcon commented on MAPREDUCE-2219:
----------------------------------------
test-patch and unit tests both passed (except for known timeouts). Will commit to trunk and 0.22 momentarily.
> JT should not try to remove mapred.system.dir during startup
> ------------------------------------------------------------
>
> Key: MAPREDUCE-2219
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2219
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: mapreduce-2219.2.txt, mapreduce-2219.txt
>
>
> During startup, the JT tries to clean up mapred.system.dir by recursively removing it and then recreating it. This requires that mapred.system.dir is inside a directory owned by the mapred user. For example, if set to /system/mapred then /system must be owned by the mapred account. This isn't documented properly and also seems unnecessary. Instead we can remove the *contents* of mapred.system.dir instead of the directory itself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2219) JT should not try to remove
mapred.system.dir during startup
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12993931#comment-12993931 ]
Hudson commented on MAPREDUCE-2219:
-----------------------------------
Integrated in Hadoop-Mapreduce-22-branch #33 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/33/])
> JT should not try to remove mapred.system.dir during startup
> ------------------------------------------------------------
>
> Key: MAPREDUCE-2219
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2219
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.22.0
>
> Attachments: mapreduce-2219.2.txt, mapreduce-2219.txt
>
>
> During startup, the JT tries to clean up mapred.system.dir by recursively removing it and then recreating it. This requires that mapred.system.dir is inside a directory owned by the mapred user. For example, if set to /system/mapred then /system must be owned by the mapred account. This isn't documented properly and also seems unnecessary. Instead we can remove the *contents* of mapred.system.dir instead of the directory itself.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira