You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Josh Wills (JIRA)" <ji...@apache.org> on 2011/07/06 03:46:16 UTC
[jira] [Created] (MAPREDUCE-2644) NodeManager fails to create
containers when NM_LOG_DIR is not explicitly set in the Configuration
NodeManager fails to create containers when NM_LOG_DIR is not explicitly set in the Configuration
-------------------------------------------------------------------------------------------------
Key: MAPREDUCE-2644
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2644
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Josh Wills
Assignee: Josh Wills
If the yarn configuration does not explicitly specify a value for the yarn.server.nodemanager.log.dir property, container allocation will fail on the NodeManager w/an NPE when the LocalDirAllocator goes to create the temp directory. In most of the code, we handle this by defaulting to /tmp/logs, but we cannot do this in the LocalDirAllocator context, so we need to set the default value explicitly in the Configuration.
Marking this as major b/c it's annoying to bump into it when you're getting your first MRv2 cluster up and running. :)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2644) NodeManager fails to create
containers when NM_LOG_DIR is not explicitly set in the Configuration
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Wills updated MAPREDUCE-2644:
----------------------------------
Attachment: (was: MAPREDUCE-2644.patch)
> NodeManager fails to create containers when NM_LOG_DIR is not explicitly set in the Configuration
> -------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2644
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2644
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: MAPREDUCE-2644.patch
>
>
> If the yarn configuration does not explicitly specify a value for the yarn.server.nodemanager.log.dir property, container allocation will fail on the NodeManager w/an NPE when the LocalDirAllocator goes to create the temp directory. In most of the code, we handle this by defaulting to /tmp/logs, but we cannot do this in the LocalDirAllocator context, so we need to set the default value explicitly in the Configuration.
> Marking this as major b/c it's annoying to bump into it when you're getting your first MRv2 cluster up and running. :)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2644) NodeManager fails to create
containers when NM_LOG_DIR is not explicitly set in the Configuration
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Wills updated MAPREDUCE-2644:
----------------------------------
Status: Patch Available (was: Open)
> NodeManager fails to create containers when NM_LOG_DIR is not explicitly set in the Configuration
> -------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2644
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2644
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: MAPREDUCE-2644.patch
>
>
> If the yarn configuration does not explicitly specify a value for the yarn.server.nodemanager.log.dir property, container allocation will fail on the NodeManager w/an NPE when the LocalDirAllocator goes to create the temp directory. In most of the code, we handle this by defaulting to /tmp/logs, but we cannot do this in the LocalDirAllocator context, so we need to set the default value explicitly in the Configuration.
> Marking this as major b/c it's annoying to bump into it when you're getting your first MRv2 cluster up and running. :)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2644) NodeManager fails to create
containers when NM_LOG_DIR is not explicitly set in the Configuration
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060237#comment-13060237 ]
Aaron T. Myers commented on MAPREDUCE-2644:
-------------------------------------------
One option would be to change {{LocalDirAllocator}} to accept a default value for the config key you pass in. Another (probably better) option would be to just add a default value to the {{mapred-default.xml}} file, which gets loaded any time an MR config object is created.
> NodeManager fails to create containers when NM_LOG_DIR is not explicitly set in the Configuration
> -------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2644
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2644
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: MAPREDUCE-2644.patch
>
>
> If the yarn configuration does not explicitly specify a value for the yarn.server.nodemanager.log.dir property, container allocation will fail on the NodeManager w/an NPE when the LocalDirAllocator goes to create the temp directory. In most of the code, we handle this by defaulting to /tmp/logs, but we cannot do this in the LocalDirAllocator context, so we need to set the default value explicitly in the Configuration.
> Marking this as major b/c it's annoying to bump into it when you're getting your first MRv2 cluster up and running. :)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2644) NodeManager fails to create
containers when NM_LOG_DIR is not explicitly set in the Configuration
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Wills updated MAPREDUCE-2644:
----------------------------------
Attachment: MAPREDUCE-2644.patch
Not my prettiest impl, but it's functional. If there's a better place in the code to do this, let me know.
> NodeManager fails to create containers when NM_LOG_DIR is not explicitly set in the Configuration
> -------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2644
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2644
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: MAPREDUCE-2644.patch
>
>
> If the yarn configuration does not explicitly specify a value for the yarn.server.nodemanager.log.dir property, container allocation will fail on the NodeManager w/an NPE when the LocalDirAllocator goes to create the temp directory. In most of the code, we handle this by defaulting to /tmp/logs, but we cannot do this in the LocalDirAllocator context, so we need to set the default value explicitly in the Configuration.
> Marking this as major b/c it's annoying to bump into it when you're getting your first MRv2 cluster up and running. :)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2644) NodeManager fails to create
containers when NM_LOG_DIR is not explicitly set in the Configuration
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060233#comment-13060233 ]
Aaron T. Myers commented on MAPREDUCE-2644:
-------------------------------------------
Hey Josh, the usual thing done in the rest of the Hadoop code when providing a default value for a config is to use the two-argument version of {{Configuration.get(...)}} whose first argument is the config key and whose second argument will be returned as the value if no value for that key is set in the conf object.
> NodeManager fails to create containers when NM_LOG_DIR is not explicitly set in the Configuration
> -------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2644
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2644
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: MAPREDUCE-2644.patch
>
>
> If the yarn configuration does not explicitly specify a value for the yarn.server.nodemanager.log.dir property, container allocation will fail on the NodeManager w/an NPE when the LocalDirAllocator goes to create the temp directory. In most of the code, we handle this by defaulting to /tmp/logs, but we cannot do this in the LocalDirAllocator context, so we need to set the default value explicitly in the Configuration.
> Marking this as major b/c it's annoying to bump into it when you're getting your first MRv2 cluster up and running. :)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2644) NodeManager fails to create
containers when NM_LOG_DIR is not explicitly set in the Configuration
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060236#comment-13060236 ]
Josh Wills commented on MAPREDUCE-2644:
---------------------------------------
Okay Aaron-- then is the right place to fix this the LocalDirAllocator under HDFS? It's the code that is being executed hee that doesn't allow for the two-arg version of Configuration.get(...).
> NodeManager fails to create containers when NM_LOG_DIR is not explicitly set in the Configuration
> -------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2644
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2644
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: MAPREDUCE-2644.patch
>
>
> If the yarn configuration does not explicitly specify a value for the yarn.server.nodemanager.log.dir property, container allocation will fail on the NodeManager w/an NPE when the LocalDirAllocator goes to create the temp directory. In most of the code, we handle this by defaulting to /tmp/logs, but we cannot do this in the LocalDirAllocator context, so we need to set the default value explicitly in the Configuration.
> Marking this as major b/c it's annoying to bump into it when you're getting your first MRv2 cluster up and running. :)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2644) NodeManager fails to create
containers when NM_LOG_DIR is not explicitly set in the Configuration
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Wills updated MAPREDUCE-2644:
----------------------------------
Attachment: MAPREDUCE-2644.patch
Moves the default setting for NM_LOG_DIR to the yarn-default.xml file, as per Vinod's comment.
> NodeManager fails to create containers when NM_LOG_DIR is not explicitly set in the Configuration
> -------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2644
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2644
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: MAPREDUCE-2644.patch
>
>
> If the yarn configuration does not explicitly specify a value for the yarn.server.nodemanager.log.dir property, container allocation will fail on the NodeManager w/an NPE when the LocalDirAllocator goes to create the temp directory. In most of the code, we handle this by defaulting to /tmp/logs, but we cannot do this in the LocalDirAllocator context, so we need to set the default value explicitly in the Configuration.
> Marking this as major b/c it's annoying to bump into it when you're getting your first MRv2 cluster up and running. :)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2644) NodeManager fails to create
containers when NM_LOG_DIR is not explicitly set in the Configuration
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13063971#comment-13063971 ]
Hadoop QA commented on MAPREDUCE-2644:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12485459/MAPREDUCE-2644.patch
against trunk revision 1145424.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/453//console
This message is automatically generated.
> NodeManager fails to create containers when NM_LOG_DIR is not explicitly set in the Configuration
> -------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2644
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2644
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: MAPREDUCE-2644.patch
>
>
> If the yarn configuration does not explicitly specify a value for the yarn.server.nodemanager.log.dir property, container allocation will fail on the NodeManager w/an NPE when the LocalDirAllocator goes to create the temp directory. In most of the code, we handle this by defaulting to /tmp/logs, but we cannot do this in the LocalDirAllocator context, so we need to set the default value explicitly in the Configuration.
> Marking this as major b/c it's annoying to bump into it when you're getting your first MRv2 cluster up and running. :)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2644) NodeManager fails to create
containers when NM_LOG_DIR is not explicitly set in the Configuration
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060299#comment-13060299 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-2644:
----------------------------------------------------
Josh, thanks for looking.
Like Aaron mentioned, the correct fix is to mention it in the default files, but add it in yarn-default.xml instead of mapred-site.xml.
Also just to be sure, you can do the same null-check and set in the base NodeManager class, that will solve it completely.
> NodeManager fails to create containers when NM_LOG_DIR is not explicitly set in the Configuration
> -------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2644
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2644
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: MAPREDUCE-2644.patch
>
>
> If the yarn configuration does not explicitly specify a value for the yarn.server.nodemanager.log.dir property, container allocation will fail on the NodeManager w/an NPE when the LocalDirAllocator goes to create the temp directory. In most of the code, we handle this by defaulting to /tmp/logs, but we cannot do this in the LocalDirAllocator context, so we need to set the default value explicitly in the Configuration.
> Marking this as major b/c it's annoying to bump into it when you're getting your first MRv2 cluster up and running. :)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2644) NodeManager fails to create
containers when NM_LOG_DIR is not explicitly set in the Configuration
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-2644:
-----------------------------------------------
Resolution: Fixed
Fix Version/s: 0.23.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this to MR-279 branch. Thanks Josh!
> NodeManager fails to create containers when NM_LOG_DIR is not explicitly set in the Configuration
> -------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2644
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2644
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Josh Wills
> Assignee: Josh Wills
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2644.patch
>
>
> If the yarn configuration does not explicitly specify a value for the yarn.server.nodemanager.log.dir property, container allocation will fail on the NodeManager w/an NPE when the LocalDirAllocator goes to create the temp directory. In most of the code, we handle this by defaulting to /tmp/logs, but we cannot do this in the LocalDirAllocator context, so we need to set the default value explicitly in the Configuration.
> Marking this as major b/c it's annoying to bump into it when you're getting your first MRv2 cluster up and running. :)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira