You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Robert Joseph Evans (JIRA)" <ji...@apache.org> on 2011/05/26 22:28:47 UTC
[jira] [Created] (MAPREDUCE-2537) MR-279: The RM writes its log to
yarn-mapred-resourcemanager-.out
MR-279: The RM writes its log to yarn-mapred-resourcemanager-<RM_Host>.out
--------------------------------------------------------------------------
Key: MAPREDUCE-2537
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2537
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
Priority: Minor
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2537) MR-279: The RM writes its log to
yarn-mapred-resourcemanager-.out
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-2537:
-------------------------------------------
Status: Patch Available (was: Open)
No unit tests, because this is a very simple shell script fix.
> MR-279: The RM writes its log to yarn-mapred-resourcemanager-<RM_Host>.out
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-2537
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2537
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Minor
> Attachments: MR-2537-v1.patch
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2537) MR-279: The RM writes its log
to yarn-mapred-resourcemanager-.out
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039953#comment-13039953 ]
Robert Joseph Evans commented on MAPREDUCE-2537:
------------------------------------------------
Output should end in .log
> MR-279: The RM writes its log to yarn-mapred-resourcemanager-<RM_Host>.out
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-2537
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2537
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Minor
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2537) MR-279: The RM writes its log
to yarn-mapred-resourcemanager-.out
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039965#comment-13039965 ]
Hadoop QA commented on MAPREDUCE-2537:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12480589/MR-2537-v1.patch
against trunk revision 1127444.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/311//console
This message is automatically generated.
> MR-279: The RM writes its log to yarn-mapred-resourcemanager-<RM_Host>.out
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-2537
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2537
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Minor
> Attachments: MR-2537-v1.patch
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2537) MR-279: The RM writes its log to
yarn-mapred-resourcemanager-.out
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-2537:
-------------------------------------------
Attachment: MR-2537-v2.patch
OK added in hadoop.root.logger hadoop.log.dir and hadoop.log.file to the -D options
> MR-279: The RM writes its log to yarn-mapred-resourcemanager-<RM_Host>.out
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-2537
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2537
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Minor
> Attachments: MR-2537-v1.patch, MR-2537-v2.patch
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2537) MR-279: The RM writes its log to
yarn-mapred-resourcemanager-.out
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-2537:
-------------------------------------------
Attachment: MR-2537-v1.patch
Actually it is set in the shell script yarn/bin/yarn-daemon.sh
> MR-279: The RM writes its log to yarn-mapred-resourcemanager-<RM_Host>.out
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-2537
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2537
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Minor
> Attachments: MR-2537-v1.patch
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2537) MR-279: The RM writes its log
to yarn-mapred-resourcemanager-.out
Posted by "Luke Lu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040302#comment-13040302 ]
Luke Lu commented on MAPREDUCE-2537:
------------------------------------
Exactly. Since the log4j.properties is from hadoop-common, adding yarn vars would not be kosher. I'd set both vars for posterity (yarn itself (non-mapreduce specific stuff) becomes a sub or top-level project).
> MR-279: The RM writes its log to yarn-mapred-resourcemanager-<RM_Host>.out
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-2537
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2537
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Minor
> Attachments: MR-2537-v1.patch
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2537) MR-279: The RM writes its log to
yarn-mapred-resourcemanager-.out
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-2537:
-------------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this. thanks robert!
> MR-279: The RM writes its log to yarn-mapred-resourcemanager-<RM_Host>.out
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-2537
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2537
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MR-2537-v1.patch, MR-2537-v2.patch
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2537) MR-279: The RM writes its log to
yarn-mapred-resourcemanager-.out
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-2537:
-------------------------------------
Fix Version/s: 0.23.0
> MR-279: The RM writes its log to yarn-mapred-resourcemanager-<RM_Host>.out
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-2537
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2537
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MR-2537-v1.patch, MR-2537-v2.patch
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2537) MR-279: The RM writes its log
to yarn-mapred-resourcemanager-.out
Posted by "Luke Lu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039957#comment-13039957 ]
Luke Lu commented on MAPREDUCE-2537:
------------------------------------
I tried to fix this pre yarn move to MR-279 branch. The problem is that we don't set -Dhadoop.root.logger for log4j.properties and it defaults to INFO,console which is redirected to .out.
I think just fixing the scripts to define hadoop.root.logger to as the same as yarn.root.logger and hadoop.log.dir, yarn.log.dir would fix the problem.
> MR-279: The RM writes its log to yarn-mapred-resourcemanager-<RM_Host>.out
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-2537
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2537
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Minor
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2537) MR-279: The RM writes its log to
yarn-mapred-resourcemanager-.out
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-2537:
-------------------------------------------
Status: Open (was: Patch Available)
> MR-279: The RM writes its log to yarn-mapred-resourcemanager-<RM_Host>.out
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-2537
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2537
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Minor
> Attachments: MR-2537-v1.patch, MR-2537-v2.patch
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2537) MR-279: The RM writes its log
to yarn-mapred-resourcemanager-.out
Posted by "Luke Lu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039974#comment-13039974 ]
Luke Lu commented on MAPREDUCE-2537:
------------------------------------
This is actually not the right fix. It basically redirect stdout to .log which is not the right thing to do. Currently in our logs dir, you'll see both .log and .out files. You'll notice that .log rolls daily but .out doesn't.
Although yarn-daemon.sh defines YARN_ROOT_LOGGER as INFO,DRFA it's not being used to define -Dhadoop.root.logger which can then be picked up by log4j.properties.
> MR-279: The RM writes its log to yarn-mapred-resourcemanager-<RM_Host>.out
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-2537
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2537
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Minor
> Attachments: MR-2537-v1.patch
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2537) MR-279: The RM writes its log
to yarn-mapred-resourcemanager-.out
Posted by "Luke Lu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040340#comment-13040340 ]
Luke Lu commented on MAPREDUCE-2537:
------------------------------------
+1, lgtm, assuming you have tested on at least a single node setup.
> MR-279: The RM writes its log to yarn-mapred-resourcemanager-<RM_Host>.out
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-2537
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2537
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Minor
> Attachments: MR-2537-v1.patch, MR-2537-v2.patch
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2537) MR-279: The RM writes its log
to yarn-mapred-resourcemanager-.out
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040227#comment-13040227 ]
Robert Joseph Evans commented on MAPREDUCE-2537:
------------------------------------------------
OK I understand now. The original description of the issue by the person who found it was very vague.
The issue is that yarn-$YARN_IDENT_STRING-$command-$HOSTNAME.out is where stdout of the command is placed, and that even though yarn.log.dir, yarn.log.file, and yarn.root.logger system properties are being set they are not being used by the log4j configuration. The log4j configuration defaults to CONSOLE which outputs the logs to stdout which in turn goes to yarn-$YARN_IDENT_STRING-$command-$HOSTNAME.out.
The fix is to either update the log4j conf to use the yarn variants of these variables, or to set the hadoop equivalents of these variables instead. I assume from your comment that we need to switch them over to use the hadoop variants. Is this correct?
> MR-279: The RM writes its log to yarn-mapred-resourcemanager-<RM_Host>.out
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-2537
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2537
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Minor
> Attachments: MR-2537-v1.patch
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2537) MR-279: The RM writes its log
to yarn-mapred-resourcemanager-.out
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040345#comment-13040345 ]
Robert Joseph Evans commented on MAPREDUCE-2537:
------------------------------------------------
Yes. I installed it on a single node cluster. I launched both the nodemanager and the resource manager, and verified that the results were being logged to the proper files.
> MR-279: The RM writes its log to yarn-mapred-resourcemanager-<RM_Host>.out
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-2537
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2537
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Minor
> Attachments: MR-2537-v1.patch, MR-2537-v2.patch
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2537) MR-279: The RM writes its log
to yarn-mapred-resourcemanager-.out
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040341#comment-13040341 ]
Hadoop QA commented on MAPREDUCE-2537:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12480673/MR-2537-v2.patch
against trunk revision 1128394.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/314//console
This message is automatically generated.
> MR-279: The RM writes its log to yarn-mapred-resourcemanager-<RM_Host>.out
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-2537
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2537
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Minor
> Attachments: MR-2537-v1.patch, MR-2537-v2.patch
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2537) MR-279: The RM writes its log to
yarn-mapred-resourcemanager-.out
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-2537:
-------------------------------------------
Status: Patch Available (was: Open)
> MR-279: The RM writes its log to yarn-mapred-resourcemanager-<RM_Host>.out
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-2537
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2537
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Minor
> Attachments: MR-2537-v1.patch, MR-2537-v2.patch
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira