You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2013/08/30 22:14:52 UTC
[jira] [Commented] (YARN-1104) NMs to support rolling logs of
stdout & stderr
[ https://issues.apache.org/jira/browse/YARN-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755087#comment-13755087 ]
Steve Loughran commented on YARN-1104:
--------------------------------------
Currently most output setups are being done in the command line itself with a ><LOG-DIR>/out.txt 2><LOG-DIR>/err.txt
we should allow apps to ask for the NM set up the logging, which could then be a direct capture from the process streams and rolling logging into the directories, maybe even allow for other options. Routing it via log4j isn't ideal because it does tend to add extra noise:
{code}
d=0}
2013-08-29 17:07:40,594 [AMRM Callback Handler Thread] DEBUG yarn.appmaster.HoyaAppMaster (HoyaAppMaster.java:getProgress(1280)) - Heartbeat, percentage =0.5
2013-08-29 17:07:40,900 [Thread-12] WARN hoya.exec.RunLongLivedApp (RunLongLivedApp.java:run(351)) - 13/08/29 17:07:40 DEBUG client.ClientScanner: Finished region={ENCODED => 92bbefc5114a871444d6665dbddd7fda, NAME => 'hbase:namesp
2013-08-29 17:07:40,900 [Thread-12] WARN hoya.exec.RunLongLivedApp (RunLongLivedApp.java:run(351)) - ace,,1377821255366.92bbefc5114a871444d6665dbddd7fda.', STARTKEY => '', ENDKEY => ''}
2013-08-29 17:07:41,101 [Thread-12] WARN hoya.exec.RunLongLivedApp (RunLongLivedApp.java:run(351)) - 13/08/29 17:07:40 INFO zookeeper.RecoverableZooKeeper: Node /yarnapps_hoya_stevel_TestClusterFlex2DownTo1/namespace/default alrea
2013-08-29 17:07:41,101 [Thread-12] WARN hoya.exec.RunLongLivedApp (RunLongLivedApp.java:run(351)) - dy exists and this is not a retry
2013-08-29 17:07:41,101 [Thread-12] WARN hoya.exec.RunLongLivedApp (RunLongLivedApp.java:run(351)) - 13/08/29 17:07:41 DEBUG hbase.ZKNamespaceManager: Updating namespace cache from node default with data: \x0A\x07default
2013-08-29 17:07:41,101 [Thread-12] WARN hoya.exec.RunLongLivedApp (RunLongLivedApp.java:run(351)) - 13/08/29 17:07:41 DEBUG hbase.ZKNamespaceManager: Updating namespace cache from node hbase with data: \x0A\x05hbase
2013-08-29 17:07:41,429 [IPC Server handler 1 on 52037] INFO yarn.appmaster.HoyaAppMaster (HoyaAppMaster.java:stopCluster(1306)) - HoyaAppMasterApi.stopCluster()
2013-08-29 17:07:41,429 [IPC Server handler 1 on 52037] INFO yarn.appmaster.HoyaAppMaster (HoyaAppMaster.java:signalAMComplete(709)) - Triggering shutdown of the AM: stopCluster() invoked
2013-08-29 17:07:41,502 [Thread-12] WARN hoya.exec.RunLongLivedApp (RunLongLivedApp.java:run(351)) - 13/08/29 17:07:41 INFO zookeeper.RecoverableZooKeeper: Node /yarnapps_hoya_stevel_TestClusterFlex2DownTo1/namespace/hbase already
2013-08-29 17:07:41,503 [Thread-12] WARN hoya.exec.RunLongLivedApp (RunLongLivedApp.java:run(351)) - exists and this is not a retry
2013-08-29 17:07:41,597 [AMRM Callback Handler Thread] DEBUG yarn.appmaster.HoyaAppMaster (HoyaAppMaster.java:getProgress(1280)) - Heartbeat, percentage =0.5
2013-08-29 17:07:41,904 [Thread-12] WARN hoya.exec.RunLongLivedApp (RunLongLivedApp.java:run(351)) - 13/08/29 17:07:41 DEBUG client.ClientScanner: Finished region={ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', EN
2013-08-29 17:07:41,904 [Thread-12] WARN hoya.exec.RunLongLivedApp (RunLongLivedApp.java:run(351)) - DKEY => ''}
{code}
Perhaps we could have a log4j formatter that does nothing at all -just prints the output direct -then route it to a specific log roll just for the outputs
> NMs to support rolling logs of stdout & stderr
> ----------------------------------------------
>
> Key: YARN-1104
> URL: https://issues.apache.org/jira/browse/YARN-1104
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
> Affects Versions: 2.1.0-beta
> Reporter: Steve Loughran
>
> Currently NMs stream the stdout and stderr streams of a container to a file. For longer lived processes those files need to be rotated so that the log doesn't overflow
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira