You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by renozhang <gi...@git.apache.org> on 2014/07/24 15:49:09 UTC
[GitHub] spark pull request: [YARN] SPARK-2668: Support log4j log to yarn c...
GitHub user renozhang opened a pull request:
https://github.com/apache/spark/pull/1573
[YARN] SPARK-2668: Support log4j log to yarn container dir
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/renozhang/spark yarn-log-dir
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1573.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1573
----
commit 0b4028a549ee225db4dabf9c1244c3a14d3a77b8
Author: peng.zhang <pe...@xiaomi.com>
Date: 2014-07-24T13:07:21Z
Support log4j log to yarn container dir
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [YARN] SPARK-2668: Add variable of yarn log di...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/1573
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [YARN] SPARK-2668: Add variable of yarn log di...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1573#issuecomment-56496395
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20697/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [YARN] SPARK-2668: Add variable of yarn log di...
Posted by renozhang <gi...@git.apache.org>.
Github user renozhang commented on the pull request:
https://github.com/apache/spark/pull/1573#issuecomment-56129984
Sorry @tgravescs , these days very busy, I'll address them this weekend.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [YARN] SPARK-2668: Add variable of yarn log di...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1573#issuecomment-56496389
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20697/consoleFull) for PR 1573 at commit [`16c5cb8`](https://github.com/apache/spark/commit/16c5cb81ac1c3ef42bc89aabe0546904d31f544e).
* This patch **fails** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [YARN] SPARK-2668: Add variable of yarn log di...
Posted by renozhang <gi...@git.apache.org>.
Github user renozhang commented on the pull request:
https://github.com/apache/spark/pull/1573#issuecomment-55369975
Sorry, I missed writing description in PR. I'll fill description in PR later.
As metioned in descritpion of Jira SPARK-2668:
Adding this varialbe is for user to define custom log4j.properties, eg:
log4j.appender.rolling_file.File = ${spark.yarn.log.dir}/spark.log
发件人: andrewor14 <no...@github.com>>
答复: apache/spark <re...@reply.github.com>>
日期: Thu, 11 Sep 2014 18:08:37 -0700
至: apache/spark <sp...@noreply.github.com>>
抄送: Peng <pe...@xiaomi.com>>
主题: Re: [spark] [YARN] SPARK-2668: Add variable of yarn log directory for reference from the log4j configuration (#1573)
Where is this config being consumed? Am I missing something obvious?
—
Reply to this email directly or view it on GitHub<https://github.com/apache/spark/pull/1573#issuecomment-55350201>.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [YARN] SPARK-2668: Add variable of yarn log di...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1573#issuecomment-56297461
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20619/consoleFull) for PR 1573 at commit [`c56aba6`](https://github.com/apache/spark/commit/c56aba6828843e91fe0208149ae050b3ddbd82bc).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [YARN] SPARK-2668: Add variable of yarn log di...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/1573#issuecomment-54986923
@renozhang sorry for the delay on this, could you upmerge to the latest?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [YARN] SPARK-2668: Support log4j log to yarn c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1573#issuecomment-50026553
QA results for PR 1573:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17119/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [YARN] SPARK-2668: Add variable of yarn log di...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1573#issuecomment-56299060
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20619/consoleFull) for PR 1573 at commit [`c56aba6`](https://github.com/apache/spark/commit/c56aba6828843e91fe0208149ae050b3ddbd82bc).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [YARN] SPARK-2668: Add variable of yarn log di...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/1573#issuecomment-55440722
Yup, thanks for answering my deleted question. I realized this afterwards after reading the JIRA.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [YARN] SPARK-2668: Add variable of yarn log di...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/1573#issuecomment-56521585
+1 thanks @renozhang !
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [YARN] SPARK-2668: Add variable of yarn log di...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/1573#issuecomment-56516302
testfailure is unrelated to this pr
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [YARN] SPARK-2668: Add variable of yarn log di...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1573#issuecomment-56491676
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20697/consoleFull) for PR 1573 at commit [`16c5cb8`](https://github.com/apache/spark/commit/16c5cb81ac1c3ef42bc89aabe0546904d31f544e).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [YARN] SPARK-2668: Add variable of yarn log di...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/1573#discussion_r17422492
--- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala ---
@@ -390,6 +390,9 @@ trait ClientBase extends Logging {
.foreach(p => javaOpts += s"-Djava.library.path=$p")
}
+ // For log4j configuration to reference
+ javaOpts += "-D=spark.yarn.log.dir=" + ApplicationConstants.LOG_DIR_EXPANSION_VAR
--- End diff --
can you rename this to be spark.yarn.app.container.log.dir
Can you also update the documentation in docs/running-on-yarn.md to have this config and a good description.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [YARN] SPARK-2668: Add variable of yarn log di...
Posted by renozhang <gi...@git.apache.org>.
Github user renozhang commented on the pull request:
https://github.com/apache/spark/pull/1573#issuecomment-55215558
@tgravescs I've update to the latest, thanks for review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [YARN] SPARK-2668: Support log4j log to yarn c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1573#issuecomment-50020378
QA tests have started for PR 1573. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17119/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [YARN] SPARK-2668: Add variable of yarn log di...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/1573#issuecomment-56044590
@renozhang can you address my comments
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [YARN] SPARK-2668: Add variable of yarn log di...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/1573#discussion_r17853745
--- Diff: docs/running-on-yarn.md ---
@@ -205,6 +205,8 @@ Note that for the first option, both executors and the application master will s
log4j configuration, which may cause issues when they run on the same node (e.g. trying to write
to the same log file).
+For a streaming application to operate 24/7, user can use RollingFileAppender in log4j.properties to avoid disk overflow of single log file. And to locate log files in YARN container log directory, configure RollingFileAppender's file location as "${spark.yarn.app.container.log.dir}/spark.log". So these log files can be viewed on YARN's container page during running. And logs will be copied to HDFS after job finished if log aggregation is turned on.
--- End diff --
Since this could be used in more then just streaming applications perhaps we should reword this a little. Perhaps put the information about ${spark.yarn.app.container.log.dir} first and then give the example using RollingFileAppender with streaming.
Something more like: (note feel free to change the exact wording)
If you need a reference to the proper location to put the log files in the YARN so that YARN can properly display and aggregate them, use "${spark.yarn.app.container.log.dir}" in your log4j.properties. For example... (then explain the streaming example).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [YARN] SPARK-2668: Add variable of yarn log di...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/1573#issuecomment-55350201
Where is this config being consumed? Am I missing something obvious?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [YARN] SPARK-2668: Add variable of yarn log di...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/1573#issuecomment-56385141
thanks @renozhang, minor request about the documentation, otherwise looks good.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [YARN] SPARK-2668: Add variable of yarn log di...
Posted by renozhang <gi...@git.apache.org>.
Github user renozhang commented on the pull request:
https://github.com/apache/spark/pull/1573#issuecomment-56297532
@tgravescs patch updated, thanks for your review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org