You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by liyezhang556520 <gi...@git.apache.org> on 2014/12/29 09:23:25 UTC
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
GitHub user liyezhang556520 opened a pull request:
https://github.com/apache/spark/pull/3824
[SPARK-4989][CORE] avoid wrong eventlog conf cause cluster down in standalone mode
when enabling eventlog in standalone mode, if give the wrong configuration, the standalone cluster will down (cause master restart, lose connection with workers).
How to reproduce: just give an invalid value to "spark.eventLog.dir", for example: spark.eventLog.dir=hdfs://tmp/logdir1, hdfs://tmp/logdir2. This will throw illegalArgumentException, which will cause the Master restart. And the whole cluster is not available.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/liyezhang556520/spark wrongConf4Cluster
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3824.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3824
----
commit 5c1fa33799bc503ac1e2d5e9838e8e364bf1f61f
Author: Zhang, Liye <li...@intel.com>
Date: 2014-12-26T08:23:53Z
cache exceptions when eventlog with wrong conf
commit 12eee8590fb9899c267b29d3a129a169b6cf6ec1
Author: Zhang, Liye <li...@intel.com>
Date: 2014-12-26T08:49:04Z
add more message in log and on webUI
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/3824
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3824#discussion_r22323425
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala ---
@@ -758,13 +760,14 @@ private[spark] class Master(
// Event logging is enabled for this application, but no event logs are found
val title = s"Application history not found (${app.id})"
var msg = s"No event logs found for application $appName in $eventLogFile."
- logWarning(msg)
+ val exception = URLEncoder.encode(Utils.exceptionString(fnf), "UTF-8")
+ logWarning(msg, fnf)
msg += " Did you specify the correct logging directory?"
msg = URLEncoder.encode(msg, "UTF-8")
- app.desc.appUiUrl = notFoundBasePath + s"?msg=$msg&title=$title"
+ app.desc.appUiUrl = notFoundBasePath + s"?msg=$msg&exception=$exception&title=$title"
--- End diff --
@srowen It looks like this same `exception` URL param is used in other exception-handling code in this same file (the first instance was added by @andrewor14 in 6afca2d1079bac6309a595b8e0ffc74ae93fa662).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3824#issuecomment-68622521
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25025/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3824#issuecomment-69137451
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25199/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3824#issuecomment-68244528
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24858/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3824#issuecomment-69137447
[Test build #25199 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25199/consoleFull) for PR 3824 at commit [`3c24d98`](https://github.com/apache/spark/commit/3c24d981f7f28b3ce8f56e57c14b48458eb378b2).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3824#issuecomment-68287342
This change seems okay to me overall, aside from one minor nit. Most of the change is just broadening the scope of the `try` block to handle some cases that didn't seem like they could fail.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3824#discussion_r22626035
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala ---
@@ -758,13 +760,14 @@ private[spark] class Master(
// Event logging is enabled for this application, but no event logs are found
val title = s"Application history not found (${app.id})"
var msg = s"No event logs found for application $appName in $eventLogFile."
- logWarning(msg)
+ val exception = URLEncoder.encode(Utils.exceptionString(fnf), "UTF-8")
+ logWarning(msg, fnf)
msg += " Did you specify the correct logging directory?"
msg = URLEncoder.encode(msg, "UTF-8")
- app.desc.appUiUrl = notFoundBasePath + s"?msg=$msg&title=$title"
+ app.desc.appUiUrl = notFoundBasePath + s"?msg=$msg&exception=$exception&title=$title"
--- End diff --
I think it's OK to not encode the exception here. This is a file not found exception so it doesn't convey much extra information.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3824#issuecomment-68239849
[Test build #24858 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24858/consoleFull) for PR 3824 at commit [`12eee85`](https://github.com/apache/spark/commit/12eee8590fb9899c267b29d3a129a169b6cf6ec1).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Posted by liyezhang556520 <gi...@git.apache.org>.
Github user liyezhang556520 commented on the pull request:
https://github.com/apache/spark/pull/3824#issuecomment-69283439
ok, I'll make new PRs for those old branches 1.0, 1.1, and 1.2.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Posted by liyezhang556520 <gi...@git.apache.org>.
Github user liyezhang556520 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3824#discussion_r22306099
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala ---
@@ -758,13 +760,14 @@ private[spark] class Master(
// Event logging is enabled for this application, but no event logs are found
val title = s"Application history not found (${app.id})"
var msg = s"No event logs found for application $appName in $eventLogFile."
- logWarning(msg)
+ val exception = URLEncoder.encode(Utils.exceptionString(fnf), "UTF-8")
+ logWarning(msg, fnf)
msg += " Did you specify the correct logging directory?"
msg = URLEncoder.encode(msg, "UTF-8")
- app.desc.appUiUrl = notFoundBasePath + s"?msg=$msg&title=$title"
+ app.desc.appUiUrl = notFoundBasePath + s"?msg=$msg&exception=$exception&title=$title"
false
case e: Exception =>
- // Relay exception message to application UI page
+ // Replay exception message to application UI page
--- End diff --
Yes, you are write, relay is correct, replay is not correct, thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3824#issuecomment-68620916
[Test build #25025 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25025/consoleFull) for PR 3824 at commit [`3c1ac2e`](https://github.com/apache/spark/commit/3c1ac2ef5ea608e00eb45085ee65ae950c21be1b).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3824#discussion_r22323517
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala ---
@@ -719,26 +719,28 @@ private[spark] class Master(
def rebuildSparkUI(app: ApplicationInfo): Boolean = {
val appName = app.desc.name
val notFoundBasePath = HistoryServer.UI_PATH_PREFIX + "/not-found"
- val eventLogFile = app.desc.eventLogDir
- .map { dir => EventLoggingListener.getLogPath(dir, app.id) }
- .getOrElse {
- // Event logging is not enabled for this application
- app.desc.appUiUrl = notFoundBasePath
- return false
- }
- val fs = Utils.getHadoopFileSystem(eventLogFile, hadoopConf)
+ var eventLogFile: String = null
--- End diff --
It looks like `eventLogFile` is only read from inside the `try` block on the following line, so why not move it inside and make it into a `val` instead?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Posted by liyezhang556520 <gi...@git.apache.org>.
Github user liyezhang556520 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3824#discussion_r22306172
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala ---
@@ -758,13 +760,14 @@ private[spark] class Master(
// Event logging is enabled for this application, but no event logs are found
val title = s"Application history not found (${app.id})"
var msg = s"No event logs found for application $appName in $eventLogFile."
- logWarning(msg)
+ val exception = URLEncoder.encode(Utils.exceptionString(fnf), "UTF-8")
+ logWarning(msg, fnf)
msg += " Did you specify the correct logging directory?"
msg = URLEncoder.encode(msg, "UTF-8")
- app.desc.appUiUrl = notFoundBasePath + s"?msg=$msg&title=$title"
+ app.desc.appUiUrl = notFoundBasePath + s"?msg=$msg&exception=$exception&title=$title"
--- End diff --
No
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/3824#issuecomment-69226708
Ok LGTM I'm merging this into master thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3824#discussion_r22626094
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala ---
@@ -757,11 +758,12 @@ private[spark] class Master(
case fnf: FileNotFoundException =>
// Event logging is enabled for this application, but no event logs are found
val title = s"Application history not found (${app.id})"
- var msg = s"No event logs found for application $appName in $eventLogFile."
- logWarning(msg)
+ var msg = s"No event logs found for application $appName in ${app.desc.eventLogDir}."
+ val exception = URLEncoder.encode(Utils.exceptionString(fnf), "UTF-8")
+ logWarning(msg, fnf)
--- End diff --
as commented elsewhere, I don't think logging the stack trace here is particularly useful
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3824#discussion_r22306027
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala ---
@@ -758,13 +760,14 @@ private[spark] class Master(
// Event logging is enabled for this application, but no event logs are found
val title = s"Application history not found (${app.id})"
var msg = s"No event logs found for application $appName in $eventLogFile."
- logWarning(msg)
+ val exception = URLEncoder.encode(Utils.exceptionString(fnf), "UTF-8")
+ logWarning(msg, fnf)
msg += " Did you specify the correct logging directory?"
msg = URLEncoder.encode(msg, "UTF-8")
- app.desc.appUiUrl = notFoundBasePath + s"?msg=$msg&title=$title"
+ app.desc.appUiUrl = notFoundBasePath + s"?msg=$msg&exception=$exception&title=$title"
false
case e: Exception =>
- // Relay exception message to application UI page
+ // Replay exception message to application UI page
--- End diff --
The word `Relay` was correct here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3824#discussion_r22306063
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala ---
@@ -758,13 +760,14 @@ private[spark] class Master(
// Event logging is enabled for this application, but no event logs are found
val title = s"Application history not found (${app.id})"
var msg = s"No event logs found for application $appName in $eventLogFile."
- logWarning(msg)
+ val exception = URLEncoder.encode(Utils.exceptionString(fnf), "UTF-8")
+ logWarning(msg, fnf)
msg += " Did you specify the correct logging directory?"
msg = URLEncoder.encode(msg, "UTF-8")
- app.desc.appUiUrl = notFoundBasePath + s"?msg=$msg&title=$title"
+ app.desc.appUiUrl = notFoundBasePath + s"?msg=$msg&exception=$exception&title=$title"
--- End diff --
This will likely be too long in general to put in a URL. Did you add this URL param elsewhere?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3824#issuecomment-68243544
[Test build #24861 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24861/consoleFull) for PR 3824 at commit [`a49c52f`](https://github.com/apache/spark/commit/a49c52fc995c7ac110d0ab07a4da2f87cf74de2d).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Posted by liyezhang556520 <gi...@git.apache.org>.
Github user liyezhang556520 commented on the pull request:
https://github.com/apache/spark/pull/3824#issuecomment-69280380
@andrewor14 , I received an email of your comment about creating other PRs to fix this issue for other older branches, but not found on this page. I think you might have removed that comment, so do I still need to make new PRs or just ignore that message?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3824#issuecomment-68244526
[Test build #24858 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24858/consoleFull) for PR 3824 at commit [`12eee85`](https://github.com/apache/spark/commit/12eee8590fb9899c267b29d3a129a169b6cf6ec1).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3824#issuecomment-69133658
[Test build #25199 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25199/consoleFull) for PR 3824 at commit [`3c24d98`](https://github.com/apache/spark/commit/3c24d981f7f28b3ce8f56e57c14b48458eb378b2).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/3824#issuecomment-69280965
Yes that would be great. It seems that not all of the changes in this PR are applicable there, however.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Posted by liyezhang556520 <gi...@git.apache.org>.
Github user liyezhang556520 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3824#discussion_r22633623
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala ---
@@ -758,13 +760,14 @@ private[spark] class Master(
// Event logging is enabled for this application, but no event logs are found
val title = s"Application history not found (${app.id})"
var msg = s"No event logs found for application $appName in $eventLogFile."
- logWarning(msg)
+ val exception = URLEncoder.encode(Utils.exceptionString(fnf), "UTF-8")
+ logWarning(msg, fnf)
msg += " Did you specify the correct logging directory?"
msg = URLEncoder.encode(msg, "UTF-8")
- app.desc.appUiUrl = notFoundBasePath + s"?msg=$msg&title=$title"
+ app.desc.appUiUrl = notFoundBasePath + s"?msg=$msg&exception=$exception&title=$title"
--- End diff --
ok, I'll revert the change, thanks @andrewor14 , @JoshRosen , @srowen
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3824#issuecomment-68622517
[Test build #25025 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25025/consoleFull) for PR 3824 at commit [`3c1ac2e`](https://github.com/apache/spark/commit/3c1ac2ef5ea608e00eb45085ee65ae950c21be1b).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Posted by liyezhang556520 <gi...@git.apache.org>.
Github user liyezhang556520 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3824#discussion_r22336254
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala ---
@@ -719,26 +719,28 @@ private[spark] class Master(
def rebuildSparkUI(app: ApplicationInfo): Boolean = {
val appName = app.desc.name
val notFoundBasePath = HistoryServer.UI_PATH_PREFIX + "/not-found"
- val eventLogFile = app.desc.eventLogDir
- .map { dir => EventLoggingListener.getLogPath(dir, app.id) }
- .getOrElse {
- // Event logging is not enabled for this application
- app.desc.appUiUrl = notFoundBasePath
- return false
- }
- val fs = Utils.getHadoopFileSystem(eventLogFile, hadoopConf)
+ var eventLogFile: String = null
--- End diff --
Hi @JoshRosen , `eventLogFile` is also used in `catch` block in this file.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3824#issuecomment-68248630
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24861/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/3824#issuecomment-69226898
Hey @liyezhang556520 this doesn't merge cleanly in older branches. Can you make new PRs against those branches?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3824#issuecomment-68248623
[Test build #24861 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24861/consoleFull) for PR 3824 at commit [`a49c52f`](https://github.com/apache/spark/commit/a49c52fc995c7ac110d0ab07a4da2f87cf74de2d).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org