You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by guoxiaolongzte <gi...@git.apache.org> on 2018/01/17 03:28:53 UTC
[GitHub] spark pull request #20287: [SPARK-23121][WEB-UI] When the Spark Streaming ap...
GitHub user guoxiaolongzte opened a pull request:
https://github.com/apache/spark/pull/20287
[SPARK-23121][WEB-UI] When the Spark Streaming app is running for a period of time, the page is incorrectly reported when accessing '/jobs' or '/jobs/job?id=13'
## What changes were proposed in this pull request?
When the Spark Streaming app is running for a period of time, the page is incorrectly reported when accessing '/ jobs /' or '/ jobs / job /? Id = 13' and ui can not be accessed.
Test command:
./bin/spark-submit --class org.apache.spark.examples.streaming.HdfsWordCount ./examples/jars/spark-examples_2.11-2.4.0-SNAPSHOT.jar /spark
The app is running for a period of time, ui can not be accessed, please see attachment.
fix before:
## How was this patch tested?
manual tests
Please review http://spark.apache.org/contributing.html before opening a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/guoxiaolongzte/spark SPARK-23121
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20287.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20287
----
commit 03a84436ef2b6227f8bcfdd0b803c9457c8bd5cd
Author: guoxiaolong <gu...@...>
Date: 2018-01-17T03:26:19Z
[SPARK-23121][WEB-UI]When the Spark Streaming app is running for a period of time, the page is incorrectly reported when accessing '/josb' or '/jobs/job?id=13'
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20287: [SPARK-23121][WEB-UI] When the Spark Streaming ap...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/20287#discussion_r161950447
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala ---
@@ -65,10 +65,13 @@ private[ui] class AllJobsPage(parent: JobsTab, store: AppStatusStore) extends We
}.map { job =>
val jobId = job.jobId
val status = job.status
- val jobDescription = store.lastStageAttempt(job.stageIds.max).description
- val displayJobDescription = jobDescription
- .map(UIUtils.makeDescription(_, "", plainText = true).text)
- .getOrElse("")
+ var displayJobDescription = ""
+ try {
+ displayJobDescription = store.lastStageAttempt(job.stageIds.max).description
+ .map(UIUtils.makeDescription(_, "", plainText = true).text).getOrElse("")
+ } catch {
+ case e => displayJobDescription = job.description.getOrElse("")
--- End diff --
No, you're catching all exceptions. This should check whether the last stage attempt exists rather than catching anything that goes wrong.
I don't think it's correct to pretend it exists but is empty. The request is for something that doesn't exist and ideally generates a 404.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20287: [SPARK-23121][WEB-UI] When the Spark Streaming ap...
Posted by smurakozi <gi...@git.apache.org>.
Github user smurakozi commented on a diff in the pull request:
https://github.com/apache/spark/pull/20287#discussion_r162566562
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobPage.scala ---
@@ -335,9 +335,12 @@ private[ui] class JobPage(parent: JobsTab, store: AppStatusStore) extends WebUIP
content ++= makeTimeline(activeStages ++ completedStages ++ failedStages,
store.executorList(false), appStartTime)
-
- content ++= UIUtils.showDagVizForJob(
- jobId, store.operationGraphForJob(jobId))
+ try {
+ content ++= UIUtils.showDagVizForJob(
+ jobId, store.operationGraphForJob(jobId))
+ } catch {
+ case e => None
+ }
--- End diff --
Same here. We should avoid the situation when the exception is thrown. Catching the exception and doing nothing just hides the problems.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20287: [SPARK-23121][WEB-UI] When the Spark Streaming ap...
Posted by smurakozi <gi...@git.apache.org>.
Github user smurakozi commented on a diff in the pull request:
https://github.com/apache/spark/pull/20287#discussion_r162566289
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala ---
@@ -427,17 +430,24 @@ private[ui] class JobDataSource(
val formattedDuration = duration.map(d => UIUtils.formatDuration(d)).getOrElse("Unknown")
val submissionTime = jobData.submissionTime
val formattedSubmissionTime = submissionTime.map(UIUtils.formatDate).getOrElse("Unknown")
- val lastStageAttempt = store.lastStageAttempt(jobData.stageIds.max)
- val lastStageDescription = lastStageAttempt.description.getOrElse("")
-
+ var lastStageDescription = ""
--- End diff --
Instead of catching the exception the logic should be modified to be prepared for a missing stageAttempt.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20287: [SPARK-23121][WEB-UI] When the Spark Streaming app is ru...
Posted by guoxiaolongzte <gi...@git.apache.org>.
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/20287
@smurakozi
Help review the code, this bug results from your added functionality.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20287: [SPARK-23121][WEB-UI] When the Spark Streaming ap...
Posted by guoxiaolongzte <gi...@git.apache.org>.
Github user guoxiaolongzte closed the pull request at:
https://github.com/apache/spark/pull/20287
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20287: [SPARK-23121][WEB-UI] When the Spark Streaming app is ru...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20287
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20287: [SPARK-23121][WEB-UI] When the Spark Streaming app is ru...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/20287
This is far from a fatal bug. The page doesn't render, but it should give some kind of error anyway. What change are you referring to? In any event, this change is unsuitable.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20287: [SPARK-23121][WEB-UI] When the Spark Streaming app is ru...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20287
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20287: [SPARK-23121][WEB-UI] When the Spark Streaming app is ru...
Posted by guoxiaolongzte <gi...@git.apache.org>.
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/20287
Well, then you can tell me how specific changes? I do not have a good idea right now. The problem is that the page crashes, it should be a fatal bug.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20287: [SPARK-23121][WEB-UI] When the Spark Streaming app is ru...
Posted by guoxiaolongzte <gi...@git.apache.org>.
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/20287
@smurakozi @vanzin @srowen
Thanks, i will close the PR.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20287: [SPARK-23121][WEB-UI] When the Spark Streaming app is ru...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20287
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org