You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by smurakozi <gi...@git.apache.org> on 2018/01/12 16:13:51 UTC
[GitHub] spark pull request #20251: [Spark-23051][core] Fix for broken job descriptio...
GitHub user smurakozi opened a pull request:
https://github.com/apache/spark/pull/20251
[Spark-23051][core] Fix for broken job description in Spark UI
## What changes were proposed in this pull request?
In 2.2, Spark UI displayed the stage description if the job description was not set. This functionality was broken, the GUI has shown no description in this case. In addition, the code uses jobName and
jobDescription instead of stageName and stageDescription when JobTableRowData is created.
In this PR the logic producing values for the job rows was modified to find the latest stage attempt for the job and use that as a fallback if job description was missing.
StageName and stageDescription are also set using values from stage and jobName/description is used only as a fallback.
## How was this patch tested?
Manual testing of the UI, using the code in the bug report.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/smurakozi/spark SPARK-23051
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20251.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20251
----
commit 0412c4b959b42cb0b41c669593e689430da59ec2
Author: Sandor Murakozi <sm...@...>
Date: 2018-01-12T15:10:24Z
Fix for SPARK-23051
commit 937e0fd579a7f133f860eb6f572f50ad7d7fdf89
Author: Sandor Murakozi <sm...@...>
Date: 2018-01-12T15:25:44Z
fixed logic determining jobDescription
commit df41000d99eef3b609ece4b072c5aee0650c21f1
Author: Sandor Murakozi <sm...@...>
Date: 2018-01-12T15:51:06Z
Formatted code
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20251: [Spark-23051][core] Fix for broken job descriptio...
Posted by smurakozi <gi...@git.apache.org>.
Github user smurakozi commented on a diff in the pull request:
https://github.com/apache/spark/pull/20251#discussion_r161293689
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala ---
@@ -429,20 +429,40 @@ private[ui] class JobDataSource(
val formattedDuration = duration.map(d => UIUtils.formatDuration(d)).getOrElse("Unknown")
val submissionTime = jobData.submissionTime
val formattedSubmissionTime = submissionTime.map(UIUtils.formatDate).getOrElse("Unknown")
- val jobDescription = UIUtils.makeDescription(jobData.description.getOrElse(""),
- basePath, plainText = false)
+
+ val lastStageAttempt = {
+ val stageAttempts = jobData.stageIds.flatMap(store.stageData(_))
--- End diff --
Do we want to have data of the last stage or the last stageAttempt?
In the second case we should load only the last stage, indeed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20251: [Spark-23051][core] Fix for broken job description in Sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20251
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20251: [Spark-23051][core] Fix for broken job description in Sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20251
**[Test build #86053 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86053/testReport)** for PR 20251 at commit [`fb9ce50`](https://github.com/apache/spark/commit/fb9ce507782a72d5e6ee6b373c3bda33fc819e13).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20251: [Spark-23051][core] Fix for broken job description in Sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20251
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20251: [Spark-23051][core] Fix for broken job descriptio...
Posted by smurakozi <gi...@git.apache.org>.
Github user smurakozi commented on a diff in the pull request:
https://github.com/apache/spark/pull/20251#discussion_r161273851
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala ---
@@ -429,20 +429,40 @@ private[ui] class JobDataSource(
val formattedDuration = duration.map(d => UIUtils.formatDuration(d)).getOrElse("Unknown")
val submissionTime = jobData.submissionTime
val formattedSubmissionTime = submissionTime.map(UIUtils.formatDate).getOrElse("Unknown")
- val jobDescription = UIUtils.makeDescription(jobData.description.getOrElse(""),
- basePath, plainText = false)
+
+ val lastStage = {
+ val stageAttempts = jobData.stageIds.flatMap(store.stageData(_))
+ if (!stageAttempts.isEmpty) {
+ val lastId = stageAttempts.map(_.stageId).max
+ stageAttempts.find(_.stageId == lastId)
--- End diff --
Indeed, nice catch, thx
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20251: [Spark-23051][core] Fix for broken job description in Sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20251
**[Test build #86053 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86053/testReport)** for PR 20251 at commit [`fb9ce50`](https://github.com/apache/spark/commit/fb9ce507782a72d5e6ee6b373c3bda33fc819e13).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20251: [Spark-23051][core] Fix for broken job description in Sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20251
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86053/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20251: [Spark-23051][core] Fix for broken job description in Sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20251
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20251: [Spark-23051][core] Fix for broken job description in Sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20251
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20251: [Spark-23051][core] Fix for broken job description in Sp...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/20251
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20251: [Spark-23051][core] Fix for broken job descriptio...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/20251#discussion_r161296230
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala ---
@@ -429,20 +429,40 @@ private[ui] class JobDataSource(
val formattedDuration = duration.map(d => UIUtils.formatDuration(d)).getOrElse("Unknown")
val submissionTime = jobData.submissionTime
val formattedSubmissionTime = submissionTime.map(UIUtils.formatDate).getOrElse("Unknown")
- val jobDescription = UIUtils.makeDescription(jobData.description.getOrElse(""),
- basePath, plainText = false)
+
+ val lastStageAttempt = {
+ val stageAttempts = jobData.stageIds.flatMap(store.stageData(_))
--- End diff --
Your code in L436 is getting last stage for this job, so loading all stages here seems unnecessary.
I guess it's not guaranteed that the attempt that ran for this particular job is the last attempt of that particular stage. But the logic you have here, as far as I can see, will return the *fist* attempt of the stage with the highest id:
```
scala> case class Foo(id: Int, str: String)
defined class Foo
scala> Seq(Foo(3, "one"), Foo(2, "two"), Foo(3, "three")).maxBy(_.id)
res1: Foo = Foo(3,one)
```
Given that you're only interested in the description anyway, and that won't change between attempts, either logic is fine. Loading the last attempt only is just a little cheaper.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20251: [Spark-23051][core] Fix for broken job description in Sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20251
**[Test build #4050 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4050/testReport)** for PR 20251 at commit [`d9cdb07`](https://github.com/apache/spark/commit/d9cdb07263f7a584cf217d30c55313283459ac92).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20251: [Spark-23051][core] Fix for broken job description in Sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20251
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86049/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20251: [Spark-23051][core] Fix for broken job descriptio...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/20251#discussion_r161287158
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala ---
@@ -429,20 +429,40 @@ private[ui] class JobDataSource(
val formattedDuration = duration.map(d => UIUtils.formatDuration(d)).getOrElse("Unknown")
val submissionTime = jobData.submissionTime
val formattedSubmissionTime = submissionTime.map(UIUtils.formatDate).getOrElse("Unknown")
- val jobDescription = UIUtils.makeDescription(jobData.description.getOrElse(""),
- basePath, plainText = false)
+
+ val lastStageAttempt = {
+ val stageAttempts = jobData.stageIds.flatMap(store.stageData(_))
+ if (stageAttempts.nonEmpty) {
+ Some(stageAttempts.maxBy(_.stageId))
+ } else {
+ None
+ }
+ }
+
+ val jobDescription = jobData.description
+ .getOrElse(lastStageAttempt.flatMap(_.description)
--- End diff --
```
.orElse(lastStageAttempt.flatMap(_.description))
.getOrElse(jobData.name)
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20251: [Spark-23051][core] Fix for broken job description in Sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20251
**[Test build #86049 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86049/testReport)** for PR 20251 at commit [`41fcdce`](https://github.com/apache/spark/commit/41fcdce67eb4f3f621a14df4833a7af70eab5f37).
* This patch **fails from timeout after a configured wait of \`300m\`**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20251: [Spark-23051][core] Fix for broken job description in Sp...
Posted by smurakozi <gi...@git.apache.org>.
Github user smurakozi commented on the issue:
https://github.com/apache/spark/pull/20251
Screenshots:
[In 2.2](https://issues.apache.org/jira/secure/attachment/12905889/in-2.2.png)
[In 2.3 before fix](https://issues.apache.org/jira/secure/attachment/12905887/Spark-23051-before.png)
[In 2.3 after the fix](https://issues.apache.org/jira/secure/attachment/12905888/Spark-23051-after.png)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20251: [Spark-23051][core] Fix for broken job descriptio...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/20251#discussion_r161268598
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala ---
@@ -429,20 +429,40 @@ private[ui] class JobDataSource(
val formattedDuration = duration.map(d => UIUtils.formatDuration(d)).getOrElse("Unknown")
val submissionTime = jobData.submissionTime
val formattedSubmissionTime = submissionTime.map(UIUtils.formatDate).getOrElse("Unknown")
- val jobDescription = UIUtils.makeDescription(jobData.description.getOrElse(""),
- basePath, plainText = false)
+
+ val lastStage = {
+ val stageAttempts = jobData.stageIds.flatMap(store.stageData(_))
+ if (!stageAttempts.isEmpty) {
+ val lastId = stageAttempts.map(_.stageId).max
+ stageAttempts.find(_.stageId == lastId)
+ } else {
+ None
+ }}
+
+ val jobDescription = jobData.description
+ .getOrElse(lastStage.flatMap(_.description).
+ getOrElse(jobData.name))
+
+ val lastStageName = lastStage.map(_.name).getOrElse(jobData.name)
+
+ val lastStageDescription = lastStage.flatMap(_.description)
+ .getOrElse(jobData.description
+ .getOrElse(jobData.name))
--- End diff --
I think you don't need to break this on a new line? it's broken a little differently than on 442 too.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20251: [Spark-23051][core] Fix for broken job descriptio...
Posted by smurakozi <gi...@git.apache.org>.
Github user smurakozi commented on a diff in the pull request:
https://github.com/apache/spark/pull/20251#discussion_r161276636
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala ---
@@ -429,20 +429,40 @@ private[ui] class JobDataSource(
val formattedDuration = duration.map(d => UIUtils.formatDuration(d)).getOrElse("Unknown")
val submissionTime = jobData.submissionTime
val formattedSubmissionTime = submissionTime.map(UIUtils.formatDate).getOrElse("Unknown")
- val jobDescription = UIUtils.makeDescription(jobData.description.getOrElse(""),
- basePath, plainText = false)
+
+ val lastStage = {
+ val stageAttempts = jobData.stageIds.flatMap(store.stageData(_))
+ if (!stageAttempts.isEmpty) {
+ val lastId = stageAttempts.map(_.stageId).max
+ stageAttempts.find(_.stageId == lastId)
+ } else {
+ None
+ }}
+
+ val jobDescription = jobData.description
+ .getOrElse(lastStage.flatMap(_.description).
+ getOrElse(jobData.name))
+
+ val lastStageName = lastStage.map(_.name).getOrElse(jobData.name)
+
+ val lastStageDescription = lastStage.flatMap(_.description)
+ .getOrElse(jobData.description
+ .getOrElse(jobData.name))
--- End diff --
No need to break, but for me, it shows the structure and logic of the code better this way.
I've formatted the calculation of the two descriptions symmetric and I'm also happy to remove unnecessary line breaks if you recommend it.
Thanks for the review and your comments @srowen
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20251: [Spark-23051][core] Fix for broken job description in Sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20251
**[Test build #86066 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86066/testReport)** for PR 20251 at commit [`d9cdb07`](https://github.com/apache/spark/commit/d9cdb07263f7a584cf217d30c55313283459ac92).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20251: [Spark-23051][core] Fix for broken job description in Sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20251
**[Test build #86049 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86049/testReport)** for PR 20251 at commit [`41fcdce`](https://github.com/apache/spark/commit/41fcdce67eb4f3f621a14df4833a7af70eab5f37).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20251: [Spark-23051][core] Fix for broken job description in Sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20251
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20251: [Spark-23051][core] Fix for broken job description in Sp...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20251
@smurakozi thanks for the fix! can you put a screenshot of the UI to show it's fixed? thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20251: [Spark-23051][core] Fix for broken job description in Sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20251
**[Test build #86058 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86058/testReport)** for PR 20251 at commit [`d9cdb07`](https://github.com/apache/spark/commit/d9cdb07263f7a584cf217d30c55313283459ac92).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20251: [Spark-23051][core] Fix for broken job descriptio...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/20251#discussion_r161287222
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala ---
@@ -429,20 +429,40 @@ private[ui] class JobDataSource(
val formattedDuration = duration.map(d => UIUtils.formatDuration(d)).getOrElse("Unknown")
val submissionTime = jobData.submissionTime
val formattedSubmissionTime = submissionTime.map(UIUtils.formatDate).getOrElse("Unknown")
- val jobDescription = UIUtils.makeDescription(jobData.description.getOrElse(""),
- basePath, plainText = false)
+
+ val lastStageAttempt = {
+ val stageAttempts = jobData.stageIds.flatMap(store.stageData(_))
+ if (stageAttempts.nonEmpty) {
+ Some(stageAttempts.maxBy(_.stageId))
+ } else {
+ None
+ }
+ }
+
+ val jobDescription = jobData.description
+ .getOrElse(lastStageAttempt.flatMap(_.description)
+ .getOrElse(jobData.name))
+
+ val lastStageName = lastStageAttempt.map(_.name).getOrElse(jobData.name)
+
+ val lastStageDescription = lastStageAttempt.flatMap(_.description)
+ .getOrElse(jobData.description
--- End diff --
Same idea (`.orElse(...).getOrElse(...)`).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20251: [Spark-23051][core] Fix for broken job description in Sp...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20251
also cc @vanzin
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20251: [Spark-23051][core] Fix for broken job descriptio...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/20251#discussion_r161267849
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala ---
@@ -429,20 +429,40 @@ private[ui] class JobDataSource(
val formattedDuration = duration.map(d => UIUtils.formatDuration(d)).getOrElse("Unknown")
val submissionTime = jobData.submissionTime
val formattedSubmissionTime = submissionTime.map(UIUtils.formatDate).getOrElse("Unknown")
- val jobDescription = UIUtils.makeDescription(jobData.description.getOrElse(""),
- basePath, plainText = false)
+
+ val lastStage = {
+ val stageAttempts = jobData.stageIds.flatMap(store.stageData(_))
--- End diff --
Just a few tiny suggestions for the code, to take or leave. You should be able to omit `(_)`. You can also use `.nonEmpty` below.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20251: [Spark-23051][core] Fix for broken job description in Sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20251
**[Test build #86058 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86058/testReport)** for PR 20251 at commit [`d9cdb07`](https://github.com/apache/spark/commit/d9cdb07263f7a584cf217d30c55313283459ac92).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20251: [Spark-23051][core] Fix for broken job descriptio...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/20251
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20251: [Spark-23051][core] Fix for broken job descriptio...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/20251#discussion_r161340604
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala ---
@@ -65,12 +65,10 @@ private[ui] class AllJobsPage(parent: JobsTab, store: AppStatusStore) extends We
}.map { job =>
val jobId = job.jobId
val status = job.status
- val displayJobDescription =
- if (job.description.isEmpty) {
- job.name
- } else {
- UIUtils.makeDescription(job.description.get, "", plainText = true).text
- }
+ val jobDescription = store.lastStageAttempt(job.stageIds.max).description
+ val displayJobDescription = jobDescription
+ .map(UIUtils.makeDescription(_, "", plainText = true).text)
--- End diff --
indented too far
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20251: [Spark-23051][core] Fix for broken job descriptio...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/20251#discussion_r161268434
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala ---
@@ -429,20 +429,40 @@ private[ui] class JobDataSource(
val formattedDuration = duration.map(d => UIUtils.formatDuration(d)).getOrElse("Unknown")
val submissionTime = jobData.submissionTime
val formattedSubmissionTime = submissionTime.map(UIUtils.formatDate).getOrElse("Unknown")
- val jobDescription = UIUtils.makeDescription(jobData.description.getOrElse(""),
- basePath, plainText = false)
+
+ val lastStage = {
+ val stageAttempts = jobData.stageIds.flatMap(store.stageData(_))
+ if (!stageAttempts.isEmpty) {
+ val lastId = stageAttempts.map(_.stageId).max
+ stageAttempts.find(_.stageId == lastId)
--- End diff --
I think these two lines can just be `Some(stageAttempts.maxBy(_.stageId))`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20251: [Spark-23051][core] Fix for broken job descriptio...
Posted by smurakozi <gi...@git.apache.org>.
Github user smurakozi commented on a diff in the pull request:
https://github.com/apache/spark/pull/20251#discussion_r161273733
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala ---
@@ -429,20 +429,40 @@ private[ui] class JobDataSource(
val formattedDuration = duration.map(d => UIUtils.formatDuration(d)).getOrElse("Unknown")
val submissionTime = jobData.submissionTime
val formattedSubmissionTime = submissionTime.map(UIUtils.formatDate).getOrElse("Unknown")
- val jobDescription = UIUtils.makeDescription(jobData.description.getOrElse(""),
- basePath, plainText = false)
+
+ val lastStage = {
+ val stageAttempts = jobData.stageIds.flatMap(store.stageData(_))
--- End diff --
stageData has this signature:
`def stageData(stageId: Int, details: Boolean = false)`
without (_) scalac doesn't use the default value of details and it complains that it can't use `(Int, Boolean)=>...` on a Seq[Int].
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20251: [Spark-23051][core] Fix for broken job description in Sp...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/20251
Merged to master/2.3
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20251: [Spark-23051][core] Fix for broken job description in Sp...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20251
Ok to test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20251: [Spark-23051][core] Fix for broken job description in Sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20251
**[Test build #4050 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4050/testReport)** for PR 20251 at commit [`d9cdb07`](https://github.com/apache/spark/commit/d9cdb07263f7a584cf217d30c55313283459ac92).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20251: [Spark-23051][core] Fix for broken job description in Sp...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/20251
Tests haven't triggered, weird.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20251: [Spark-23051][core] Fix for broken job descriptio...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/20251#discussion_r161286995
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala ---
@@ -429,20 +429,40 @@ private[ui] class JobDataSource(
val formattedDuration = duration.map(d => UIUtils.formatDuration(d)).getOrElse("Unknown")
val submissionTime = jobData.submissionTime
val formattedSubmissionTime = submissionTime.map(UIUtils.formatDate).getOrElse("Unknown")
- val jobDescription = UIUtils.makeDescription(jobData.description.getOrElse(""),
- basePath, plainText = false)
+
+ val lastStageAttempt = {
+ val stageAttempts = jobData.stageIds.flatMap(store.stageData(_))
--- End diff --
Instead of loading all stages, can't you use `stageIds.max` + `store.lastStageAttempt`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20251: [Spark-23051][core] Fix for broken job description in Sp...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/20251
(test is already running, the failure was from a previous run)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20251: [Spark-23051][core] Fix for broken job descriptio...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/20251#discussion_r161268478
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala ---
@@ -429,20 +429,40 @@ private[ui] class JobDataSource(
val formattedDuration = duration.map(d => UIUtils.formatDuration(d)).getOrElse("Unknown")
val submissionTime = jobData.submissionTime
val formattedSubmissionTime = submissionTime.map(UIUtils.formatDate).getOrElse("Unknown")
- val jobDescription = UIUtils.makeDescription(jobData.description.getOrElse(""),
- basePath, plainText = false)
+
+ val lastStage = {
+ val stageAttempts = jobData.stageIds.flatMap(store.stageData(_))
+ if (!stageAttempts.isEmpty) {
+ val lastId = stageAttempts.map(_.stageId).max
+ stageAttempts.find(_.stageId == lastId)
+ } else {
+ None
+ }}
--- End diff --
Nit: move last brace onto next line for symmetry
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20251: [Spark-23051][core] Fix for broken job description in Sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20251
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86066/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20251: [Spark-23051][core] Fix for broken job description in Sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20251
**[Test build #86066 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86066/testReport)** for PR 20251 at commit [`d9cdb07`](https://github.com/apache/spark/commit/d9cdb07263f7a584cf217d30c55313283459ac92).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20251: [Spark-23051][core] Fix for broken job description in Sp...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/20251
ok to test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20251: [Spark-23051][core] Fix for broken job description in Sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20251
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86058/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org