You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by uncleGen <gi...@git.apache.org> on 2014/09/29 10:38:46 UTC
[GitHub] spark pull request: [SPARK-3719][CORE]:"complete/failed stages" is...
GitHub user uncleGen opened a pull request:
https://github.com/apache/spark/pull/2574
[SPARK-3719][CORE]:"complete/failed stages" is better to show the total ...
...number of stages
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/uncleGen/spark master-clean-14929
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2574.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2574
----
commit ebb59ac65e352422575a2f4b5675557e334f82c0
Author: uncleGen <hu...@gmail.com>
Date: 2014-09-29T08:36:46Z
[SPARK-3719][CORE]:"complete/failed stages" is better to show the total number of stages
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3719][CORE]:"complete/failed stages" is...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2574#discussion_r18596525
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressPage.scala ---
@@ -70,11 +72,11 @@ private[ui] class JobProgressPage(parent: JobProgressTab) extends WebUIPage("")
</li>
<li>
<a href="#completed"><strong>Completed Stages:</strong></a>
- {completedStages.size}
+ {totalCompletedStages}
--- End diff --
@uncleGen The web UI only remembers information on a fixed number of stages in order to limit its memory usage, so it's possible that the displayed number of completed stages will be much higher than the number of rows displayed in the stage table. I think Reynold's suggestion was to make this more explicit by adding "(only showing yy)" to emphasize that some garbage-collected stage information ins't being displayed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3719][CORE][UI]:"complete/failed stages...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2574#issuecomment-58738484
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21624/consoleFull) for PR 2574 at commit [`4fee5a8`](https://github.com/apache/spark/commit/4fee5a8400e87f7bb33363194cc3039feb3dbed6).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3719][CORE]:"complete/failed stages" is...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2574#issuecomment-57131964
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20966/consoleFull) for PR 2574 at commit [`ebb59ac`](https://github.com/apache/spark/commit/ebb59ac65e352422575a2f4b5675557e334f82c0).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3719][CORE][UI]:"complete/failed stages...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2574#issuecomment-58909899
Thanks for updating this. We should use the same conditional formatting logic for the table headings, too. Right now, the UI says "Only showing 0" when displaying a fresh web UI, which is confusing:
It would probably be easiest to make a helper function for formatting the number, then call that function in both locations.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3719][CORE][UI]:"complete/failed stages...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2574#issuecomment-58738534
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21623/Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3719][CORE]:"complete/failed stages" is...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2574#issuecomment-57132298
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20966/consoleFull) for PR 2574 at commit [`ebb59ac`](https://github.com/apache/spark/commit/ebb59ac65e352422575a2f4b5675557e334f82c0).
* This patch **fails** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3719][CORE][UI]:"complete/failed stages...
Posted by uncleGen <gi...@git.apache.org>.
Github user uncleGen commented on the pull request:
https://github.com/apache/spark/pull/2574#issuecomment-63651117
@JoshRosen [[SPARK-4168][WebUI] ](https://github.com/apache/spark/commit/97a466eca0a629f17e9662ca2b59eeca99142c54) The patch solved the same problem, and I will close this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3719][CORE][UI]:"complete/failed stages...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2574#issuecomment-58739666
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21624/Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3719][CORE]:"complete/failed stages" is...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2574#issuecomment-58634418
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21588/Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3719][CORE][UI]:"complete/failed stages...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2574#issuecomment-58739664
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21624/consoleFull) for PR 2574 at commit [`4fee5a8`](https://github.com/apache/spark/commit/4fee5a8400e87f7bb33363194cc3039feb3dbed6).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3719][CORE]:"complete/failed stages" is...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2574#discussion_r18321787
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressPage.scala ---
@@ -70,11 +72,11 @@ private[ui] class JobProgressPage(parent: JobProgressTab) extends WebUIPage("")
</li>
<li>
<a href="#completed"><strong>Completed Stages:</strong></a>
- {completedStages.size}
+ {totalCompletedStages}
--- End diff --
I agree; this will help to avoid user confusion if there's a big difference between the count and number of displayed stages.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3719][CORE]:"complete/failed stages" is...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2574#issuecomment-57141453
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20967/consoleFull) for PR 2574 at commit [`42b27b9`](https://github.com/apache/spark/commit/42b27b95c7373e321cbf527f00cd22807b869a53).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3719][CORE][UI]:"complete/failed stages...
Posted by uncleGen <gi...@git.apache.org>.
Github user uncleGen closed the pull request at:
https://github.com/apache/spark/pull/2574
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3719][CORE]:"complete/failed stages" is...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2574#issuecomment-58702467
Hi @uncleGen,
I don't think this is quite what Reynold and I had in mind. Right now, it looks like the displayed number is essentially `min(#displayed stages, #total stages)`. I think we were hoping to always display "total stages" and add additional text to indicate that fewer stages are being displayed in the table. For instance, if I configure the web UI to only remember information on 100 stages but I have run 300 stages to date, then the UI would say "Completed Stages: 300 (only showing 100)". But, if all stages were displayed (e.g. totalStages <= maxStagesRetained), then we would simply show "Completed Stages: 50" in order to avoid redundancy.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3719][CORE]:"complete/failed stages" is...
Posted by uncleGen <gi...@git.apache.org>.
Github user uncleGen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2574#discussion_r18561760
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressPage.scala ---
@@ -70,11 +72,11 @@ private[ui] class JobProgressPage(parent: JobProgressTab) extends WebUIPage("")
</li>
<li>
<a href="#completed"><strong>Completed Stages:</strong></a>
- {completedStages.size}
+ {totalCompletedStages}
--- End diff --
Sorry for the delay on this, I do not realize the necessity about this. @rxin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3719][CORE]:"complete/failed stages" is...
Posted by uncleGen <gi...@git.apache.org>.
Github user uncleGen commented on the pull request:
https://github.com/apache/spark/pull/2574#issuecomment-58735669
@JoshRosen Sorry for my misunderstanding, I will correct it as soon as possible.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3719][CORE]:"complete/failed stages" is...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2574#issuecomment-57132299
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20966/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3719][CORE]:"complete/failed stages" is...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2574#issuecomment-58634416
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21588/consoleFull) for PR 2574 at commit [`304c4fc`](https://github.com/apache/spark/commit/304c4fc6a9acc9a65516966b990590d3bb7247df).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3719][CORE]:"complete/failed stages" is...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2574#issuecomment-57134536
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20967/consoleFull) for PR 2574 at commit [`42b27b9`](https://github.com/apache/spark/commit/42b27b95c7373e321cbf527f00cd22807b869a53).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3719][CORE]:"complete/failed stages" is...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2574#discussion_r18170469
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressPage.scala ---
@@ -70,11 +72,11 @@ private[ui] class JobProgressPage(parent: JobProgressTab) extends WebUIPage("")
</li>
<li>
<a href="#completed"><strong>Completed Stages:</strong></a>
- {completedStages.size}
+ {totalCompletedStages}
--- End diff --
how about saying "xx (only showing yy)" when yy != xx?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3719][CORE]:"complete/failed stages" is...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2574#issuecomment-57141456
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20967/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3719][CORE]:"complete/failed stages" is...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2574#issuecomment-58628221
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21588/consoleFull) for PR 2574 at commit [`304c4fc`](https://github.com/apache/spark/commit/304c4fc6a9acc9a65516966b990590d3bb7247df).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3719][CORE]:"complete/failed stages" is...
Posted by uncleGen <gi...@git.apache.org>.
Github user uncleGen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2574#discussion_r18622733
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressPage.scala ---
@@ -70,11 +72,11 @@ private[ui] class JobProgressPage(parent: JobProgressTab) extends WebUIPage("")
</li>
<li>
<a href="#completed"><strong>Completed Stages:</strong></a>
- {completedStages.size}
+ {totalCompletedStages}
--- End diff --
Ah, I see, and will do some modifications. Thank you! @JoshRosen
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org