You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by jerryshao <gi...@git.apache.org> on 2018/01/18 13:16:57 UTC
[GitHub] spark pull request #20315: [SPARK-23147][UI] Fix task page table IndexOutOfB...
GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/20315
[SPARK-23147][UI] Fix task page table IndexOutOfBound Exception
## What changes were proposed in this pull request?
Stage's task page table will throw an exception when there's no complete tasks. Furthermore, because the `dataSize` doesn't take running tasks into account, so sometimes UI cannot show the running tasks. Besides table will only be displayed when first task is finished according to the default sortColumn("index").
To reproduce this issue, user could try `sc.parallelize(1 to 20, 20).map { i => Thread.sleep(10000); i }.collect()` or `sc.parallelize(1 to 20, 20).map { i => Thread.sleep((20 - i) * 1000); i }.collect` to reproduce the above issue.
Here propose a solution to fix it. Not sure if it is a right fix, please help to review.
## How was this patch tested?
Manual test.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jerryshao/apache-spark SPARK-23147
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20315.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20315
----
commit 568aa847745431f62d79147f80731aa7cb7945b1
Author: jerryshao <ss...@...>
Date: 2018-01-18T13:09:03Z
Fix task page table exception
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20315: [SPARK-23147][UI] Fix task page table IndexOutOfBound Ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20315
**[Test build #86341 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86341/testReport)** for PR 20315 at commit [`568aa84`](https://github.com/apache/spark/commit/568aa847745431f62d79147f80731aa7cb7945b1).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20315: [SPARK-23147][UI] Fix task page table IndexOutOfBound Ex...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/20315
Merging to master / 2.3.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20315: [SPARK-23147][UI] Fix task page table IndexOutOfBound Ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20315
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86341/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20315: [SPARK-23147][UI] Fix task page table IndexOutOfBound Ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20315
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20315: [SPARK-23147][UI] Fix task page table IndexOutOfBound Ex...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/20315
Timed out on kafka test, no reason to run tests again.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20315: [SPARK-23147][UI] Fix task page table IndexOutOfB...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/20315#discussion_r162419169
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -676,7 +676,7 @@ private[ui] class TaskDataSource(
private var _tasksToShow: Seq[TaskData] = null
- override def dataSize: Int = stage.numCompleteTasks + stage.numFailedTasks + stage.numKilledTasks
--- End diff --
Probably just a side effect of me testing mostly with replayed logs and short-lived tasks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20315: [SPARK-23147][UI] Fix task page table IndexOutOfB...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/20315
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20315: [SPARK-23147][UI] Fix task page table IndexOutOfBound Ex...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/20315
Fix seems to be in line with what Spark 2.2 did, so LGTM pending tests.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20315: [SPARK-23147][UI] Fix task page table IndexOutOfB...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20315#discussion_r162338495
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -676,7 +676,7 @@ private[ui] class TaskDataSource(
private var _tasksToShow: Seq[TaskData] = null
- override def dataSize: Int = stage.numCompleteTasks + stage.numFailedTasks + stage.numKilledTasks
--- End diff --
@vanzin I think you made this change, why not also track the running tasks? I'm not sure what is your original purpose.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20315: [SPARK-23147][UI] Fix task page table IndexOutOfBound Ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20315
**[Test build #86341 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86341/testReport)** for PR 20315 at commit [`568aa84`](https://github.com/apache/spark/commit/568aa847745431f62d79147f80731aa7cb7945b1).
* This patch **fails from timeout after a configured wait of \`300m\`**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org