You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by devaraj-kavali <gi...@git.apache.org> on 2017/02/04 01:45:28 UTC
[GitHub] spark pull request #16801: [SPARK-13619] [WEBUI] [CORE] Jobs page UI shows w...
GitHub user devaraj-kavali opened a pull request:
https://github.com/apache/spark/pull/16801
[SPARK-13619] [WEBUI] [CORE] Jobs page UI shows wrong number of failed tasks
## What changes were proposed in this pull request?
When the Failed/Killed Task End events come after the Job End event then these task events are simply getting ignored without considering/updating into JobUIData. It is happening because jobId information is getting deleted from stageIdToActiveJobIds during the Job End event and Task End event is not able to find the Job information to update.
## How was this patch tested?
### Current behaviour of Spark Jobs page for Running Application and History page,
#### Completed Jobs (1)
| Job Id | Description | Submitted | Duration | Stages: Succeeded/Total | Tasks (for all stages): Succeeded/Total |
| --- | --- | --- | --- | --- | --- |
| 0 | saveAsTextFile at JavaWordCountWithSlowTask.java:49 | 2017/01/25 09:03:14 | 1.4 min | 2/2 | 400/400 (17 killed) |
#### Completed Stages (2)
| Stage Id | Description | Submitted | Duration | Tasks: Succeeded/Total | Input | Output | Shuffle Read | Shuffle Write |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | saveAsTextFile at JavaWordCountWithSlowTask.java:49 +details | 2017/01/25 09:04:34 | 5 s | 200/200 (2 failed) (1 killed) | | 6.8 KB | 2.3 MB | |
| 0 | mapToPair at JavaWordCountWithSlowTask.java:33 +details | 2017/01/25 09:03:15 | 1.3 min | 200/200 (16 killed) | 1915.5 MB | | | 2.3 MB |
### Behaviour of the Web Pages after applying the patch,
#### Completed Jobs (1)
| Job Id | Description | Submitted | Duration | Stages: Succeeded/Total | Tasks (for all stages): Succeeded/Total |
| --- | --- | --- | --- | --- | --- |
| 0 | saveAsTextFile at JavaWordCountWithSlowTask.java:49 | 2017/01/25 09:03:14 | 1.4 min | 2/2 | 400/400 (2 failed) (17 killed) |
#### Completed Stages (2)
| Stage Id | Description | Submitted | Duration | Tasks: Succeeded/Total | Input | Output | Shuffle Read | Shuffle Write |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | saveAsTextFile at JavaWordCountWithSlowTask.java:49 +details | 2017/01/25 09:04:34 | 5 s | 200/200 (2 failed) (1 killed) | | 6.8 KB | 2.3 MB |
| 0 | mapToPair at JavaWordCountWithSlowTask.java:33 +details | 2017/01/25 09:03:15 | 1.3 min | 200/200 (16 killed) | 1915.5 MB | | | 2.3 MB |
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/devaraj-kavali/spark SPARK-13619
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16801.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16801
----
commit 3b51ef0e5ddd58e0bd8f90a52ca08145e5cdef4d
Author: Devaraj K <de...@apache.org>
Date: 2017-02-04T01:40:35Z
[SPARK-13619] [WEBUI] [CORE] Jobs page UI shows wrong number of failed
tasks
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16801: [SPARK-13619] [WEBUI] [CORE] Jobs page UI shows wrong nu...
Posted by devaraj-kavali <gi...@git.apache.org>.
Github user devaraj-kavali commented on the issue:
https://github.com/apache/spark/pull/16801
Will identify better solution to fix this issue and create a new PR, closing it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16801: [SPARK-13619] [WEBUI] [CORE] Jobs page UI shows w...
Posted by devaraj-kavali <gi...@git.apache.org>.
Github user devaraj-kavali closed the pull request at:
https://github.com/apache/spark/pull/16801
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16801: [SPARK-13619] [WEBUI] [CORE] Jobs page UI shows w...
Posted by devaraj-kavali <gi...@git.apache.org>.
Github user devaraj-kavali commented on a diff in the pull request:
https://github.com/apache/spark/pull/16801#discussion_r122792258
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -234,7 +234,6 @@ class JobProgressListener(conf: SparkConf) extends SparkListener with Logging {
}
for (stageId <- jobData.stageIds) {
stageIdToActiveJobIds.get(stageId).foreach { jobsUsingStage =>
- jobsUsingStage.remove(jobEnd.jobId)
--- End diff --
Thanks @vanzin for looking into this. I agree, the change will prevent to remove entries from the map. I will update the PR with the better solution.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16801: [SPARK-13619] [WEBUI] [CORE] Jobs page UI shows wrong nu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16801
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16801: [SPARK-13619] [WEBUI] [CORE] Jobs page UI shows wrong nu...
Posted by devaraj-kavali <gi...@git.apache.org>.
Github user devaraj-kavali commented on the issue:
https://github.com/apache/spark/pull/16801
@jiangxb1987, Please find the difference.
Current behaviour of Spark Jobs page for Running Application and History page,
400/400 (17 killed)
Behaviour of the Web Pages after applying the patch,
400/400 (2 failed) (17 killed)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16801: [SPARK-13619] [WEBUI] [CORE] Jobs page UI shows wrong nu...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/16801
Sorry but I didn't get what's different of the changed web pages you posted?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16801: [SPARK-13619] [WEBUI] [CORE] Jobs page UI shows w...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16801#discussion_r122326702
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -234,7 +234,6 @@ class JobProgressListener(conf: SparkConf) extends SparkListener with Logging {
}
for (stageId <- jobData.stageIds) {
stageIdToActiveJobIds.get(stageId).foreach { jobsUsingStage =>
- jobsUsingStage.remove(jobEnd.jobId)
--- End diff --
I understand the issue, but now you're creating a memory leak, since these mappings will never be removed, right? Which will also cause the condition in the next line to never trigger, causing a memory leak in a different map too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org