You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/07/08 04:38:48 UTC
[GitHub] [spark] venkata91 opened a new pull request #33253: [SPARK-36038][CORE]Basic speculation metrics summary for a stage
venkata91 opened a new pull request #33253:
URL: https://github.com/apache/spark/pull/33253
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error message, please read the guideline first:
https://spark.apache.org/error-message-guidelines.html
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Currently there are no speculation metrics available for Spark either at application/job/stage level. This PR is to add some basic speculation metrics for a stage when speculation execution is enabled. This is similar to the existing stage level metrics tracking numTotal (total number of speculated tasks), numCompleted (total number of successful speculated tasks), numFailed (total number of failed speculated tasks), numKilled (total number of killed speculated tasks) etc. With this new set of metrics, it helps further understanding speculative execution feature in the context of the application and also helps in further tuning the speculative execution config knobs.
cc @AngersZhuuuu @ron8hu
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Additional metrics for speculative execution.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Unit tests added and also deployed in our internal platform for quite some time now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-877038535
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-876941119
**[Test build #140841 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140841/testReport)** for PR 33253 at commit [`be65ed1`](https://github.com/apache/spark/commit/be65ed153015c4bd3c97ca54cdb567cc787e69fe).
* This patch **fails RAT tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-885842985
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46089/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-882843282
**[Test build #141270 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141270/testReport)** for PR 33253 at commit [`606b3a4`](https://github.com/apache/spark/commit/606b3a40b791475dd71243a8d20425f0d7ddc291).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak edited a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak edited a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-881910368
Some tests seem to fail. Could you fix them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r676861664
##########
File path: core/src/test/resources/HistoryServerExpectations/stage_with_speculation_summary_expectation.json
##########
@@ -0,0 +1,748 @@
+{
Review comment:
since the speculation summary is part of stage api which is why we see the whole output, not sure how I can minimize this further. I think already this app only ran 10 tasks only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-876941144
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140841/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r675304991
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala
##########
@@ -524,6 +529,11 @@ private[spark] class AppStatusStore(
} else {
None
}
+ val speculationStageSummary: Option[v1.SpeculationStageSummary] = if (withSummaries) {
Review comment:
How about check the condition of `withDetail` instead of `withSummaries` like `executorSummaries`? Then, the new REST API `/speculationSummary` is not needed because `withDetail` is `true` by default when we access to a path of the form of `/stage/stageId/attemptId`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-916454953
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47631/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 edited a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 edited a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-882008124
> Some tests seem to fail. Could you fix them?
@sarutak it seems like something is wrong with the Github actions on my account therefore the tests are not able to run. I filed a support request with Github but they haven't still figured out why it is not working. Is there any other way to run the tests like the previous PR builder jenkins job?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-885886478
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46089/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-885389696
> killedTaskSummary when original attempt succeeded - https://user-images.githubusercontent.com/8871522/126575315-5e766870-eab4-4afc-a02f-ede63148692b.png
killedTaskSummary when speculative attempt succeeded - https://user-images.githubusercontent.com/8871522/126575323-0c7c2fc9-5f28-4b2c-a47e-bbd4ceed2597.png
I know your attempt succeeded but as I hit, `killedTasksSummary` can be empty.
It's not expected behavior right?
If so, we should investigate the cause and resolve it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-926871863
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48124/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r681201707
##########
File path: core/src/test/resources/HistoryServerExpectations/stage_with_speculation_summary_expectation.json
##########
@@ -0,0 +1,748 @@
+{
Review comment:
Do you see a big concern here? The thing is with this change then I have to make changes all those expectation files again. Let me know if you think it could be a issue, will try to fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 edited a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 edited a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-885821250
> > > Ah I see, currently we are populating killedTasksSummary in SpeculationStageSummary only if the killed reason is another attempt succeeded. For other reasons, we are not populating it since this is mainly specific to speculation to understand how many speculated tasks killed vs original tasks killed. So yeah killedTasksSummary can be empty. What do you think should be the right behavior? Also
> >
> >
> > Even if another attempt succeeds, `killedTasksSummary` can be empty when a speculative task is the last task in a job.
> > In this case, the killed reason is `Stage finished`.
> > https://github.com/apache/spark/blob/2fe12a75206d4dbef6d7678b876c16876136cdd0/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1661-L1675
>
> Ah interesting. This is a weird corner case. In this case, if the `TaskKilled` reason is `stage finished` for a speculative task then we increment `original attempt succeeded` counter. That should do it right?
@sarutak Addressed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r693123565
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -1208,6 +1232,33 @@ private[spark] class AppStatusListener(
}
}
+ private def killedTaskSummaryForSpeculationStageSummary(
+ reason: TaskEndReason,
+ oldSummary: Map[String, Int],
+ isSpeculative: Boolean): Map[String, Int] = {
+ reason match {
+ case k: TaskKilled if k.reason.contains("another attempt succeeded") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary.updated("speculated attempt succeeded",
+ oldSummary.getOrElse("speculated attempt succeeded", 0) + 1)
+ }
+ // If the stage is finished and speculative tasks get killed, then the
+ // kill reason is "stage finished"
+ case k: TaskKilled if k.reason.contains("Stage finished") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary
Review comment:
> Do you mean handling the case where the task that got killed has a speculative attempt but got killed because of the speculative attempt succeeded?
Yes, but I understand it's difficult to know precisely.
I don't think that resetting the counter is a good idea as it can confuse users.
My main concern here is that users are confused by the wrongly incremented counter. So one possible compromise is to have note that the counter can be wrongly incremented.
I noticed that `if` block here can also wrongly increment the counter because a speculative task can be zombie.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-890289655
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141917/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-886252329
**[Test build #141609 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141609/testReport)** for PR 33253 at commit [`8b3ce9b`](https://github.com/apache/spark/commit/8b3ce9bc0119a5dda0a29ebda10197bbd504a38b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-887828035
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46243/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r683969271
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -1208,6 +1232,33 @@ private[spark] class AppStatusListener(
}
}
+ private def killedTaskSummaryForSpeculationStageSummary(
+ reason: TaskEndReason,
+ oldSummary: Map[String, Int],
+ isSpeculative: Boolean): Map[String, Int] = {
+ reason match {
+ case k: TaskKilled if k.reason.contains("another attempt succeeded") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary.updated("speculated attempt succeeded",
+ oldSummary.getOrElse("speculated attempt succeeded", 0) + 1)
+ }
+ // If the stage is finished and speculative tasks get killed, then the
+ // kill reason is "stage finished"
+ case k: TaskKilled if k.reason.contains("Stage finished") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary
Review comment:
@sarutak missed to respond here, even if it is zombie `StageData` tracks metrics per stage attempt so the killed task could be either the original task or speculated task. So we are good right? I don't think we need special handling for zombie tasks. Am I missing something here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33253: [SPARK-36038][CORE] Basic speculation metrics summary for a stage
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r665885169
##########
File path: core/src/test/resources/HistoryServerExpectations/limit_app_list_json_expectation.json
##########
@@ -9,9 +24,9 @@
"sparkUser" : "terryk",
"completed" : true,
"appSparkVersion" : "3.1.0-SNAPSHOT",
- "lastUpdatedEpoch" : 0,
+ "startTimeEpoch" : 1594091460235,
Review comment:
Why this changed?
##########
File path: core/src/main/scala/org/apache/spark/status/LiveEntity.scala
##########
@@ -649,7 +678,6 @@ private class SchedulerPool(name: String) extends LiveEntity {
}
private[spark] object LiveEntityHelpers {
-
Review comment:
remove unrelated change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-896427344
**[Test build #142302 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142302/testReport)** for PR 33253 at commit [`1021b73`](https://github.com/apache/spark/commit/1021b73340a313e1c4a6ddaee8079f9651bf095a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-885391140
> > killedTaskSummary when original attempt succeeded - https://user-images.githubusercontent.com/8871522/126575315-5e766870-eab4-4afc-a02f-ede63148692b.png
> > killedTaskSummary when speculative attempt succeeded - https://user-images.githubusercontent.com/8871522/126575323-0c7c2fc9-5f28-4b2c-a47e-bbd4ceed2597.png
>
> I know your attempt succeeded but as I hit, `killedTasksSummary` can be empty.
> It's not expected behavior right?
> If so, we should investigate the cause and resolve it.
Ah I see, currently we are populating `killedTasksSummary` in `SpeculationStageSummary` only if the killed reason is `another attempt succeeded`. For other reasons, we are not populating it since this is mainly specific to speculation to understand how many speculated tasks killed vs original tasks killed. So yeah `killedTasksSummary` can be empty. What do you think should be the right behavior? Also
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-916454453
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143129/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r675309447
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -746,6 +752,24 @@ private[spark] class AppStatusListener(
maybeUpdate(esummary, now)
}
+ val speculationStageSummary = stage.speculationStageSummary
+ if (event.taskInfo.speculative) {
+ speculationStageSummary.numActiveTasks -= 1
+ speculationStageSummary.numCompletedTasks += completedDelta
+ speculationStageSummary.numFailedTasks += failedDelta
+ speculationStageSummary.numKilledTasks += killedDelta
+ }
+ speculationStageSummary.killedTasksSummary = killedTaskSummaryForSpeculationStageSummary(
+ event.reason,
+ speculationStageSummary.killedTasksSummary,
+ event.taskInfo.speculative)
+
+ if (isLastTask) {
Review comment:
`speculationsStageSummary` is optional so we should update it only for speculative tasks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r683969998
##########
File path: core/src/test/resources/HistoryServerExpectations/stage_with_speculation_summary_expectation.json
##########
@@ -0,0 +1,748 @@
+{
Review comment:
Will try to reduce it over the weekend.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-925251483
**[Test build #143513 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143513/testReport)** for PR 33253 at commit [`66dcaa4`](https://github.com/apache/spark/commit/66dcaa4c291b1ea550e2f4871e24c1ae3565b25b).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `public class NettyLogger `
* `class IndexNameTypeHolder(object):`
* ` new_class = type(\"NameType\", (NameTypeHolder,), `
* ` new_class = param.type if isinstance(param, np.dtype) else param`
* `public final class AlwaysFalse extends Filter `
* `public final class AlwaysTrue extends Filter `
* `public final class And extends BinaryFilter `
* `abstract class BinaryComparison extends Filter `
* `abstract class BinaryFilter extends Filter `
* `public final class EqualNullSafe extends BinaryComparison `
* `public final class EqualTo extends BinaryComparison `
* `public abstract class Filter implements Expression, Serializable `
* `public final class GreaterThan extends BinaryComparison `
* `public final class GreaterThanOrEqual extends BinaryComparison `
* `public final class In extends Filter `
* `public final class IsNotNull extends Filter `
* `public final class IsNull extends Filter `
* `public final class LessThan extends BinaryComparison `
* `public final class LessThanOrEqual extends BinaryComparison `
* `public final class Not extends Filter `
* `public final class Or extends BinaryFilter `
* `public final class StringContains extends StringPredicate `
* `public final class StringEndsWith extends StringPredicate `
* `abstract class StringPredicate extends Filter `
* `public final class StringStartsWith extends StringPredicate `
* `public class ColumnarBatch implements AutoCloseable `
* `case class Sec(child: Expression)`
* `case class Csc(child: Expression)`
* `trait OperationHelper extends AliasHelper with PredicateHelper `
* `class SQLOpenHashSet[@specialized(Long, Int, Double, Float) T: ClassTag](`
* `case class OptimizeSkewedJoin(`
* `case class SkewJoinAwareCost(`
* `case class SimpleCostEvaluator(forceOptimizeSkewedJoin: Boolean) extends CostEvaluator `
* `case class WriterBucketSpec(`
* `case class EnsureRequirements(`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-917697575
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143180/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-926222600
**[Test build #143567 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143567/testReport)** for PR 33253 at commit [`1e42d59`](https://github.com/apache/spark/commit/1e42d59d173b02db430f016c95842d9566c6833d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-876901452
@venkata91
> @sarutak It seems like Github actions is disabled for my fork. I have requested them to enable it. Just curious, why do I need to enable it for my fork (repo)?
It's to run tests.
BTW, please show the screenshot in the description rather than conversation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-926182151
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48076/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 edited a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 edited a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-932437690
Created a doc JIRA - https://issues.apache.org/jira/browse/SPARK-36908. Will document the feature as part of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-930405537
@sarutak Addressed both of your comments. Thanks for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-876871451
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-881909197
@sarutak @AngersZhuuuu can you please take a look again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-884578963
> @venkata91 I checked out this change and accessed `/speculationSummary` but `killedTasksSummary` seems empty.
> Could you investigate the cause?
> 
I checked it, it looks to be fine. Attaching the images here.
killedTaskSummary when original attempt succeeded - https://user-images.githubusercontent.com/8871522/126575315-5e766870-eab4-4afc-a02f-ede63148692b.png
killedTaskSummary when speculative attempt succeeded - https://user-images.githubusercontent.com/8871522/126575323-0c7c2fc9-5f28-4b2c-a47e-bbd4ceed2597.png
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Basic speculation metrics summary for a stage
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-876415950
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45312/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r705697237
##########
File path: core/src/main/resources/org/apache/spark/ui/static/stagespage-template.html
##########
@@ -31,10 +31,25 @@ <h4 id="summaryMetricsTitle" class="title-table"></h4>
</tbody>
</table>
</div>
+ <h4 id="speculationSummaryTitle" class="title-table">Speculation Summary</h4>
+ <div id="speculationSummary" class="container-fluid">
+ <table id="speculation-metrics-table" class="table table-striped compact table-dataTable cell-border">
+ <thead>
+ <th>Total</th>
+ <th>Active</th>
+ <th>Complete</th>
+ <th>Failed</th>
+ <th>Killed</th>
+ </thead>
+ <tbody>
+ </tbody>
+ </table>
+ </div>
+ </div>
<h4 id="aggregatedMetrics" class="collapse-table">
<span class="expand-input-rate-arrow arrow-closed" id="arrowtoggle2"></span>
<a class="title-table">Aggregated Metrics by Executor</a>
- </h4>
+ </h4>
Review comment:
Addressed it
##########
File path: core/src/test/resources/HistoryServerExpectations/stage_with_speculation_summary_expectation.json
##########
@@ -0,0 +1,510 @@
+{
+ "status" : "COMPLETE",
+ "stageId" : 0,
+ "attemptId" : 0,
+ "numTasks" : 4,
+ "numActiveTasks" : 0,
+ "numCompleteTasks" : 4,
+ "numFailedTasks" : 0,
+ "numKilledTasks" : 1,
+ "numCompletedIndices" : 4,
+ "submissionTime" : "2021-08-10T23:27:53.488GMT",
+ "firstTaskLaunchedTime" : "2021-08-10T23:27:53.885GMT",
+ "completionTime" : "2021-08-10T23:28:57.679GMT",
+ "executorDeserializeTime" : 12793,
+ "executorDeserializeCpuTime" : 5317155711,
+ "executorRunTime" : 113648,
+ "executorCpuTime" : 284330976,
+ "resultSize" : 3360,
+ "jvmGcTime" : 0,
+ "resultSerializationTime" : 4,
+ "memoryBytesSpilled" : 0,
+ "diskBytesSpilled" : 0,
+ "peakExecutionMemory" : 0,
+ "inputBytes" : 0,
+ "inputRecords" : 0,
+ "outputBytes" : 0,
+ "outputRecords" : 0,
+ "shuffleRemoteBlocksFetched" : 0,
+ "shuffleLocalBlocksFetched" : 0,
+ "shuffleFetchWaitTime" : 0,
+ "shuffleRemoteBytesRead" : 0,
+ "shuffleRemoteBytesReadToDisk" : 0,
+ "shuffleLocalBytesRead" : 0,
+ "shuffleReadBytes" : 0,
+ "shuffleReadRecords" : 0,
+ "shuffleWriteBytes" : 0,
+ "shuffleWriteTime" : 0,
+ "shuffleWriteRecords" : 0,
+ "name" : "collect at <console>:27",
+ "details" : "org.apache.spark.rdd.RDD.collect(RDD.scala:1029)\n$line17.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:27)\n$line17.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:36)\n$line17.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:38)\n$line17.$read$$iw$$iw$$iw$$iw$$iw.<init>(<console>:40)\n$line17.$read$$iw$$iw$$iw$$iw.<init>(<console>:42)\n$line17.$read$$iw$$iw$$iw.<init>(<console>:44)\n$line17.$read$$iw$$iw.<init>(<console>:46)\n$line17.$read$$iw.<init>(<console>:48)\n$line17.$read.<init>(<console>:50)\n$line17.$read$.<init>(<console>:54)\n$line17.$read$.<clinit>(<console>)\n$line17.$eval$.$print$lzycompute(<console>:7)\n$line17.$eval$.$print(<console>:6)\n$line17.$eval.$print(<console>)\nsun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\nsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\nsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\njava.lang.reflect.Method.invoke(Method.java:498)
\nscala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:745)",
+ "schedulingPool" : "default",
+ "rddIds" : [ 1, 0 ],
+ "accumulatorUpdates" : [ ],
+ "tasks" : {
+ "0" : {
+ "taskId" : 0,
+ "index" : 0,
+ "attempt" : 0,
+ "launchTime" : "2021-08-10T23:27:53.885GMT",
+ "duration" : 2234,
+ "executorId" : "7",
+ "host" : "ltx1-hcl12291.grid.linkedin.com",
+ "status" : "SUCCESS",
+ "taskLocality" : "PROCESS_LOCAL",
+ "speculative" : false,
+ "accumulatorUpdates" : [ ],
+ "taskMetrics" : {
+ "executorDeserializeTime" : 2048,
+ "executorDeserializeCpuTime" : 1171756284,
+ "executorRunTime" : 74,
+ "executorCpuTime" : 65263482,
+ "resultSize" : 840,
+ "jvmGcTime" : 0,
+ "resultSerializationTime" : 1,
+ "memoryBytesSpilled" : 0,
+ "diskBytesSpilled" : 0,
+ "peakExecutionMemory" : 0,
+ "inputMetrics" : {
+ "bytesRead" : 0,
+ "recordsRead" : 0
+ },
+ "outputMetrics" : {
+ "bytesWritten" : 0,
+ "recordsWritten" : 0
+ },
+ "shuffleReadMetrics" : {
+ "remoteBlocksFetched" : 0,
+ "localBlocksFetched" : 0,
+ "fetchWaitTime" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
+ "localBytesRead" : 0,
+ "recordsRead" : 0
+ },
+ "shuffleWriteMetrics" : {
+ "bytesWritten" : 0,
+ "writeTime" : 0,
+ "recordsWritten" : 0
+ }
+ },
+ "executorLogs" : {
+ "stdout" : "http://ltx1-hcl12291.grid.linkedin.com:8042/node/containerlogs/container_e18_1628109047826_1317105_01_000008/vsowrira/stdout?start=-4096",
Review comment:
Fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-916479910
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47633/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-925340652
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48024/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-926859915
**[Test build #143612 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143612/testReport)** for PR 33253 at commit [`ac1659e`](https://github.com/apache/spark/commit/ac1659e156eca5899e1eff765698c9986eec5d4c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-876879218
**[Test build #140832 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140832/testReport)** for PR 33253 at commit [`0593c30`](https://github.com/apache/spark/commit/0593c309c16a7a2de1422f459d2f4a50194f78c3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-916333912
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47628/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-876981018
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45352/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Basic speculation metrics summary for a stage
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-876384811
**[Test build #140800 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140800/testReport)** for PR 33253 at commit [`6f18a51`](https://github.com/apache/spark/commit/6f18a51f9330ace609f14969b9d02966c02106b6).
* This patch **fails RAT tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-886231930
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46125/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-876879436
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140832/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-882741810
**[Test build #141270 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141270/testReport)** for PR 33253 at commit [`606b3a4`](https://github.com/apache/spark/commit/606b3a40b791475dd71243a8d20425f0d7ddc291).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-877466668
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45379/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r675343104
##########
File path: core/src/main/scala/org/apache/spark/status/api/v1/api.scala
##########
@@ -89,6 +89,14 @@ class ExecutorStageSummary private[spark](
val peakMemoryMetrics: Option[ExecutorMetrics],
val isExcludedForStage: Boolean)
+class SpeculationStageSummary private[spark](
Review comment:
Do you think we don't need this anymore given the REST API is not required anymore? Not sure if this can be removed even after getting rid of the new speculationSummary API. Is it possible?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-926903690
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48124/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-926903661
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48124/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-882741810
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-880224087
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45548/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-926903690
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48124/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-916427622
**[Test build #143127 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143127/testReport)** for PR 33253 at commit [`24ecae3`](https://github.com/apache/spark/commit/24ecae34ae13ce6e27d27a5e44abf80498045986).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r688209434
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -1208,6 +1232,33 @@ private[spark] class AppStatusListener(
}
}
+ private def killedTaskSummaryForSpeculationStageSummary(
+ reason: TaskEndReason,
+ oldSummary: Map[String, Int],
+ isSpeculative: Boolean): Map[String, Int] = {
+ reason match {
+ case k: TaskKilled if k.reason.contains("another attempt succeeded") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary.updated("speculated attempt succeeded",
+ oldSummary.getOrElse("speculated attempt succeeded", 0) + 1)
+ }
+ // If the stage is finished and speculative tasks get killed, then the
+ // kill reason is "stage finished"
+ case k: TaskKilled if k.reason.contains("Stage finished") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary
Review comment:
Sorry for the late reply.
> I don't think we need special handling for zombie tasks.
Yes, I don't think we need to handle zombie tasks too. What I mean is that, if the reason is `Stage finished`, the killed task can be one of the following.
(1) Speculative attempt of a task
(2) A task which has speculative attempt
(3) A task which neither is speculative nor has speculative, but is zombie
For (1), it is handled [here](https://github.com/apache/spark/pull/33253/files#diff-0d63c25e93eb1fbec25754f0bee7714c7e3c1538bc7370d94b147524db5c1f63R1251).
I don't think we need to handle the case (3) but should we handle (2) in the [else block](https://github.com/apache/spark/pull/33253/files#diff-0d63c25e93eb1fbec25754f0bee7714c7e3c1538bc7370d94b147524db5c1f63R1255) somehow?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-877432095
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140868/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-901272772
@sarutak Gentle reminder. I think we have only one last pending comment from you. I am hoping to resolve it soon and get the changes merged. Thanks!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-890288643
**[Test build #141917 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141917/testReport)** for PR 33253 at commit [`0bba76f`](https://github.com/apache/spark/commit/0bba76fd11f14412eca3268e8c775a944ce6a378).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Basic speculation metrics summary for a stage
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-876129312
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #33253: [SPARK-36038][CORE] Basic speculation metrics summary for a stage
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-876370996
ok to test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-885886478
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46089/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-886252329
**[Test build #141609 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141609/testReport)** for PR 33253 at commit [`8b3ce9b`](https://github.com/apache/spark/commit/8b3ce9bc0119a5dda0a29ebda10197bbd504a38b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-926182151
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48076/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-963372486
@sarutak Oh. my bad. I think we have fix for this internally. In the refactoring, I probably missed to incorporate that check. @thejdeep would help fix the issue and post a new PR. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-962913353
@venkata91 Sorry, but I'll revert this change. I've noticed that a stage page will stuck if we we use HistoryServer and the corresponding stage contains no speculative task.
Please check it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-876962291
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45352/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33253: [SPARK-36038][CORE] Basic speculation metrics summary for a stage
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r665884822
##########
File path: core/src/main/resources/org/apache/spark/ui/static/stagepage.js
##########
@@ -162,29 +162,29 @@ function displayRowsForSummaryMetricsTable(row, type, columnIndex) {
str = formatBytes(row.data.bytesWritten[columnIndex], type) + " / " +
row.data.recordsWritten[columnIndex];
return str;
-
+
Review comment:
Pls remove unrelated change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-879629379
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45509/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r675766548
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala
##########
@@ -524,6 +529,11 @@ private[spark] class AppStatusStore(
} else {
None
}
+ val speculationStageSummary: Option[v1.SpeculationStageSummary] = if (withSummaries) {
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-880208807
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45548/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-882741810
**[Test build #141270 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141270/testReport)** for PR 33253 at commit [`606b3a4`](https://github.com/apache/spark/commit/606b3a40b791475dd71243a8d20425f0d7ddc291).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-879611464
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45509/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-882743479
> @venkata91
>
> > it seems like something is wrong with the Github actions on my account therefore the tests are not able to run. I filed a support request with Github but they haven't still figured out why it is not working
>
> Yes, please resolve the GA issue on your account. Once GA on your account works, you can re-run jobs by yourself.
>
> Also, I said that the CI failed by checking [here](https://github.com/apache/spark/pull/33253#issuecomment-880226935).
Yeah thanks for pointing out that. I thought it was due to some other issues after looking at the exception as the tests on that page were successful. After looking at the core tests only realized there were failures in HistoryServerSuite. Fixed it now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33253: [SPARK-36038][CORE] Basic speculation metrics summary for a stage
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-876384323
**[Test build #140800 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140800/testReport)** for PR 33253 at commit [`6f18a51`](https://github.com/apache/spark/commit/6f18a51f9330ace609f14969b9d02966c02106b6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-885819487
**[Test build #141571 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141571/testReport)** for PR 33253 at commit [`be7131f`](https://github.com/apache/spark/commit/be7131fd6801e50d71f6af4cd21c4e388e3c6be8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-916339443
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47628/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-916454925
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47631/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r688678951
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -1208,6 +1232,33 @@ private[spark] class AppStatusListener(
}
}
+ private def killedTaskSummaryForSpeculationStageSummary(
+ reason: TaskEndReason,
+ oldSummary: Map[String, Int],
+ isSpeculative: Boolean): Map[String, Int] = {
+ reason match {
+ case k: TaskKilled if k.reason.contains("another attempt succeeded") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary.updated("speculated attempt succeeded",
+ oldSummary.getOrElse("speculated attempt succeeded", 0) + 1)
+ }
+ // If the stage is finished and speculative tasks get killed, then the
+ // kill reason is "stage finished"
+ case k: TaskKilled if k.reason.contains("Stage finished") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary
Review comment:
@sarutak Do you mean handling the case where the task that got killed has a speculative attempt but got killed because of the speculative attempt succeeded? I think this probably would be difficult to address as the stage is finished and all the in memory data structures would be cleaned up, only the `AppStatusStore` would have the information. I don't think it is worth the effort to handle that checking `AppStatusStore`. In that case, the `speculation attempt succeeded` count can be one less, it is not a huge discrepancy IMO. But in the case of zombie task attempts the counters can get completely wrong. May be we can reset the counters to zero in the case of failed/killed task attempts. Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-887732237
**[Test build #141730 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141730/testReport)** for PR 33253 at commit [`f955caa`](https://github.com/apache/spark/commit/f955caab06533bbc5470574b43577779a863dd90).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-896484057
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142302/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-882532843
@venkata91
> it seems like something is wrong with the Github actions on my account therefore the tests are not able to run. I filed a support request with Github but they haven't still figured out why it is not working
Yes, please resolve the GA issue on your account. Once GA on your account works, you can re-run jobs by yourself.
Also, I said that the CI failed by checking [here](https://github.com/apache/spark/pull/33253#issuecomment-880226935).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-877431762
**[Test build #140868 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140868/testReport)** for PR 33253 at commit [`cdef2eb`](https://github.com/apache/spark/commit/cdef2eb34703b0ae3cc6d68cc0798006cc2b4af1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-879580409
**[Test build #140995 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140995/testReport)** for PR 33253 at commit [`77920b8`](https://github.com/apache/spark/commit/77920b86a9acdff45d9d35e3179b9daafe3bb84f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-882850320
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141270/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-886016909
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46114/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-890284764
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46426/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-885821250
> > > Ah I see, currently we are populating killedTasksSummary in SpeculationStageSummary only if the killed reason is another attempt succeeded. For other reasons, we are not populating it since this is mainly specific to speculation to understand how many speculated tasks killed vs original tasks killed. So yeah killedTasksSummary can be empty. What do you think should be the right behavior? Also
> >
> >
> > Even if another attempt succeeds, `killedTasksSummary` can be empty when a speculative task is the last task in a job.
> > In this case, the killed reason is `Stage finished`.
> > https://github.com/apache/spark/blob/2fe12a75206d4dbef6d7678b876c16876136cdd0/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1661-L1675
>
> Ah interesting. This is a weird corner case. In this case, if the `TaskKilled` reason is `stage finished` for a speculative task then we increment `original attempt succeeded` counter. That should do it right?
Addressed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Basic speculation metrics summary for a stage
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-876469513
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45312/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #33253: [SPARK-36038][CORE] Basic speculation metrics summary for a stage
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r666113548
##########
File path: core/src/test/scala/org/apache/spark/status/AppStatusStoreSuite.scala
##########
@@ -204,4 +255,15 @@ class AppStatusStoreSuite extends SparkFunSuite {
taskMetrics.shuffleWriteMetrics.incRecordsWritten(i)
taskMetrics
}
+
+ private def newSpeculationSummaryData(
+ stageId: Int,
+ stageAttemptId: Int): SpeculationStageSummaryWrapper = {
+ val speculationStageSummary = new SpeculationStageSummary(10, 2, 5, 1, 2,
+ Map("original attempt succeeded" -> 1,
+ "speculation attempt succeeded" -> 1
+ ))
+ new SpeculationStageSummaryWrapper(stageId, stageAttemptId, speculationStageSummary)
+ }
+
Review comment:
ditto.
##########
File path: core/src/test/resources/HistoryServerExpectations/stage_with_speculation_summary_expectation.json
##########
@@ -0,0 +1,7110 @@
+[ {
Review comment:
Why does this file need to be +7000 lines?
Currently, `stage_task_list_w__offset___length_expectation.json` has the maximum number of lines among files under `HistoryServerExpectations` and it's 2451 lines.
Can you minimize this file?
##########
File path: core/src/main/scala/org/apache/spark/status/LiveEntity.scala
##########
@@ -392,6 +392,31 @@ private class LiveExecutorStageSummary(
}
+private class LiveSpeculationStageSummary(
+ stageId: Int,
+ attemptId: Int) extends LiveEntity {
+
+ var numTasks = 0
+ var numActiveTasks = 0
+ var numCompletedTasks = 0
+ var numFailedTasks = 0
+ var numKilledTasks = 0
+ var killedTasksSummary : Map[String, Int] = Map()
+
+ override protected def doUpdate(): Any = {
+ val info = new v1.SpeculationStageSummary(
+ numTasks,
+ numActiveTasks,
+ numCompletedTasks,
+ numFailedTasks,
+ numKilledTasks,
+ killedTasksSummary
+ )
+ new SpeculationStageSummaryWrapper(stageId, attemptId, info)
+ }
+
Review comment:
Please remove unnecessary blank line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-916426671
**[Test build #143127 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143127/testReport)** for PR 33253 at commit [`24ecae3`](https://github.com/apache/spark/commit/24ecae34ae13ce6e27d27a5e44abf80498045986).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-926859915
**[Test build #143612 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143612/testReport)** for PR 33253 at commit [`ac1659e`](https://github.com/apache/spark/commit/ac1659e156eca5899e1eff765698c9986eec5d4c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-886216194
**[Test build #141608 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141608/testReport)** for PR 33253 at commit [`570e241`](https://github.com/apache/spark/commit/570e2414c7a2bf9473fcc60775e4df15ca4881cd).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Basic speculation metrics summary for a stage
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-876469513
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45312/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Basic speculation metrics summary for a stage
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-876384838
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140800/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-930405537
@sarutak Addressed both of your comments. Thanks for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #33253: [SPARK-36038][CORE] Basic speculation metrics summary for a stage
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-876498283
cc @jiangxb1987
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-882743479
> @venkata91
>
> > it seems like something is wrong with the Github actions on my account therefore the tests are not able to run. I filed a support request with Github but they haven't still figured out why it is not working
>
> Yes, please resolve the GA issue on your account. Once GA on your account works, you can re-run jobs by yourself.
>
> Also, I said that the CI failed by checking [here](https://github.com/apache/spark/pull/33253#issuecomment-880226935).
Yeah thanks for pointing out that. I thought it was due to some other issues after looking at the exception as the tests on that page were successful. After looking at the core tests only realized there were failures in HistoryServerSuite. Fixed it now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-879563645
@venkata91 Could you fix the style issue first?
```
[error] /home/runner/work/spark/spark/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala:195: File line length exceeds 100 characters
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-932014324
@venkata91
BTW, could you update the doc for WebUI to explain this feature, or just file a JIRA?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-932005869
@venkata91 Let me remove `cc @AngersZhuuuu @ron8hu` from the desription.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-886019505
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141597/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-876873242
> As @AngersZhuuuu suggests, please show the screenshot from which we can confirm the change.
> Also, please enable GitHub Actions on your repository.
@sarutak It seems like Github actions is disabled for my fork. I have requested them to enable it. Just curious, why do I need to enable it for my fork (repo)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-930481813
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r680625508
##########
File path: core/src/main/scala/org/apache/spark/status/api/v1/api.scala
##########
@@ -89,6 +89,14 @@ class ExecutorStageSummary private[spark](
val peakMemoryMetrics: Option[ExecutorMetrics],
val isExcludedForStage: Boolean)
+class SpeculationStageSummary private[spark](
Review comment:
> Speculation summary can be at different level right like job, app, stage etc.
What do you mean? Speculation is about task.
http://spark.apache.org/docs/latest/configuration.html#scheduling
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r675343104
##########
File path: core/src/main/scala/org/apache/spark/status/api/v1/api.scala
##########
@@ -89,6 +89,14 @@ class ExecutorStageSummary private[spark](
val peakMemoryMetrics: Option[ExecutorMetrics],
val isExcludedForStage: Boolean)
+class SpeculationStageSummary private[spark](
Review comment:
Do you think we don't need this anymore given the REST API is not required anymore?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r703743424
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -1208,6 +1232,33 @@ private[spark] class AppStatusListener(
}
}
+ private def killedTaskSummaryForSpeculationStageSummary(
+ reason: TaskEndReason,
+ oldSummary: Map[String, Int],
+ isSpeculative: Boolean): Map[String, Int] = {
+ reason match {
+ case k: TaskKilled if k.reason.contains("another attempt succeeded") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary.updated("speculated attempt succeeded",
+ oldSummary.getOrElse("speculated attempt succeeded", 0) + 1)
+ }
+ // If the stage is finished and speculative tasks get killed, then the
+ // kill reason is "stage finished"
+ case k: TaskKilled if k.reason.contains("Stage finished") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary
Review comment:
@sarutak Gentle reminder. This has been pending for 2 months now. It would be great if you can help me here. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r697670769
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -1208,6 +1232,33 @@ private[spark] class AppStatusListener(
}
}
+ private def killedTaskSummaryForSpeculationStageSummary(
+ reason: TaskEndReason,
+ oldSummary: Map[String, Int],
+ isSpeculative: Boolean): Map[String, Int] = {
+ reason match {
+ case k: TaskKilled if k.reason.contains("another attempt succeeded") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary.updated("speculated attempt succeeded",
+ oldSummary.getOrElse("speculated attempt succeeded", 0) + 1)
+ }
+ // If the stage is finished and speculative tasks get killed, then the
+ // kill reason is "stage finished"
+ case k: TaskKilled if k.reason.contains("Stage finished") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary
Review comment:
Please give me some time to think about it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-877432077
**[Test build #140868 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140868/testReport)** for PR 33253 at commit [`cdef2eb`](https://github.com/apache/spark/commit/cdef2eb34703b0ae3cc6d68cc0798006cc2b4af1).
* This patch **fails RAT tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-882741810
**[Test build #141270 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141270/testReport)** for PR 33253 at commit [`606b3a4`](https://github.com/apache/spark/commit/606b3a40b791475dd71243a8d20425f0d7ddc291).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-879598200
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45509/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r693123565
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -1208,6 +1232,33 @@ private[spark] class AppStatusListener(
}
}
+ private def killedTaskSummaryForSpeculationStageSummary(
+ reason: TaskEndReason,
+ oldSummary: Map[String, Int],
+ isSpeculative: Boolean): Map[String, Int] = {
+ reason match {
+ case k: TaskKilled if k.reason.contains("another attempt succeeded") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary.updated("speculated attempt succeeded",
+ oldSummary.getOrElse("speculated attempt succeeded", 0) + 1)
+ }
+ // If the stage is finished and speculative tasks get killed, then the
+ // kill reason is "stage finished"
+ case k: TaskKilled if k.reason.contains("Stage finished") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary
Review comment:
> Do you mean handling the case where the task that got killed has a speculative attempt but got killed because of the speculative attempt succeeded?
Yes, but I understand it's difficult to know precisely.
I don't think that resetting the counter is a good idea as it can confuse users.
My main concern here is that users are confused by the wrongly incremented counter. So one possible compromise is to have note that the counter can be wrongly incremented in the UI document.
I noticed that `if` block here can also wrongly increment the counter because a speculative task can be zombie.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-876879420
**[Test build #140832 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140832/testReport)** for PR 33253 at commit [`0593c30`](https://github.com/apache/spark/commit/0593c309c16a7a2de1422f459d2f4a50194f78c3).
* This patch **fails RAT tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-880227726
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141031/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-885405263
> Ah I see, currently we are populating killedTasksSummary in SpeculationStageSummary only if the killed reason is another attempt succeeded. For other reasons, we are not populating it since this is mainly specific to speculation to understand how many speculated tasks killed vs original tasks killed. So yeah killedTasksSummary can be empty. What do you think should be the right behavior? Also
Even if another attempt succeeds, `killedTasksSummary` can be empty when a speculative task is the last task in a job.
In this case, the killed reason is `Stage finished`.
https://github.com/apache/spark/blob/2fe12a75206d4dbef6d7678b876c16876136cdd0/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1661-L1675
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-916450532
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47631/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-876950853
@venkata91 I checked out your change and run it on GA on my account but it fails to build.
```
[error] /home/runner/work/spark/spark/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala: illegal start of simple expression: Token(RPAREN,),10124,))
```
Could you enable your GA?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-876917577
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45343/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-877038535
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-880227726
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141031/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-886231930
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46125/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r676860540
##########
File path: core/src/main/scala/org/apache/spark/status/api/v1/api.scala
##########
@@ -89,6 +89,14 @@ class ExecutorStageSummary private[spark](
val peakMemoryMetrics: Option[ExecutorMetrics],
val isExcludedForStage: Boolean)
+class SpeculationStageSummary private[spark](
Review comment:
Speculation summary can be at different level right like job, app, stage etc. which is why I named this `SpeculationStageSummary`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Basic speculation metrics summary for a stage
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-876384838
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140800/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-925244423
**[Test build #143513 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143513/testReport)** for PR 33253 at commit [`66dcaa4`](https://github.com/apache/spark/commit/66dcaa4c291b1ea550e2f4871e24c1ae3565b25b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-885415899
> > Ah I see, currently we are populating killedTasksSummary in SpeculationStageSummary only if the killed reason is another attempt succeeded. For other reasons, we are not populating it since this is mainly specific to speculation to understand how many speculated tasks killed vs original tasks killed. So yeah killedTasksSummary can be empty. What do you think should be the right behavior? Also
>
> Even if another attempt succeeds, `killedTasksSummary` can be empty when a speculative task is the last task in a job.
> In this case, the killed reason is `Stage finished`.
> https://github.com/apache/spark/blob/2fe12a75206d4dbef6d7678b876c16876136cdd0/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1661-L1675
Ah interesting. This is a weird corner case. In this case, if the `TaskKilled` reason is `stage finished` for a speculative task then we increment `original attempt succeeded` counter. That should do it right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-882532843
@venkata91
> it seems like something is wrong with the Github actions on my account therefore the tests are not able to run. I filed a support request with Github but they haven't still figured out why it is not working
Yes, please resolve the GA issue on your account. Once GA on your account works, you can re-run jobs by yourself.
Also, I said that the CI failed by checking [here](https://github.com/apache/spark/pull/33253#issuecomment-880226935).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Basic speculation metrics summary for a stage
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-876384323
**[Test build #140800 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140800/testReport)** for PR 33253 at commit [`6f18a51`](https://github.com/apache/spark/commit/6f18a51f9330ace609f14969b9d02966c02106b6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-916426671
**[Test build #143127 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143127/testReport)** for PR 33253 at commit [`24ecae3`](https://github.com/apache/spark/commit/24ecae34ae13ce6e27d27a5e44abf80498045986).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-917695032
**[Test build #143180 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143180/testReport)** for PR 33253 at commit [`9302892`](https://github.com/apache/spark/commit/93028927fd0233930c8ffcf8bface40c7c5903c2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-882814110
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45784/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r705516760
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -1208,6 +1232,33 @@ private[spark] class AppStatusListener(
}
}
+ private def killedTaskSummaryForSpeculationStageSummary(
+ reason: TaskEndReason,
+ oldSummary: Map[String, Int],
+ isSpeculative: Boolean): Map[String, Int] = {
+ reason match {
+ case k: TaskKilled if k.reason.contains("another attempt succeeded") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary.updated("speculated attempt succeeded",
+ oldSummary.getOrElse("speculated attempt succeeded", 0) + 1)
+ }
+ // If the stage is finished and speculative tasks get killed, then the
+ // kill reason is "stage finished"
+ case k: TaskKilled if k.reason.contains("Stage finished") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary
Review comment:
That makes sense too. Even in our internal we don't really use these counters much. I can remove this in that case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-926223897
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143567/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-926147566
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48076/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-887732237
**[Test build #141730 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141730/testReport)** for PR 33253 at commit [`f955caa`](https://github.com/apache/spark/commit/f955caab06533bbc5470574b43577779a863dd90).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-925340601
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48024/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r680634284
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -1208,6 +1232,33 @@ private[spark] class AppStatusListener(
}
}
+ private def killedTaskSummaryForSpeculationStageSummary(
+ reason: TaskEndReason,
+ oldSummary: Map[String, Int],
+ isSpeculative: Boolean): Map[String, Int] = {
+ reason match {
+ case k: TaskKilled if k.reason.contains("another attempt succeeded") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary.updated("speculated attempt succeeded",
+ oldSummary.getOrElse("speculated attempt succeeded", 0) + 1)
+ }
+ // If the stage is finished and speculative tasks get killed, then the
+ // kill reason is "stage finished"
+ case k: TaskKilled if k.reason.contains("Stage finished") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary
Review comment:
> Hmm, it seems difficult to judge that one attempt succeeded because the other fails (Both attempts can fail).
Take back my comment. `Stage finished` is the reason when a task killed but the stage which the killed task belongs to successfully finished.
https://github.com/apache/spark/blob/2fe12a75206d4dbef6d7678b876c16876136cdd0/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1641-L1675
But, the killed task can be zombie task.
See #22771
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-886216194
**[Test build #141608 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141608/testReport)** for PR 33253 at commit [`570e241`](https://github.com/apache/spark/commit/570e2414c7a2bf9473fcc60775e4df15ca4881cd).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-890273447
**[Test build #141917 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141917/testReport)** for PR 33253 at commit [`0bba76f`](https://github.com/apache/spark/commit/0bba76fd11f14412eca3268e8c775a944ce6a378).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-926182128
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48076/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-926120535
**[Test build #143567 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143567/testReport)** for PR 33253 at commit [`1e42d59`](https://github.com/apache/spark/commit/1e42d59d173b02db430f016c95842d9566c6833d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-926944613
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143612/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-879579744
> @venkata91 Could you fix the style issue first?
>
> ```
> [error] /home/runner/work/spark/spark/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala:195: File line length exceeds 100 characters
> ```
Yeah fixed it. Also now Github actions seem to run which is good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-880226935
**[Test build #141031 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141031/testReport)** for PR 33253 at commit [`7d15e5b`](https://github.com/apache/spark/commit/7d15e5b291352a6ddd76ab983cb64cb191f43047).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-886262328
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46126/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-877466668
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45379/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-879580591
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140995/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-886018157
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46114/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-886265014
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46126/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-877452053
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45379/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-882814110
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Basic speculation metrics summary for a stage
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-876441615
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45312/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE]Basic speculation metrics summary for a stage
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-876129312
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-896465136
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-880156016
**[Test build #141031 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141031/testReport)** for PR 33253 at commit [`7d15e5b`](https://github.com/apache/spark/commit/7d15e5b291352a6ddd76ab983cb64cb191f43047).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-882802284
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45784/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r675304991
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala
##########
@@ -524,6 +529,11 @@ private[spark] class AppStatusStore(
} else {
None
}
+ val speculationStageSummary: Option[v1.SpeculationStageSummary] = if (withSummaries) {
Review comment:
How about check the condition of `withDetail` instead of `withSummaries` like `executorSummaries? Then, the new REST API `/speculationSummary` is not needed because `withDetail` is `true` by default when we access to a path of the form of `/stage/stageId/attemptId`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-932006516
Mergint to `master`. Thank you @venkata91 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r688281638
##########
File path: core/src/main/resources/org/apache/spark/ui/static/stagespage-template.html
##########
@@ -31,10 +31,25 @@ <h4 id="summaryMetricsTitle" class="title-table"></h4>
</tbody>
</table>
</div>
+ <h4 id="speculationSummaryTitle" class="title-table">Speculation Summary</h4>
+ <div id="speculationSummary" class="container-fluid">
+ <table id="speculation-metrics-table" class="table table-striped compact table-dataTable cell-border">
+ <thead>
+ <th>Total</th>
+ <th>Active</th>
+ <th>Complete</th>
+ <th>Failed</th>
+ <th>Killed</th>
+ </thead>
+ <tbody>
+ </tbody>
+ </table>
+ </div>
+ </div>
<h4 id="aggregatedMetrics" class="collapse-table">
<span class="expand-input-rate-arrow arrow-closed" id="arrowtoggle2"></span>
<a class="title-table">Aggregated Metrics by Executor</a>
- </h4>
+ </h4>
Review comment:
remove this space
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r682624830
##########
File path: core/src/test/resources/HistoryServerExpectations/stage_with_speculation_summary_expectation.json
##########
@@ -0,0 +1,748 @@
+{
Review comment:
It just seems redundant. If it's difficult to reduce it, we can replace it later with a small one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-886257004
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46126/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r680634284
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -1208,6 +1232,33 @@ private[spark] class AppStatusListener(
}
}
+ private def killedTaskSummaryForSpeculationStageSummary(
+ reason: TaskEndReason,
+ oldSummary: Map[String, Int],
+ isSpeculative: Boolean): Map[String, Int] = {
+ reason match {
+ case k: TaskKilled if k.reason.contains("another attempt succeeded") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary.updated("speculated attempt succeeded",
+ oldSummary.getOrElse("speculated attempt succeeded", 0) + 1)
+ }
+ // If the stage is finished and speculative tasks get killed, then the
+ // kill reason is "stage finished"
+ case k: TaskKilled if k.reason.contains("Stage finished") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary
Review comment:
> Hmm, it seems difficult to judge that one attempt succeeded because the other fails (Both attempts can fail).
Take back my comment. `Stage finished` is the reason when a task killed but the stage which the killed task belongs to successfully finished.
https://github.com/apache/spark/blob/2fe12a75206d4dbef6d7678b876c16876136cdd0/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1641-L1675
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r688678951
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -1208,6 +1232,33 @@ private[spark] class AppStatusListener(
}
}
+ private def killedTaskSummaryForSpeculationStageSummary(
+ reason: TaskEndReason,
+ oldSummary: Map[String, Int],
+ isSpeculative: Boolean): Map[String, Int] = {
+ reason match {
+ case k: TaskKilled if k.reason.contains("another attempt succeeded") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary.updated("speculated attempt succeeded",
+ oldSummary.getOrElse("speculated attempt succeeded", 0) + 1)
+ }
+ // If the stage is finished and speculative tasks get killed, then the
+ // kill reason is "stage finished"
+ case k: TaskKilled if k.reason.contains("Stage finished") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary
Review comment:
@sarutak Do you mean handling the case where the task that got killed has a speculative attempt but got killed because of the speculative attempt succeeded? I think this probably would be difficult to address as the stage is finished and all the in memory data structures would be cleaned up, only the `AppStatusStore` would have the information.
I don't think it is worth the effort to handle that checking `AppStatusStore`. In that case, the `speculation attempt succeeded` count can be one less, it is not a huge discrepancy IMO.
But in the case of zombie task attempts the counters can get completely wrong. May be we can reset the counters to zero in the case of failed/killed task attempts. Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r669284266
##########
File path: core/src/test/resources/HistoryServerExpectations/running_app_list_json_expectation.json
##########
@@ -1 +1 @@
-[ ]
+[ ]
Review comment:
Unnecessary change here.
##########
File path: core/src/test/resources/spark-events/application_1625351839633_843287
##########
@@ -0,0 +1,37 @@
+{"Event":"SparkListenerLogStart","Spark Version":"3.3.0-SNAPSHOT"}
Review comment:
This file need to be added to `.rat-exclude`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-886272470
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141609/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Basic speculation metrics summary for a stage
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-876563353
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-882008124
> Some tests seem to fail. Could you fix them?
@sarutak it seems like something is wrong with the Github actions on my account therefore the tests are not able to run. I filed a support request with Github but they haven't still figured out why it is not working. Is there any other way to run the tests the previous PR builder jenkins
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-896427344
**[Test build #142302 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142302/testReport)** for PR 33253 at commit [`1021b73`](https://github.com/apache/spark/commit/1021b73340a313e1c4a6ddaee8079f9651bf095a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-884464753
@venkata91 I checked out this change and accessed `/speculationSummary` but `killedTasksSummary` seems empty.
Could you investigate the cause?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-886011306
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46114/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-916403607
**[Test build #143124 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143124/testReport)** for PR 33253 at commit [`42890bb`](https://github.com/apache/spark/commit/42890bb4108723cb3af92f1f9460117d6c969257).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-916427639
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143127/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-896465136
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46809/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-896480094
**[Test build #142302 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142302/testReport)** for PR 33253 at commit [`1021b73`](https://github.com/apache/spark/commit/1021b73340a313e1c4a6ddaee8079f9651bf095a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-896459643
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46809/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-876879218
**[Test build #140832 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140832/testReport)** for PR 33253 at commit [`0593c30`](https://github.com/apache/spark/commit/0593c309c16a7a2de1422f459d2f4a50194f78c3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-876563353
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-925299166
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48024/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-930512972
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143724/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-930403786
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r676861017
##########
File path: core/src/main/scala/org/apache/spark/status/storeTypes.scala
##########
@@ -399,6 +399,21 @@ private[spark] class ExecutorStageSummaryWrapper(
}
+private[spark] class SpeculationStageSummaryWrapper(
Review comment:
how about this one does this one need to be removed? no I guess?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r681200947
##########
File path: core/src/main/scala/org/apache/spark/status/api/v1/api.scala
##########
@@ -89,6 +89,14 @@ class ExecutorStageSummary private[spark](
val peakMemoryMetrics: Option[ExecutorMetrics],
val isExcludedForStage: Boolean)
+class SpeculationStageSummary private[spark](
Review comment:
I understand speculation is about a task. What I mean here is we can have the summary of speculated tasks aggregated for stage level or job or app level which is why I named the class as `SpeculationStageSummary` so that it will be clear that it is specific to the stage. Hope that clarifies it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-926944613
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143612/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-886238821
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141608/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-930403786
**[Test build #143724 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143724/testReport)** for PR 33253 at commit [`9c13307`](https://github.com/apache/spark/commit/9c13307d41da10b1def5db082f3720fc1f88fd4d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-925251604
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143513/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-876907600
> @venkata91
> > @sarutak It seems like Github actions is disabled for my fork. I have requested them to enable it. Just curious, why do I need to enable it for my fork (repo)?
>
> It's to run tests.
>
> BTW, please show the screenshot in the description rather than conversation.
I did link it in the description as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-925244423
**[Test build #143513 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143513/testReport)** for PR 33253 at commit [`66dcaa4`](https://github.com/apache/spark/commit/66dcaa4c291b1ea550e2f4871e24c1ae3565b25b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak edited a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak edited a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-962913353
@venkata91 Sorry, but I'll revert this change. I've noticed that a stage page will stuck if we use HistoryServer and the corresponding stage contains no speculative task.
Please check it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r717193191
##########
File path: core/src/test/scala/org/apache/spark/status/AppStatusStoreSuite.scala
##########
@@ -136,6 +139,51 @@ class AppStatusStoreSuite extends SparkFunSuite {
}
}
+ test("speculation summary") {
Review comment:
Please add JIRA ID for each added test title like `SPARK-36038: speculation summary`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-917679130
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47683/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-917677641
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47683/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-886019505
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141597/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-877402668
@venkata91 Sorry if I'm confusing you. I told you two things (your change seems to fail to build and please enable GA on your account to run test).
The failure and that GA is disabled are not relevant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-880156016
**[Test build #141031 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141031/testReport)** for PR 33253 at commit [`7d15e5b`](https://github.com/apache/spark/commit/7d15e5b291352a6ddd76ab983cb64cb191f43047).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-916415874
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143124/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r686426357
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -1208,6 +1232,33 @@ private[spark] class AppStatusListener(
}
}
+ private def killedTaskSummaryForSpeculationStageSummary(
+ reason: TaskEndReason,
+ oldSummary: Map[String, Int],
+ isSpeculative: Boolean): Map[String, Int] = {
+ reason match {
+ case k: TaskKilled if k.reason.contains("another attempt succeeded") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary.updated("speculated attempt succeeded",
+ oldSummary.getOrElse("speculated attempt succeeded", 0) + 1)
+ }
+ // If the stage is finished and speculative tasks get killed, then the
+ // kill reason is "stage finished"
+ case k: TaskKilled if k.reason.contains("Stage finished") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary
Review comment:
@sarutak Gentle reminder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-887844111
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-886272470
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141609/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-879580409
**[Test build #140995 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140995/testReport)** for PR 33253 at commit [`77920b8`](https://github.com/apache/spark/commit/77920b86a9acdff45d9d35e3179b9daafe3bb84f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-876917577
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45343/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-896447486
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46809/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-885893957
**[Test build #141571 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141571/testReport)** for PR 33253 at commit [`be7131f`](https://github.com/apache/spark/commit/be7131fd6801e50d71f6af4cd21c4e388e3c6be8).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-887682089
> I think this class is still needed.
> How about SpeculationSummary instead of SpeculationStageSummary?
> Speculation is not about stages but tasks.
@sarutak Speculation summary can be at different level right like job, app, stage etc. which is why I named this SpeculationStageSummary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-879629379
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45509/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-877411510
> @venkata91 Sorry if I'm confusing you. I told you two things (your change seems to fail to build and please enable GA on your account to run test).
> The failure and that GA is disabled are not relevant.
@sarutak I think I figured out the issue, there was a `,` after the last case (`stage with speculation summary`) - fixed it. I guess this should solve the issue. Somehow it didn't fail on my local though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-926937820
**[Test build #143612 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143612/testReport)** for PR 33253 at commit [`ac1659e`](https://github.com/apache/spark/commit/ac1659e156eca5899e1eff765698c9986eec5d4c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class SkewJoinChildWrapper(plan: SparkPlan) extends LeafExecNode `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-930481813
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48235/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r705517784
##########
File path: core/src/main/scala/org/apache/spark/status/LiveEntity.scala
##########
@@ -392,6 +392,30 @@ private class LiveExecutorStageSummary(
}
+private class LiveSpeculationStageSummary(
+ stageId: Int,
+ attemptId: Int) extends LiveEntity {
+
+ var numTasks = 0
+ var numActiveTasks = 0
+ var numCompletedTasks = 0
+ var numFailedTasks = 0
+ var numKilledTasks = 0
+ var killedTasksSummary : Map[String, Int] = Map()
Review comment:
As we have agreed to remove the `killedTasksSummary` for speculation specifically, this comment is not relevant anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-916454434
**[Test build #143129 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143129/testReport)** for PR 33253 at commit [`a57626f`](https://github.com/apache/spark/commit/a57626fa904b7863e656da70ee946617b7de6705).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-930436091
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48235/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-926223897
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143567/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-930403786
**[Test build #143724 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143724/testReport)** for PR 33253 at commit [`9c13307`](https://github.com/apache/spark/commit/9c13307d41da10b1def5db082f3720fc1f88fd4d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-916301987
**[Test build #143124 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143124/testReport)** for PR 33253 at commit [`42890bb`](https://github.com/apache/spark/commit/42890bb4108723cb3af92f1f9460117d6c969257).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-880188286
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45548/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-890443720
Gentle reminder @sarutak @AngersZhuuuu Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-882814110
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45784/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-876940801
**[Test build #140841 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140841/testReport)** for PR 33253 at commit [`be65ed1`](https://github.com/apache/spark/commit/be65ed153015c4bd3c97ca54cdb567cc787e69fe).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-881910368
Some tests seem to fail. Could you fix it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-877432095
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140868/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r669897745
##########
File path: core/src/main/resources/org/apache/spark/ui/static/stagespage-template.html
##########
@@ -31,6 +31,20 @@ <h4 id="summaryMetricsTitle" class="title-table"></h4>
</tbody>
</table>
</div>
+ <h4 id="speculationSummaryTitle" class="title-table">Speculation Summary</h4>
Review comment:
Addressed it by checking if `numSpeculatedTasks > 0` if so `show`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-887837336
**[Test build #141730 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141730/testReport)** for PR 33253 at commit [`f955caa`](https://github.com/apache/spark/commit/f955caab06533bbc5470574b43577779a863dd90).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-879141226
> > @venkata91 I checked out your change and run it on GA on my account but it fails to build.
> > ```
> > [error] /home/runner/work/spark/spark/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala: illegal start of simple expression: Token(RPAREN,),10124,))
> > ```
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Could you enable your GA?
>
> @sarutak Currently it is disabled for my repo by Github itself. I have filed a support ticket with Github. Once it is resolved, will enable it. But I thought Apache Spark's repo github actions would only run the build and test workflow. It seems like I'm wrong there.
@sarutak Finally got it resolved, now it is enabled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 closed pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 closed pull request #33253:
URL: https://github.com/apache/spark/pull/33253
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-877001267
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45352/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-877431762
**[Test build #140868 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140868/testReport)** for PR 33253 at commit [`cdef2eb`](https://github.com/apache/spark/commit/cdef2eb34703b0ae3cc6d68cc0798006cc2b4af1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r676237515
##########
File path: core/src/test/resources/HistoryServerExpectations/stage_with_speculation_summary_expectation.json
##########
@@ -0,0 +1,748 @@
+{
Review comment:
This change includes over 700 lines of code. Can we minimize this?
##########
File path: core/src/main/scala/org/apache/spark/status/api/v1/api.scala
##########
@@ -89,6 +89,14 @@ class ExecutorStageSummary private[spark](
val peakMemoryMetrics: Option[ExecutorMetrics],
val isExcludedForStage: Boolean)
+class SpeculationStageSummary private[spark](
Review comment:
I think this class is still needed.
How about `SpeculationSummary` instead of `SpeculationStageSummary`?
Speculation is not about stages but tasks.
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -1208,6 +1232,33 @@ private[spark] class AppStatusListener(
}
}
+ private def killedTaskSummaryForSpeculationStageSummary(
+ reason: TaskEndReason,
+ oldSummary: Map[String, Int],
+ isSpeculative: Boolean): Map[String, Int] = {
+ reason match {
+ case k: TaskKilled if k.reason.contains("another attempt succeeded") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary.updated("speculated attempt succeeded",
+ oldSummary.getOrElse("speculated attempt succeeded", 0) + 1)
+ }
+ // If the stage is finished and speculative tasks get killed, then the
+ // kill reason is "stage finished"
+ case k: TaskKilled if k.reason.contains("Stage finished") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary
Review comment:
If we reach this `else` block, is it correct to return `oldSummary`?
If the reason of `Stage finished` is that the speculative task finishes, should we do like [here](https://github.com/apache/spark/pull/33253/files#diff-0d63c25e93eb1fbec25754f0bee7714c7e3c1538bc7370d94b147524db5c1f63R1245-R1246)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-886088386
It seems like some pyspark tests are failing not related to this PR. cc @sarutak
```Failed example:
sc.parallelize([1]).isEmpty()
Exception raised:
Traceback (most recent call last):
File "/usr/lib/pypy3/lib-python/3/doctest.py", line 1332, in __run
compileflags, 1), test.globs)
File "<doctest __main__.RDD.isEmpty[1]>", line 1, in <module>
sc.parallelize([1]).isEmpty()
File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/rdd.py", line 1608, in isEmpty
return self.getNumPartitions() == 0 or len(self.take(1)) == 0
File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/rdd.py", line 1568, in take
res = self.context.runJob(self, takeUpToNumLeft, p)
File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/context.py", line 1228, in runJob
return list(_load_from_socket(sock_info, mappedRDD._jrdd_deserializer))
File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/rdd.py", line 164, in _load_from_socket
sockfile = _create_local_socket(sock_info)
File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/rdd.py", line 141, in _create_local_socket
sockfile, sock = local_connect_and_auth(port, auth_secret)
File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/java_gateway.py", line 214, in local_connect_and_auth
raise RuntimeError("could not open socket: %s" % errors)
RuntimeError: could not open socket: ["tried to connect to ('127.0.0.1', 42447), but an error occurred: [Errno 104] Connection reset by peer"]
**********************************************************************
1 of 2 in __main__.RDD.isEmpty
***Test Failed*** 1 failures.
Had test failures in pyspark.rdd with pypy3; see logs.```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r704911903
##########
File path: core/src/test/resources/HistoryServerExpectations/stage_with_speculation_summary_expectation.json
##########
@@ -0,0 +1,510 @@
+{
+ "status" : "COMPLETE",
+ "stageId" : 0,
+ "attemptId" : 0,
+ "numTasks" : 4,
+ "numActiveTasks" : 0,
+ "numCompleteTasks" : 4,
+ "numFailedTasks" : 0,
+ "numKilledTasks" : 1,
+ "numCompletedIndices" : 4,
+ "submissionTime" : "2021-08-10T23:27:53.488GMT",
+ "firstTaskLaunchedTime" : "2021-08-10T23:27:53.885GMT",
+ "completionTime" : "2021-08-10T23:28:57.679GMT",
+ "executorDeserializeTime" : 12793,
+ "executorDeserializeCpuTime" : 5317155711,
+ "executorRunTime" : 113648,
+ "executorCpuTime" : 284330976,
+ "resultSize" : 3360,
+ "jvmGcTime" : 0,
+ "resultSerializationTime" : 4,
+ "memoryBytesSpilled" : 0,
+ "diskBytesSpilled" : 0,
+ "peakExecutionMemory" : 0,
+ "inputBytes" : 0,
+ "inputRecords" : 0,
+ "outputBytes" : 0,
+ "outputRecords" : 0,
+ "shuffleRemoteBlocksFetched" : 0,
+ "shuffleLocalBlocksFetched" : 0,
+ "shuffleFetchWaitTime" : 0,
+ "shuffleRemoteBytesRead" : 0,
+ "shuffleRemoteBytesReadToDisk" : 0,
+ "shuffleLocalBytesRead" : 0,
+ "shuffleReadBytes" : 0,
+ "shuffleReadRecords" : 0,
+ "shuffleWriteBytes" : 0,
+ "shuffleWriteTime" : 0,
+ "shuffleWriteRecords" : 0,
+ "name" : "collect at <console>:27",
+ "details" : "org.apache.spark.rdd.RDD.collect(RDD.scala:1029)\n$line17.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:27)\n$line17.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:36)\n$line17.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:38)\n$line17.$read$$iw$$iw$$iw$$iw$$iw.<init>(<console>:40)\n$line17.$read$$iw$$iw$$iw$$iw.<init>(<console>:42)\n$line17.$read$$iw$$iw$$iw.<init>(<console>:44)\n$line17.$read$$iw$$iw.<init>(<console>:46)\n$line17.$read$$iw.<init>(<console>:48)\n$line17.$read.<init>(<console>:50)\n$line17.$read$.<init>(<console>:54)\n$line17.$read$.<clinit>(<console>)\n$line17.$eval$.$print$lzycompute(<console>:7)\n$line17.$eval$.$print(<console>:6)\n$line17.$eval.$print(<console>)\nsun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\nsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\nsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\njava.lang.reflect.Method.invoke(Method.java:498)
\nscala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:745)",
+ "schedulingPool" : "default",
+ "rddIds" : [ 1, 0 ],
+ "accumulatorUpdates" : [ ],
+ "tasks" : {
+ "0" : {
+ "taskId" : 0,
+ "index" : 0,
+ "attempt" : 0,
+ "launchTime" : "2021-08-10T23:27:53.885GMT",
+ "duration" : 2234,
+ "executorId" : "7",
+ "host" : "ltx1-hcl12291.grid.linkedin.com",
+ "status" : "SUCCESS",
+ "taskLocality" : "PROCESS_LOCAL",
+ "speculative" : false,
+ "accumulatorUpdates" : [ ],
+ "taskMetrics" : {
+ "executorDeserializeTime" : 2048,
+ "executorDeserializeCpuTime" : 1171756284,
+ "executorRunTime" : 74,
+ "executorCpuTime" : 65263482,
+ "resultSize" : 840,
+ "jvmGcTime" : 0,
+ "resultSerializationTime" : 1,
+ "memoryBytesSpilled" : 0,
+ "diskBytesSpilled" : 0,
+ "peakExecutionMemory" : 0,
+ "inputMetrics" : {
+ "bytesRead" : 0,
+ "recordsRead" : 0
+ },
+ "outputMetrics" : {
+ "bytesWritten" : 0,
+ "recordsWritten" : 0
+ },
+ "shuffleReadMetrics" : {
+ "remoteBlocksFetched" : 0,
+ "localBlocksFetched" : 0,
+ "fetchWaitTime" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
+ "localBytesRead" : 0,
+ "recordsRead" : 0
+ },
+ "shuffleWriteMetrics" : {
+ "bytesWritten" : 0,
+ "writeTime" : 0,
+ "recordsWritten" : 0
+ }
+ },
+ "executorLogs" : {
+ "stdout" : "http://ltx1-hcl12291.grid.linkedin.com:8042/node/containerlogs/container_e18_1628109047826_1317105_01_000008/vsowrira/stdout?start=-4096",
Review comment:
The URL is exposed. Is it OK?
##########
File path: core/src/test/resources/HistoryServerExpectations/stage_with_speculation_summary_expectation.json
##########
@@ -0,0 +1,510 @@
+{
+ "status" : "COMPLETE",
+ "stageId" : 0,
+ "attemptId" : 0,
+ "numTasks" : 4,
+ "numActiveTasks" : 0,
+ "numCompleteTasks" : 4,
+ "numFailedTasks" : 0,
+ "numKilledTasks" : 1,
+ "numCompletedIndices" : 4,
+ "submissionTime" : "2021-08-10T23:27:53.488GMT",
+ "firstTaskLaunchedTime" : "2021-08-10T23:27:53.885GMT",
+ "completionTime" : "2021-08-10T23:28:57.679GMT",
+ "executorDeserializeTime" : 12793,
+ "executorDeserializeCpuTime" : 5317155711,
+ "executorRunTime" : 113648,
+ "executorCpuTime" : 284330976,
+ "resultSize" : 3360,
+ "jvmGcTime" : 0,
+ "resultSerializationTime" : 4,
+ "memoryBytesSpilled" : 0,
+ "diskBytesSpilled" : 0,
+ "peakExecutionMemory" : 0,
+ "inputBytes" : 0,
+ "inputRecords" : 0,
+ "outputBytes" : 0,
+ "outputRecords" : 0,
+ "shuffleRemoteBlocksFetched" : 0,
+ "shuffleLocalBlocksFetched" : 0,
+ "shuffleFetchWaitTime" : 0,
+ "shuffleRemoteBytesRead" : 0,
+ "shuffleRemoteBytesReadToDisk" : 0,
+ "shuffleLocalBytesRead" : 0,
+ "shuffleReadBytes" : 0,
+ "shuffleReadRecords" : 0,
+ "shuffleWriteBytes" : 0,
+ "shuffleWriteTime" : 0,
+ "shuffleWriteRecords" : 0,
+ "name" : "collect at <console>:27",
+ "details" : "org.apache.spark.rdd.RDD.collect(RDD.scala:1029)\n$line17.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:27)\n$line17.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:36)\n$line17.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:38)\n$line17.$read$$iw$$iw$$iw$$iw$$iw.<init>(<console>:40)\n$line17.$read$$iw$$iw$$iw$$iw.<init>(<console>:42)\n$line17.$read$$iw$$iw$$iw.<init>(<console>:44)\n$line17.$read$$iw$$iw.<init>(<console>:46)\n$line17.$read$$iw.<init>(<console>:48)\n$line17.$read.<init>(<console>:50)\n$line17.$read$.<init>(<console>:54)\n$line17.$read$.<clinit>(<console>)\n$line17.$eval$.$print$lzycompute(<console>:7)\n$line17.$eval$.$print(<console>:6)\n$line17.$eval.$print(<console>)\nsun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\nsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\nsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\njava.lang.reflect.Method.invoke(Method.java:498)
\nscala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:745)",
+ "schedulingPool" : "default",
+ "rddIds" : [ 1, 0 ],
+ "accumulatorUpdates" : [ ],
+ "tasks" : {
+ "0" : {
+ "taskId" : 0,
+ "index" : 0,
+ "attempt" : 0,
+ "launchTime" : "2021-08-10T23:27:53.885GMT",
+ "duration" : 2234,
+ "executorId" : "7",
+ "host" : "ltx1-hcl12291.grid.linkedin.com",
+ "status" : "SUCCESS",
+ "taskLocality" : "PROCESS_LOCAL",
+ "speculative" : false,
+ "accumulatorUpdates" : [ ],
+ "taskMetrics" : {
+ "executorDeserializeTime" : 2048,
+ "executorDeserializeCpuTime" : 1171756284,
+ "executorRunTime" : 74,
+ "executorCpuTime" : 65263482,
+ "resultSize" : 840,
+ "jvmGcTime" : 0,
+ "resultSerializationTime" : 1,
+ "memoryBytesSpilled" : 0,
+ "diskBytesSpilled" : 0,
+ "peakExecutionMemory" : 0,
+ "inputMetrics" : {
+ "bytesRead" : 0,
+ "recordsRead" : 0
+ },
+ "outputMetrics" : {
+ "bytesWritten" : 0,
+ "recordsWritten" : 0
+ },
+ "shuffleReadMetrics" : {
+ "remoteBlocksFetched" : 0,
+ "localBlocksFetched" : 0,
+ "fetchWaitTime" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
+ "localBytesRead" : 0,
+ "recordsRead" : 0
+ },
+ "shuffleWriteMetrics" : {
+ "bytesWritten" : 0,
+ "writeTime" : 0,
+ "recordsWritten" : 0
+ }
+ },
+ "executorLogs" : {
+ "stdout" : "http://ltx1-hcl12291.grid.linkedin.com:8042/node/containerlogs/container_e18_1628109047826_1317105_01_000008/vsowrira/stdout?start=-4096",
+ "stderr" : "http://ltx1-hcl12291.grid.linkedin.com:8042/node/containerlogs/container_e18_1628109047826_1317105_01_000008/vsowrira/stderr?start=-4096"
+ },
+ "schedulerDelay" : 111,
+ "gettingResultTime" : 0
+ },
+ "1" : {
+ "taskId" : 1,
+ "index" : 1,
+ "attempt" : 0,
+ "launchTime" : "2021-08-10T23:27:53.903GMT",
+ "duration" : 2647,
+ "executorId" : "5",
+ "host" : "ltx1-hcl5290.grid.linkedin.com",
Review comment:
ditto.
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -1208,6 +1232,33 @@ private[spark] class AppStatusListener(
}
}
+ private def killedTaskSummaryForSpeculationStageSummary(
+ reason: TaskEndReason,
+ oldSummary: Map[String, Int],
+ isSpeculative: Boolean): Map[String, Int] = {
+ reason match {
+ case k: TaskKilled if k.reason.contains("another attempt succeeded") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary.updated("speculated attempt succeeded",
+ oldSummary.getOrElse("speculated attempt succeeded", 0) + 1)
+ }
+ // If the stage is finished and speculative tasks get killed, then the
+ // kill reason is "stage finished"
+ case k: TaskKilled if k.reason.contains("Stage finished") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary
Review comment:
I re-considered around this logic, and as I commented [here](https://github.com/apache/spark/pull/33253/files#diff-0d63c25e93eb1fbec25754f0bee7714c7e3c1538bc7370d94b147524db5c1f63R762), I'd like to suggest that `killedTaskSummaryForSpeculationStageSummary` should be only for `isSpeculative == true`, and simply have the messages `another attempt succeeded` and `Stage finished` rather than `original attempt succeeded` and `speculated attempt succeeded` ?
Then, we don't need `killedTaskSummaryForSpeculationStageSummary` and can use the existing `killedTaskSummary`.
##########
File path: core/src/main/scala/org/apache/spark/status/LiveEntity.scala
##########
@@ -392,6 +392,30 @@ private class LiveExecutorStageSummary(
}
+private class LiveSpeculationStageSummary(
+ stageId: Int,
+ attemptId: Int) extends LiveEntity {
+
+ var numTasks = 0
+ var numActiveTasks = 0
+ var numCompletedTasks = 0
+ var numFailedTasks = 0
+ var numKilledTasks = 0
+ var killedTasksSummary : Map[String, Int] = Map()
Review comment:
It's nit but how about `killedSummary` like other `Live*` classes.
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -746,6 +752,24 @@ private[spark] class AppStatusListener(
maybeUpdate(esummary, now)
}
+ val speculationStageSummary = stage.speculationStageSummary
+ if (event.taskInfo.speculative) {
+ speculationStageSummary.numActiveTasks -= 1
+ speculationStageSummary.numCompletedTasks += completedDelta
+ speculationStageSummary.numFailedTasks += failedDelta
+ speculationStageSummary.numKilledTasks += killedDelta
+ }
+ speculationStageSummary.killedTasksSummary = killedTaskSummaryForSpeculationStageSummary(
Review comment:
`SpeculationStageSummary` is about speculative tasks so `speculationStageSummary.killedTasksSummary` should be only for `event.taskInfo.speculative == true` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-919063719
Could you add updated screenshots on the **description** ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r688209434
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -1208,6 +1232,33 @@ private[spark] class AppStatusListener(
}
}
+ private def killedTaskSummaryForSpeculationStageSummary(
+ reason: TaskEndReason,
+ oldSummary: Map[String, Int],
+ isSpeculative: Boolean): Map[String, Int] = {
+ reason match {
+ case k: TaskKilled if k.reason.contains("another attempt succeeded") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary.updated("speculated attempt succeeded",
+ oldSummary.getOrElse("speculated attempt succeeded", 0) + 1)
+ }
+ // If the stage is finished and speculative tasks get killed, then the
+ // kill reason is "stage finished"
+ case k: TaskKilled if k.reason.contains("Stage finished") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary
Review comment:
Sorry for the late reply.
> I don't think we need special handling for zombie tasks.
Yes, I don't think we need to handle zombie tasks too. What I mean is that, if the reason is `Stage finished`, I think the killed task can be one of the following.
(1) Speculative attempt of a task
(2) A task which has speculative attempt
(3) A task which neither is speculative nor has speculative, but is zombie
For (1), it is handled [here](https://github.com/apache/spark/pull/33253/files#diff-0d63c25e93eb1fbec25754f0bee7714c7e3c1538bc7370d94b147524db5c1f63R1251).
I don't think we need to handle the case (3) but should we handle (2) in the [else block](https://github.com/apache/spark/pull/33253/files#diff-0d63c25e93eb1fbec25754f0bee7714c7e3c1538bc7370d94b147524db5c1f63R1255) somehow?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-917669601
**[Test build #143180 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143180/testReport)** for PR 33253 at commit [`9302892`](https://github.com/apache/spark/commit/93028927fd0233930c8ffcf8bface40c7c5903c2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-882743479
> @venkata91
>
> > it seems like something is wrong with the Github actions on my account therefore the tests are not able to run. I filed a support request with Github but they haven't still figured out why it is not working
>
> Yes, please resolve the GA issue on your account. Once GA on your account works, you can re-run jobs by yourself.
>
> Also, I said that the CI failed by checking [here](https://github.com/apache/spark/pull/33253#issuecomment-880226935).
Yeah thanks for pointing out that. I thought it was due to some other issues after looking at the exception as the tests on that page were successful. After looking at the core tests only realized there were failures in HistoryServerSuite. Fixed it now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-886222062
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46125/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-890273447
**[Test build #141917 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141917/testReport)** for PR 33253 at commit [`0bba76f`](https://github.com/apache/spark/commit/0bba76fd11f14412eca3268e8c775a944ce6a378).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-882741810
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 edited a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 edited a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-882008124
> Some tests seem to fail. Could you fix them?
@sarutak it seems like something is wrong with the Github actions on my account therefore the tests are not able to run. I filed a support request with Github but they haven't still figured out why it is not working. Is there any other way to run the tests like the previous PR builder jenkins job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-887844111
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-890284764
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46426/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r693288502
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -1208,6 +1232,33 @@ private[spark] class AppStatusListener(
}
}
+ private def killedTaskSummaryForSpeculationStageSummary(
+ reason: TaskEndReason,
+ oldSummary: Map[String, Int],
+ isSpeculative: Boolean): Map[String, Int] = {
+ reason match {
+ case k: TaskKilled if k.reason.contains("another attempt succeeded") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary.updated("speculated attempt succeeded",
+ oldSummary.getOrElse("speculated attempt succeeded", 0) + 1)
+ }
+ // If the stage is finished and speculative tasks get killed, then the
+ // kill reason is "stage finished"
+ case k: TaskKilled if k.reason.contains("Stage finished") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary
Review comment:
@sarutak I have seen some other counters being reset to zero in the cases where stage attempt is killed or failed. Counters can be wrong only for failed/killed attempts right? Successful attempts should have the counts all correct right except for the one which gets killed with `stage finished` reason but that also only if it is not a speculative task. Isn't it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r682624830
##########
File path: core/src/test/resources/HistoryServerExpectations/stage_with_speculation_summary_expectation.json
##########
@@ -0,0 +1,748 @@
+{
Review comment:
It just seems redundant. If it's difficult to reduce it, we can replace it later with a small one.
##########
File path: core/src/main/scala/org/apache/spark/status/api/v1/api.scala
##########
@@ -89,6 +89,14 @@ class ExecutorStageSummary private[spark](
val peakMemoryMetrics: Option[ExecutorMetrics],
val isExcludedForStage: Boolean)
+class SpeculationStageSummary private[spark](
Review comment:
O.K. We already have `ExecutorStageSummary` which means aggregated information for a stage. So, the name `SpeculationStageSummary` can be acceptable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r696924643
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -1208,6 +1232,33 @@ private[spark] class AppStatusListener(
}
}
+ private def killedTaskSummaryForSpeculationStageSummary(
+ reason: TaskEndReason,
+ oldSummary: Map[String, Int],
+ isSpeculative: Boolean): Map[String, Int] = {
+ reason match {
+ case k: TaskKilled if k.reason.contains("another attempt succeeded") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary.updated("speculated attempt succeeded",
+ oldSummary.getOrElse("speculated attempt succeeded", 0) + 1)
+ }
+ // If the stage is finished and speculative tasks get killed, then the
+ // kill reason is "stage finished"
+ case k: TaskKilled if k.reason.contains("Stage finished") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary
Review comment:
@sarutak If you don't mind, can you please suggest what should be the next steps? Do you think just documenting this should be enough? Its pending for quite some time now, would like to merge this one soon. It would be great if you can take a look at this again. Thanks :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-885894658
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141571/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-917679121
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47683/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-917679130
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47683/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-882850320
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141270/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r674267708
##########
File path: core/src/main/scala/org/apache/spark/status/storeTypes.scala
##########
@@ -399,6 +399,21 @@ private[spark] class ExecutorStageSummaryWrapper(
}
+private[spark] class SpeculationStageSummaryWrapper(
+ val stageId: Int,
+ val stageAttemptId: Int,
+ val info: SpeculationStageSummary) {
+
+ @JsonIgnore @KVIndex
+ private val _id: Array[Int] = Array(stageId, stageAttemptId)
+
+ @JsonIgnore @KVIndex("stage")
+ private def stage: Array[Int] = Array(stageId, stageAttemptId)
+
+ private[this] val id: Array[Int] = _id
+
Review comment:
Please remove blank line.
##########
File path: core/src/main/scala/org/apache/spark/status/storeTypes.scala
##########
@@ -399,6 +399,21 @@ private[spark] class ExecutorStageSummaryWrapper(
}
+private[spark] class SpeculationStageSummaryWrapper(
Review comment:
ditto.
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala
##########
@@ -524,6 +529,11 @@ private[spark] class AppStatusStore(
} else {
None
}
+ val speculationStageSummary: Option[v1.SpeculationStageSummary] = if (withSummaries) {
Review comment:
If `StageData` can contain speculation summary regardless of the value of `withSummaries`, we don't need to add new API `/speculationSummary` right?
##########
File path: core/src/main/scala/org/apache/spark/status/api/v1/api.scala
##########
@@ -89,6 +89,14 @@ class ExecutorStageSummary private[spark](
val peakMemoryMetrics: Option[ExecutorMetrics],
val isExcludedForStage: Boolean)
+class SpeculationStageSummary private[spark](
Review comment:
ditto.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r688209434
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -1208,6 +1232,33 @@ private[spark] class AppStatusListener(
}
}
+ private def killedTaskSummaryForSpeculationStageSummary(
+ reason: TaskEndReason,
+ oldSummary: Map[String, Int],
+ isSpeculative: Boolean): Map[String, Int] = {
+ reason match {
+ case k: TaskKilled if k.reason.contains("another attempt succeeded") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary.updated("speculated attempt succeeded",
+ oldSummary.getOrElse("speculated attempt succeeded", 0) + 1)
+ }
+ // If the stage is finished and speculative tasks get killed, then the
+ // kill reason is "stage finished"
+ case k: TaskKilled if k.reason.contains("Stage finished") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary
Review comment:
Sorry for the late reply.
> I don't think we need special handling for zombie tasks.
Yes, I don't think we need to handle zombie tasks too. What I mean is that, if the reason is `Stage finished`, I think the killed task can be one of the following.
(1) Speculative attempt of a task
(2) A task which has speculative attempt
(3) A task which neither is speculative nor has speculative, but is zombie
For (1), it is handled [here](https://github.com/apache/spark/pull/33253/files#diff-0d63c25e93eb1fbec25754f0bee7714c7e3c1538bc7370d94b147524db5c1f63R1251).
I don't think we need to handle the case (3) but should we handle (2) in the [else block](https://github.com/apache/spark/pull/33253/files#diff-0d63c25e93eb1fbec25754f0bee7714c7e3c1538bc7370d94b147524db5c1f63R1255) somehow?
cc: @Ngone51 and @mridulm for the code around here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-917669601
**[Test build #143180 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143180/testReport)** for PR 33253 at commit [`9302892`](https://github.com/apache/spark/commit/93028927fd0233930c8ffcf8bface40c7c5903c2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
Posted by GitBox <gi...@apache.org>.
venkata91 commented on pull request #33253:
URL: https://github.com/apache/spark/pull/33253#issuecomment-926001947
`StackOverflowError` is happening on the sql module which is not related to this change I think. Will try to update again with the latest master to see if this succeeds. cc @sarutak
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org