You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/11/29 07:54:49 UTC
[GitHub] [spark] yaooqinn opened a new pull request #34735: [SPARK-37481][Core] Fix disappearance of skipped stages after they retry
yaooqinn opened a new pull request #34735:
URL: https://github.com/apache/spark/pull/34735
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'core/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
When skipped stages retry, their skipped info will be lost on the UI, and then we may see a stage with 200 tasks indeed, shows that it only has 3 tasks but its `retry 1` has 15 tasks and completely different inputs/outputs.
Detailed in the Gist here - https://gist.github.com/yaooqinn/6acb7b74b343a6a6dffe8401f6b7b45c
In this PR, we mark these skipped stages completed and increase the attempId once they get visited.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
fix problems when we have skipped stage retries.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
Yes, the UI will keep the skipped stages info
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
manually as recorded in https://gist.github.com/yaooqinn/6acb7b74b343a6a6dffe8401f6b7b45c
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 closed pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
Ngone51 closed pull request #34735:
URL: https://github.com/apache/spark/pull/34735
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-990871491
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50545/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-991853731
**[Test build #146107 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146107/testReport)** for PR 34735 at commit [`0723853`](https://github.com/apache/spark/commit/0723853e08ccef441368928d9f169ce186ae32d7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-987874585
Sorry for misunderstanding your point :)
> But thinking more, I feel this PR might be a strict improvement over the current state.
Atleast users can see what the original (skipped) stage details were, even though it was skipped: and is now getting reexecuted with a subset of partitions.
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-983394118
cc @jiangxb1987 @Ngone51
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-991876365
**[Test build #146106 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146106/testReport)** for PR 34735 at commit [`ecc7db1`](https://github.com/apache/spark/commit/ecc7db1d5a65f969d05aee895e09309a0b2dc28c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-991859566
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50581/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34735: [SPARK-37481][Core] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-981507375
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145702/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn edited a comment on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
yaooqinn edited a comment on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-986509805
> But showing a skipped stage, when it runs initially, as "retry 1" can be confusing
Per the figure in PR desc, we will both keep the skipped info and retry info, it's clear for us to know that `stage 2` once get skipped because `stage 0` has all the map outputs, and gets retried because `stage 3` failed with fetch failed issues.
But in contrast， the above 4）really makes us confused about
- Why a stage and its retries can have different tasks and outputs?
- Does the RDD become mutable?'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34735: [SPARK-37481][Core] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-981628371
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50188/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-987856439
BTW, one query - specific to SQL (does not apply to vanilla RDD jobs).
In context of skewed joins or partition coalascing, when AQE changes number of partitions post shuffle, how does this work ?
Will stage 2 retry 1 preserve the earlier stage 2 skipped attempt 0 task counts (if it was getting fully re-executed) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-987453178
> > Per the figure in PR desc, we will both keep the skipped info and retry info, it's clear for us to know that stage 2 once get skipped because stage 0 has all the map outputs, and gets retried because stage 3 failed with fetch failed issues.
>
> How are users inferring the last part ? That stage 0 was retried due to stage 3 failure ?
aqe divides them to jobs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34735: [SPARK-37481][Core] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-981734225
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145719/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-991867877
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50580/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-991858629
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50580/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-991868676
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-991876060
**[Test build #146107 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146107/testReport)** for PR 34735 at commit [`0723853`](https://github.com/apache/spark/commit/0723853e08ccef441368928d9f169ce186ae32d7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-992058557
Thanks! Merged to Master/3.2.
@yaooqinn Could you manually backport the fix to 3.1, 3.0? It has conflicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #34735:
URL: https://github.com/apache/spark/pull/34735#discussion_r766031321
##########
File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
##########
@@ -718,6 +718,12 @@ private[spark] class DAGScheduler(
// read from merged output as the MergeStatuses are not available.
if (!mapStage.isAvailable || !mapStage.shuffleDep.shuffleMergeFinalized) {
missing += mapStage
+ } else if (mapStage.nextAttemptId == 0) {
+ // Forward the nextAttemptId if skipped and get visited for the first time.
+ // Otherwise, once it gets retried,
+ // 1) the stuffs in stage info become distorting, e.g. task num, input byte, e.t.c
+ // 2) the first attempt starts from 0-idx, it will not be marked as a retry
+ mapStage.nextAttemptId += 1
Review comment:
+1 on explicitly making intent to skip clear, instead of only bumping the attempt id.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-990913147
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146070/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn removed a comment on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
yaooqinn removed a comment on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-987733559
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-991866213
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50581/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-987853351
I understand about AQE leveraging shuffle map stage to submit a job.
My query was around - "How are users inferring the last part ? That stage 0 was retried due to stage 3 failure ?"
My query was a response to this:
>Per the figure in PR desc, we will both keep the skipped info and retry info, it's clear for us to know that stage 2 once get skipped because stage 0 has all the map outputs, and gets retried because stage 3 failed with fetch failed issues.
The inference that stage 3 failed with fetch failure resulted in stage 2 getting re-executed is what I want to make sure users can understand - we dont expose why stage 2 was initially skipped, and what caused it to be re-executed : in the case of skipped stages specifically.
But thinking more, I feel this PR might be a strict improvement over the current state.
Atleast users can see what the original (skipped) stage details were, even though it was skipped: and is now getting reexecuted with a different set of partitions.
+CC @tgravescs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #34735:
URL: https://github.com/apache/spark/pull/34735#discussion_r765951281
##########
File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
##########
@@ -718,6 +718,12 @@ private[spark] class DAGScheduler(
// read from merged output as the MergeStatuses are not available.
if (!mapStage.isAvailable || !mapStage.shuffleDep.shuffleMergeFinalized) {
missing += mapStage
+ } else if (mapStage.nextAttemptId == 0) {
+ // Forward the nextAttemptId if skipped and get visited for the first time.
+ // Otherwise, once it gets retried,
+ // 1) the stuffs in stage info become distorting, e.g. task num, input byte, e.t.c
+ // 2) the first attempt starts from 0-idx, it will not be marked as a retry
+ mapStage.nextAttemptId += 1
Review comment:
its incrementing nextAttemptId (ie the next attempt, not the current one). The skipped attempt never calls makeNewStageAttempt so the first time it was skipped this was never incremented. The default of nextAttemptId on Stage creation is 0 and since it was never incremented in the skipped stage when you call makeNewStageAttempt for the failed attempt it is run as 0 but the next attempt would be 1. So the skipped attempt and the rerun failed attempt both are attempt 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34735: [SPARK-37481][Core] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-981494208
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50172/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34735: [SPARK-37481][Core] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-981583332
**[Test build #145719 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145719/testReport)** for PR 34735 at commit [`00cd505`](https://github.com/apache/spark/commit/00cd5057a2846bfa9e61ddaa8ecf6b40dc977251).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-987733559
Sorry, I made a mistake about 4), it shall retry all tasks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34735: [SPARK-37481][Core] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-981627028
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34735: [SPARK-37481][Core] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-981508808
**[Test build #145712 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145712/testReport)** for PR 34735 at commit [`eb4f184`](https://github.com/apache/spark/commit/eb4f1841566ca8ef6547236321d7a18ffdeb9c1c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34735: [SPARK-37481][Core] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-981507375
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145702/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34735: [SPARK-37481][Core] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-981677410
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50188/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34735: [SPARK-37481][Core] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-981460726
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50172/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #34735:
URL: https://github.com/apache/spark/pull/34735#discussion_r767147403
##########
File path: core/src/main/scala/org/apache/spark/scheduler/Stage.scala
##########
@@ -107,6 +107,13 @@ private[scheduler] abstract class Stage(
nextAttemptId += 1
}
+ /** Forward the nextAttemptId if skipped and get visited for the first time. */
+ def skipIfNecessary(): Unit = {
Review comment:
sure，will update
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #34735:
URL: https://github.com/apache/spark/pull/34735#discussion_r765178838
##########
File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
##########
@@ -718,6 +718,12 @@ private[spark] class DAGScheduler(
// read from merged output as the MergeStatuses are not available.
if (!mapStage.isAvailable || !mapStage.shuffleDep.shuffleMergeFinalized) {
missing += mapStage
+ } else if (mapStage.nextAttemptId == 0) {
+ // Forward the nextAttemptId if skipped and get visited for the first time.
+ // Otherwise, once it gets retried,
+ // 1) the stuffs in stage info become distorting, e.g. task num, input byte, e.t.c
+ // 2) the first attempt starts from 0-idx, it will not be marked as a retry
+ mapStage.nextAttemptId += 1
Review comment:
ok, not really seeing any better way to do this. The one thing that might be nice is instead of exposing nextAttemptId here, make like a mapStage.skip() type function which can internally check if mapStage.nextAttemptId == 0 and increment if needed. its really not to big of a deal though as we reach into Stage a lot from the DAGScheduler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-982506264
also cc @sarutak
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34735: [SPARK-37481][Core] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-981723374
**[Test build #145719 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145719/testReport)** for PR 34735 at commit [`00cd505`](https://github.com/apache/spark/commit/00cd5057a2846bfa9e61ddaa8ecf6b40dc977251).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-986461809
What is the root cause of this issue ? Can you describe why this is happening ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-986509805
> But showing a skipped stage, when it runs initially, as "retry 1" can be confusing
Per the figure in PR desc, we will both keep the skipped info and retry info, it's clear for us to know that `stage 2` once get skipped because `stage 0` has all the map outputs, and gets retried because `stage 3` failed with fetch failed issues.
But in contrast， the above 4）really makes us confused - 'Why a stage and its retries can have different tasks and outputs? Does the RDD become mutable?'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-991853299
**[Test build #146106 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146106/testReport)** for PR 34735 at commit [`ecc7db1`](https://github.com/apache/spark/commit/ecc7db1d5a65f969d05aee895e09309a0b2dc28c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-987734847
> Sorry, I made a mistake about 4), it shall retry all tasks.
I noticed once on an online job failure that may be caused by another hidden bug, which leads me to the pitfall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-987842726
# SQL
select stage_id_with_retry(v) from (select v from values (1), (2), (3) t(v) group by v)
# Plan
# Jobs
2 jobs in total
## Job 0 Details
## Job 1 Details
# Jobs (AQE off)
only 1 job
## Job 0 details (AQE off)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm edited a comment on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
mridulm edited a comment on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-987856439
~~BTW, one query - specific to SQL (does not apply to vanilla RDD jobs).
In context of skewed joins or partition coalascing, when AQE changes number of partitions post shuffle, how does this work ?~~
~~Will stage 2 retry 1 preserve the earlier stage 2 skipped attempt 0 task counts (if it was getting fully re-executed) ?~~
Thinking more, not applicable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34735: [SPARK-37481][Core] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-981583332
**[Test build #145719 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145719/testReport)** for PR 34735 at commit [`00cd505`](https://github.com/apache/spark/commit/00cd5057a2846bfa9e61ddaa8ecf6b40dc977251).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34735: [SPARK-37481][Core] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-981388161
**[Test build #145702 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145702/testReport)** for PR 34735 at commit [`ba01356`](https://github.com/apache/spark/commit/ba0135686614ed5a90ecf7da83ca52401161c02c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-987077336
> Per the figure in PR desc, we will both keep the skipped info and retry info, it's clear for us to know that stage 2 once get skipped because stage 0 has all the map outputs, and gets retried because stage 3 failed with fetch failed issues.
How are users inferring the last part ? That stage 0 was retried due to stage 3 failure ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn edited a comment on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
yaooqinn edited a comment on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-987842726
# SQL
select stage_id_with_retry(v) from (select v from values (1), (2), (3) t(v) group by v)
# Plan
# Jobs
2 jobs in total
## Job 0 Details
## Job 1 Details
# Plan (AQE off)
# Jobs (AQE off)
only 1 job
## Job 0 details (AQE off)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #34735:
URL: https://github.com/apache/spark/pull/34735#discussion_r765487268
##########
File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
##########
@@ -718,6 +718,12 @@ private[spark] class DAGScheduler(
// read from merged output as the MergeStatuses are not available.
if (!mapStage.isAvailable || !mapStage.shuffleDep.shuffleMergeFinalized) {
missing += mapStage
+ } else if (mapStage.nextAttemptId == 0) {
+ // Forward the nextAttemptId if skipped and get visited for the first time.
+ // Otherwise, once it gets retried,
+ // 1) the stuffs in stage info become distorting, e.g. task num, input byte, e.t.c
+ // 2) the first attempt starts from 0-idx, it will not be marked as a retry
+ mapStage.nextAttemptId += 1
Review comment:
> its really not to big of a deal though as we reach into Stage a lot from the DAGScheduler.
+1, I prefer to minimize the diff
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-990910651
**[Test build #146070 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146070/testReport)** for PR 34735 at commit [`c97ed28`](https://github.com/apache/spark/commit/c97ed280488e36f1f1a1ede9f5636cc4640688e5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class TimedeltaOps(DataTypeOps):`
* `class TimedeltaIndex(Index):`
* `class MissingPandasLikeTimedeltaIndex(MissingPandasLikeIndex):`
* `class PandasSQLStringFormatter(string.Formatter):`
* `class UDFBasicProfiler(BasicProfiler):`
* `class SQLStringFormatter(string.Formatter):`
* `class ExecutorPodsPollingSnapshotSource(`
* `class ExecutorPodsWatchSnapshotSource(`
* `class ExecutorRollPlugin extends SparkPlugin with Logging `
* ` class AnsiCombinedTypeCoercionRule(rules: Seq[TypeCoercionRule]) extends`
* ` class CombinedTypeCoercionRule(rules: Seq[TypeCoercionRule]) extends TypeCoercionRule `
* `case class PrettyPythonUDF(`
* `case class MapContainsKey(`
* `case class TryElementAt(left: Expression, right: Expression, child: Expression)`
* `case class ConvertTimezone(`
* `case class AesEncrypt(`
* `case class AesDecrypt(`
* `case class CreateTable(`
* `case class TableSpec(`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-990751072
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146062/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-991868676
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-991877601
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #34735:
URL: https://github.com/apache/spark/pull/34735#discussion_r767147403
##########
File path: core/src/main/scala/org/apache/spark/scheduler/Stage.scala
##########
@@ -107,6 +107,13 @@ private[scheduler] abstract class Stage(
nextAttemptId += 1
}
+ /** Forward the nextAttemptId if skipped and get visited for the first time. */
+ def skipIfNecessary(): Unit = {
Review comment:
sure，will update
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-990751072
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146062/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-990843544
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50545/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm edited a comment on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
mridulm edited a comment on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-987853351
I understand about AQE leveraging shuffle map stage to submit a job.
My query was around - "How are users inferring the last part ? That stage 0 was retried due to stage 3 failure ?"
My query was a response to this:
>Per the figure in PR desc, we will both keep the skipped info and retry info, it's clear for us to know that stage 2 once get skipped because stage 0 has all the map outputs, and gets retried because stage 3 failed with fetch failed issues.
The inference that stage 3 failed with fetch failure resulted in stage 2 getting re-executed is what I want to make sure users can understand - we dont expose why stage 2 was initially skipped, and what caused it to be re-executed : in the case of skipped stages specifically.
But thinking more, I feel this PR might be a strict improvement over the current state.
Atleast users can see what the original (skipped) stage details were, even though it was skipped: and is now getting reexecuted with a subset of partitions.
+CC @tgravescs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-991853299
**[Test build #146106 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146106/testReport)** for PR 34735 at commit [`ecc7db1`](https://github.com/apache/spark/commit/ecc7db1d5a65f969d05aee895e09309a0b2dc28c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-990754762
**[Test build #146070 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146070/testReport)** for PR 34735 at commit [`c97ed28`](https://github.com/apache/spark/commit/c97ed280488e36f1f1a1ede9f5636cc4640688e5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34735: [SPARK-37481][Core] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-981734225
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145719/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34735: [SPARK-37481][Core] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-981506202
**[Test build #145702 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145702/testReport)** for PR 34735 at commit [`ba01356`](https://github.com/apache/spark/commit/ba0135686614ed5a90ecf7da83ca52401161c02c).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34735: [SPARK-37481][Core] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-981508808
**[Test build #145712 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145712/testReport)** for PR 34735 at commit [`eb4f184`](https://github.com/apache/spark/commit/eb4f1841566ca8ef6547236321d7a18ffdeb9c1c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34735: [SPARK-37481][Core] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-981627028
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #34735: [SPARK-37481][Core] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-982406102
cc @cloud-fan @HyukjinKwon thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-991853731
**[Test build #146107 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146107/testReport)** for PR 34735 at commit [`0723853`](https://github.com/apache/spark/commit/0723853e08ccef441368928d9f169ce186ae32d7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-992059862
> Thanks! Merged to Master/3.2.
>
> @yaooqinn Could you manually backport the fix to 3.1, 3.0? It has conflicts.
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-987030971
> If it gets retried more than once, the total tasks/inputs/outputs of each attempt may be inconsistent, for example, you have a skipped stage with 200 tasks indeed, but you won't see this. In fact, you might see Stage X - Tasks: (Succeeded/Total) (3/3) and Stage X (retry 1) - Tasks: (Succeeded/Total) (15/15).
For any stage retry, which is not recomputing all partitions (which is the common case), this is the behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn edited a comment on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
yaooqinn edited a comment on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-987734847
> Sorry, I made a mistake about 4), it shall retry all tasks.
I noticed once on an online job failure for such behavior that may be caused by another hidden bug, which leads me to the pitfall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-986484286
I am trying to understand the change here, I saw the gist but it was not clear to me why/what the expected behavior we are trying to expose here.
Are we trying to preserve skipped stages when they are retried ? If yes, why ?
Or is it simply to surface the fact that stage 2, when it shows up in stage list, is a retry due to some failure ? (this is useful information to show).
But showing a skipped stage, when it runs initially, as "retry 1" can be confusing - since it is running for first time (as stage 2). Not sure if it is easy to show what was the original stage which ran, and resulted in stage 2 becoming skipped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-987698358
> aqe divides them to jobs
Can you clarify please ? I am not sure what you mean here ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-986464047
FYI, https://gist.github.com/yaooqinn/6acb7b74b343a6a6dffe8401f6b7b45c#gistcomment-3977382 @mridulm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #34735:
URL: https://github.com/apache/spark/pull/34735#discussion_r765931693
##########
File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
##########
@@ -718,6 +718,12 @@ private[spark] class DAGScheduler(
// read from merged output as the MergeStatuses are not available.
if (!mapStage.isAvailable || !mapStage.shuffleDep.shuffleMergeFinalized) {
missing += mapStage
+ } else if (mapStage.nextAttemptId == 0) {
+ // Forward the nextAttemptId if skipped and get visited for the first time.
+ // Otherwise, once it gets retried,
+ // 1) the stuffs in stage info become distorting, e.g. task num, input byte, e.t.c
+ // 2) the first attempt starts from 0-idx, it will not be marked as a retry
+ mapStage.nextAttemptId += 1
Review comment:
https://github.com/apache/spark/blob/c4e975e175c01f67ece7ae492a79554ad1b44106/core/src/main/scala/org/apache/spark/scheduler/Stage.scala#L98-L108
Don't we automatically increase the nextAttemptId whenever submitting a new stage attempt? Why the skipped stage be an abnormal case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-990754762
**[Test build #146070 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146070/testReport)** for PR 34735 at commit [`c97ed28`](https://github.com/apache/spark/commit/c97ed280488e36f1f1a1ede9f5636cc4640688e5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #34735:
URL: https://github.com/apache/spark/pull/34735#discussion_r766693615
##########
File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
##########
@@ -718,6 +718,12 @@ private[spark] class DAGScheduler(
// read from merged output as the MergeStatuses are not available.
if (!mapStage.isAvailable || !mapStage.shuffleDep.shuffleMergeFinalized) {
missing += mapStage
+ } else if (mapStage.nextAttemptId == 0) {
+ // Forward the nextAttemptId if skipped and get visited for the first time.
+ // Otherwise, once it gets retried,
+ // 1) the stuffs in stage info become distorting, e.g. task num, input byte, e.t.c
+ // 2) the first attempt starts from 0-idx, it will not be marked as a retry
+ mapStage.nextAttemptId += 1
Review comment:
oh I misread it. It's for the increment is for the next stage attempt. thanks @tgravescs @mridulm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34735: [SPARK-37481][Core] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-981605041
**[Test build #145712 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145712/testReport)** for PR 34735 at commit [`eb4f184`](https://github.com/apache/spark/commit/eb4f1841566ca8ef6547236321d7a18ffdeb9c1c).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34735: [SPARK-37481][Core] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-981677410
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50188/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34735: [SPARK-37481][Core] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-981580328
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50181/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34735: [SPARK-37481][Core] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-981626856
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50181/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #34735:
URL: https://github.com/apache/spark/pull/34735#discussion_r766030145
##########
File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
##########
@@ -718,6 +718,12 @@ private[spark] class DAGScheduler(
// read from merged output as the MergeStatuses are not available.
if (!mapStage.isAvailable || !mapStage.shuffleDep.shuffleMergeFinalized) {
missing += mapStage
+ } else if (mapStage.nextAttemptId == 0) {
+ // Forward the nextAttemptId if skipped and get visited for the first time.
+ // Otherwise, once it gets retried,
+ // 1) the stuffs in stage info become distorting, e.g. task num, input byte, e.t.c
+ // 2) the first attempt starts from 0-idx, it will not be marked as a retry
+ mapStage.nextAttemptId += 1
Review comment:
Agree with Tom, this is specific to the skipped -> running transition; not applicable for retry which, as you @Ngone51 mentioned, will increment via `makeNewStageAttempt`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #34735:
URL: https://github.com/apache/spark/pull/34735#discussion_r766693615
##########
File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
##########
@@ -718,6 +718,12 @@ private[spark] class DAGScheduler(
// read from merged output as the MergeStatuses are not available.
if (!mapStage.isAvailable || !mapStage.shuffleDep.shuffleMergeFinalized) {
missing += mapStage
+ } else if (mapStage.nextAttemptId == 0) {
+ // Forward the nextAttemptId if skipped and get visited for the first time.
+ // Otherwise, once it gets retried,
+ // 1) the stuffs in stage info become distorting, e.g. task num, input byte, e.t.c
+ // 2) the first attempt starts from 0-idx, it will not be marked as a retry
+ mapStage.nextAttemptId += 1
Review comment:
Oh, I misread it. The increment is for the next stage attempt. thanks @tgravescs @mridulm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-990871491
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50545/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-990913147
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146070/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #34735:
URL: https://github.com/apache/spark/pull/34735#discussion_r766703706
##########
File path: core/src/main/scala/org/apache/spark/scheduler/Stage.scala
##########
@@ -107,6 +107,13 @@ private[scheduler] abstract class Stage(
nextAttemptId += 1
}
+ /** Forward the nextAttemptId if skipped and get visited for the first time. */
+ def skipIfNecessary(): Unit = {
Review comment:
How about `increaseAttemptIdOnFirstSkip()`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-991877601
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-990793960
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50545/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #34735:
URL: https://github.com/apache/spark/pull/34735#discussion_r764315653
##########
File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
##########
@@ -718,6 +718,12 @@ private[spark] class DAGScheduler(
// read from merged output as the MergeStatuses are not available.
if (!mapStage.isAvailable || !mapStage.shuffleDep.shuffleMergeFinalized) {
missing += mapStage
+ } else if (mapStage.nextAttemptId == 0) {
+ // Forward the nextAttemptId if skipped and get visited for the first time.
+ // Otherwise, once it gets retried,
+ // 1) the stuffs in stage info become distorting, e.g. task num, input byte, e.t.c
+ // 2) the first attempt starts from 0-idx, it will not be marked as a retry
+ mapStage.nextAttemptId += 1
Review comment:
this doesn't feel like the best way to do this. I also don't think we should expose nextAttemptId here, I need to look a bit more at it and will get back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34735: [SPARK-37481][Core] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-981388161
**[Test build #145702 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145702/testReport)** for PR 34735 at commit [`ba01356`](https://github.com/apache/spark/commit/ba0135686614ed5a90ecf7da83ca52401161c02c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34735: [SPARK-37481][Core] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-981419007
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50172/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34735: [SPARK-37481][Core] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-981494208
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50172/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34735: [SPARK-37481][Core] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-981665205
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50188/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-987827244
Stage 2 skipped
> > aqe divides them to jobs
>
> Can you clarify please? I am not sure what you mean here ...
Stage 2 was skipped via stage 0, stage 2 was retried via stage 3. In AQE, we break `Exchange` to `QueryStages`, which submitShuffleMapStages or jobs, so here stage 2 is logically a shallow copy of stage 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #34735:
URL: https://github.com/apache/spark/pull/34735#issuecomment-986503970
> Are we trying to preserve skipped stages when they are retried? If yes, why ?
Yes. When the skipped stages are retried,
1. the skipped info on UI get lost
2. the retry info the first retry for those stages get lost
3. the total tasks of those stages get lost when partially retry
4. If it gets retried more than once, the total tasks/inputs/outputs of each attempt may be inconsistent, for example, you have a skipped stage with 200 tasks indeed, but you won't see this. In fact, you might see `Stage X - Tasks: (Succeeded/Total) (3/3)` and `Stage X (retry 1) - Tasks: (Succeeded/Total) (15/15)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org