You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by tsudukim <gi...@git.apache.org> on 2014/07/12 05:00:33 UTC
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
GitHub user tsudukim opened a pull request:
https://github.com/apache/spark/pull/1384
SPARK-2298: Show stage attempt in UI
Added attempt ID column into stage page of webUI.
Added attemptId handling code into StageInfo, JsonProtocol.
Modified DAGScheduler to identify stages whose stageId is same but attemptId is different.
Modified testcode for stage attempt ID.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tsudukim/spark feature/SPARK-2298
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1384.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1384
----
commit 0ea38ae39e34dacb8adef96cfbb7ce728bb3406a
Author: Masayoshi TSUZUKI <ts...@oss.nttdata.co.jp>
Date: 2014-07-11T19:02:35Z
SPARK-2298 Show stage attempt in UI
Added attempt ID column into stage page of webUI.
Added attemptId handling code into StageInfo, JsonProtocol.
Modified DAGScheduler to identify stages whose stageId is same but attemptId is different.
Modified testcode for stage attempt ID.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-54101534
I think this was ultimately fixed by #1545 so we can close this issue. But feel free to open another PR if that one did not fix this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Posted by tsudukim <gi...@git.apache.org>.
Github user tsudukim commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-48804094
@xrin OK, thanks. Then attempt id is still required in the web ui for users to know stage + attempt. Have I got that right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-48804157
Yup - but let's avoid exposing the concept of TaskSet to users in the UI. That's only for internal engineering.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-49208410
Let's hold off merging this one until we merge #1262. Then it will be easier to index the information based on stage + attempt.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Posted by tsudukim <gi...@git.apache.org>.
Github user tsudukim commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-48802747
@pwendell Thank you for your response. You mean like this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-50149066
@tsudukim yes,SPARK-2298 is that i want to. but i think a simple way is on this PR add a jobid column to stage table.it is very easy to achieve it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-49111190
@tsudukim @kayousterhout so I think in general here, our handling of stage re-submissions is broken in the UI. For instance, I looked in the `JobProgressListener` and we index many things on `StageId` that might better be indexed on `StageId, AttemptId`. Also, we should probably give the `AttemptId` when start a task so that we understand which stage attempt to associated it with. I also don't understand exactly what happens when a stage gets re-attempted, do we send a "stage completed" event? It might be good to fix the way we deal with stage re-submissions to make this work better overall.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Posted by tsudukim <gi...@git.apache.org>.
Github user tsudukim commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-48803681
@andrewor14 Thank you for your comment.
I think it is more weird if the display style of ID/attempt changes by conditions.
Surely most stages will only have 1 attempt, but as the set of stage id and stage attempt id is the identifier of the taskset, I'd like to show the attempt id even if the stage is the first one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-49111726
@tsudukim I created a JIRA to deal with the broader issue. If you want to take that on as well, let me know: https://issues.apache.org/jira/browse/SPARK-2501 it might make sense to wrap it into this patch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Posted by tsudukim <gi...@git.apache.org>.
Github user tsudukim commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-49944917
@lianhuiwang It appears to be a different problem to SPARK-2298.
Is your aim same as this ticket?
https://issues.apache.org/jira/browse/SPARK-1362
If so, how about creating another PR to modify it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Posted by tsudukim <gi...@git.apache.org>.
Github user tsudukim commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-49944747
@rxin Surely we can also fix them all in one patch. But it can be a little bit hard work to modify them compatibly in one patch so I just have thought to separate into several tasks and to make #1384 as first step by showing only attemptId to distinguish attempts.
You can take whichever is convenient for you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-48803729
@tsudukim The concept of TaskSet should be internal to Spark. Users shouldn't have to aware of task set. Users should only care about stage + attempt.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-48802862
Hm the latest screenshot looks a little funky to me.
Most stages will only have 1 attempt, so I think it makes sense to only show the attempt if this is not the first one. Something like:
```
ID
4
2
3 (Attempt 2)
1
0
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-48801078
/cc @rxin who is interested in this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Posted by tsudukim <gi...@git.apache.org>.
Github user tsudukim commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-48963364
I'm wondering how to show it. I gave it a shot. Is it smart?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Posted by tsudukim <gi...@git.apache.org>.
Github user tsudukim commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-49209319
@rxin OK. After that, I think I can make this patch better.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Posted by tsudukim <gi...@git.apache.org>.
Github user tsudukim commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-49207731
@pwendell I agree that there are many room for improvement about handling of stageId and attemptId. It might be better to break this problems into some sub-tasks. I think this patch should be one of them. Or did you mean we should fix all of this problem in one patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Posted by tsudukim <gi...@git.apache.org>.
Github user tsudukim commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-48800698
Attempt Id shows up in web ui. Submitted and Duration became individual value to stage attempts.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Posted by tsudukim <gi...@git.apache.org>.
Github user tsudukim commented on a diff in the pull request:
https://github.com/apache/spark/pull/1384#discussion_r15019018
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -478,6 +479,7 @@ private[spark] object JsonProtocol {
def stageInfoFromJson(json: JValue): StageInfo = {
val stageId = (json \ "Stage ID").extract[Int]
+ val attemptId = (json \ "Attempt ID").extract[Int]
--- End diff --
Sure.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-49860984
i think you can add jobid to stageTable. because jobid is very useful when a application has many jobs.that can distinguish every job's stages.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/1384
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-48800948
To make this a bit more concise, what about having one column on the left whose header is `ID: Attempt` and separating them with a colon. Current the word "Stage" is redundant there.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1384#discussion_r14975903
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -478,6 +479,7 @@ private[spark] object JsonProtocol {
def stageInfoFromJson(json: JValue): StageInfo = {
val stageId = (json \ "Stage ID").extract[Int]
+ val attemptId = (json \ "Attempt ID").extract[Int]
--- End diff --
Is this backwards compatible?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-48800648
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-49111342
@rxin is this something you've thought about in your various schedule refactoring things?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Posted by tsudukim <gi...@git.apache.org>.
Github user tsudukim commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-49209857
@rxin in #1262, can I expect the key of the stagedata in JobProgressListener become stageId + attemptId instead of stageId only?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-49850442
It turned out much trickier than I thought to add attempt id. I submitted a PR here #1545
That PR already modifies the UI, since that's the only way I could test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1384#discussion_r14977192
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -478,6 +479,7 @@ private[spark] object JsonProtocol {
def stageInfoFromJson(json: JValue): StageInfo = {
val stageId = (json \ "Stage ID").extract[Int]
+ val attemptId = (json \ "Attempt ID").extract[Int]
--- End diff --
Could you add a test in `JsonProtocolSuite`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Posted by tsudukim <gi...@git.apache.org>.
Github user tsudukim commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-49495727
Modified PR as your comments. thank you!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Posted by tsudukim <gi...@git.apache.org>.
Github user tsudukim commented on a diff in the pull request:
https://github.com/apache/spark/pull/1384#discussion_r14976665
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -478,6 +479,7 @@ private[spark] object JsonProtocol {
def stageInfoFromJson(json: JValue): StageInfo = {
val stageId = (json \ "Stage ID").extract[Int]
+ val attemptId = (json \ "Attempt ID").extract[Int]
--- End diff --
Ah, no it's not. How about this?
```
val attemptId = (json \ "Attempt ID").extractOpt[Int].getOrElse(0)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---