You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/03/08 10:06:44 UTC
[GitHub] [spark] Ngone51 opened a new pull request #27849:
[SPARK-31081][UI][SQL] Make SQLMetrics more readable from UI
Ngone51 opened a new pull request #27849: [SPARK-31081][UI][SQL] Make SQLMetrics more readable from UI
URL: https://github.com/apache/spark/pull/27849
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
1. To make `SQLMetric` more readable from UI
2. Make display of `stageId/StageAttemptId/taskId` configurable
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
It makes metrics harder to read after #26843 and user may not interest in extra info(`stageId/StageAttemptId/taskId `) when they do not need debug.
### Does this PR introduce any user-facing change?
<!--
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If no, write 'No'.
-->
Yes.
Before this PR:
<img width="844" alt="old_metric" src="https://user-images.githubusercontent.com/16397174/76160657-f993e880-6166-11ea-96df-7672bfd1a89a.png">
After this PR with `stageId/StageAttemptId/taskId`:
<img width="794" alt="new_with_taskInfo" src="https://user-images.githubusercontent.com/16397174/76160670-13cdc680-6167-11ea-9d03-d91246f57e5b.png">
After this PR without `stageId/StageAttemptId/taskId`:
<img width="734" alt="new_without_taskInfo" src="https://user-images.githubusercontent.com/16397174/76160672-16302080-6167-11ea-86e9-99b52ee3545f.png">
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Updated `SQLMetricsSuite` and tested manually.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #27849:
[SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of
sql metrics configurable in UI
Posted by GitBox <gi...@apache.org>.
gatorsmile commented on issue #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-598536237
@Ngone51 Let us focus on the high priority tasks. We can improve it by adding a checkbox in Spark 3.1?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27849: [SPARK-31081][UI][SQL]
Make SQLMetrics more readable from UI
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27849: [SPARK-31081][UI][SQL] Make SQLMetrics more readable from UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-596209367
**[Test build #119534 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119534/testReport)** for PR 27849 at commit [`6f61906`](https://github.com/apache/spark/commit/6f61906e8b840d8204941f94a8e9f0ed896bb4c1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on issue #27849:
[SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of
sql metrics configurable in UI
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on issue #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-598538460
I checked with @Ngone51 offline. He is fine with the checkbox idea. But he is not very familiar with the front-end work. I am also busy with other priorities recently.
@sarutak @HeartSaVioR would you like to take this task (create a checkbox for hild/show the max stage/stage attamp/task id in the graph) over?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27849:
[SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of
sql metrics configurable in UI
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-597673836
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24403/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on issue #27849:
[SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of
sql metrics configurable in UI
Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on issue #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-598539105
Thanks for pinging me and I'm sorry I'm not familiar with front-end work as well - that's why I only had to skim the front-end side while reviewing structured streaming UI PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27849:
[SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of
sql metrics configurable in UI
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-597672780
**[Test build #119674 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119674/testReport)** for PR 27849 at commit [`a41d9c4`](https://github.com/apache/spark/commit/a41d9c4e710539025b003e2c2f5c672f3ffb715b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27849:
[SPARK-31081][UI][SQL] Make SQLMetrics more readable from UI
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27849: [SPARK-31081][UI][SQL] Make SQLMetrics more readable from UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-596209548
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119534/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27849:
[SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of
sql metrics configurable in UI
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#discussion_r391187139
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala
##########
@@ -116,8 +117,8 @@ object SQLMetrics {
// data size total (min, med, max):
// 100GB (100MB, 1GB, 10GB)
val acc = new SQLMetric(SIZE_METRIC, -1)
- acc.register(sc, name = Some(s"$name total (min, med, max (stageId (attemptId): taskId))"),
- countFailedValues = false)
+ acc.register(sc, name = Some(s"$name total (min, med, ${attachTaskId("max", sc.conf)})"),
Review comment:
The new indentation looks incorrect at this line?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on issue #27849:
[SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of
sql metrics configurable in UI
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on issue #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-598534947
@tgravescs as @gatorsmile and I mentioned in https://github.com/apache/spark/pull/26843#issuecomment-572791878, the new change is not readable. I prefer to hide the slowest task as well.
> Can we leave on by default or don't display and have a checkbox like we have on other tables? (disclaimer - I haven't looked how hard that would be in code to add the checkbox)
I like the idea of "don't display and have a checkbox like we have on other tables". @Ngone51 will you consider that?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27849: [SPARK-31081][UI][SQL]
Make the display of stageId/stageAttemptId/taskId of sql metrics
configurable in UI
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-598531330
The UI is mostly for debugging but the stage id stuff is not always needed. At least this has been distracting me multiple times when I try to debug something with the SQL UI.
The new proposal from @Ngone51 looks good, it's much less distracting to have a short `(1.0:14)` at the end. @tgravescs If you don't like it, I think we at least should have this config so that people can disable it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #27849:
[SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of
sql metrics configurable in UI
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on issue #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-598539105
Thanks for pinging me and I'm sorry I'm not familiar with front-end work as well - that's why I only skimmed while reviewing structured streaming UI PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27849:
[SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of
sql metrics configurable in UI
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#discussion_r391425285
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala
##########
@@ -233,4 +228,13 @@ object SQLMetrics {
SparkListenerDriverAccumUpdates(executionId.toLong, metrics.map(m => m.id -> m.value)))
}
}
+
+ private def attachTaskId(name: String, conf: SparkConf): String = {
Review comment:
`attachTaskInfo`? stage id is part of the task info which tells which stage the task comes from.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27849: [SPARK-31081][UI][SQL]
Make SQLMetrics more readable from UI
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27849: [SPARK-31081][UI][SQL] Make SQLMetrics more readable from UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-596187989
**[Test build #119534 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119534/testReport)** for PR 27849 at commit [`6f61906`](https://github.com/apache/spark/commit/6f61906e8b840d8204941f94a8e9f0ed896bb4c1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27849:
[SPARK-31081][UI][SQL] Make SQLMetrics more readable from UI
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27849: [SPARK-31081][UI][SQL] Make SQLMetrics more readable from UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-596209548
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119534/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27849:
[SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of
sql metrics configurable in UI
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-597814423
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119674/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27849:
[SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of
sql metrics configurable in UI
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-597814411
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27849:
[SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of
sql metrics configurable in UI
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#discussion_r391424573
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
##########
@@ -135,6 +135,14 @@ object StaticSQLConf {
.intConf
.createWithDefault(1000)
+ val DISPLAY_TASK_ID_FOR_MAX_METRIC =
+ buildStaticConf("spark.sql.ui.displayTaskInfoForMaxMetric")
+ .doc("If turn on, Spark will display stageId-stageAttemptId-taskId of the max metrics to " +
+ "tell where the max value comes from. It's useful to help debug job quicker.")
+ .version("3.0.0")
Review comment:
This is a UI issue to me. The SQL web UI is really hard to read now with the stage id stuff. Ideally, we should revisit #26843 and think of a better way to improve readability. But no one has proposed an idea yet.
This PR disables the stage id stuff, which seems like a good compromise for 3.0: we keep the UI unchanged, but for people who really need the stage id info, they can still enable it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #27849:
[SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of
sql metrics configurable in UI
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#discussion_r391357062
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala
##########
@@ -233,4 +228,13 @@ object SQLMetrics {
SparkListenerDriverAccumUpdates(executionId.toLong, metrics.map(m => m.id -> m.value)))
}
}
+
+ private def attachTaskId(name: String, conf: SparkConf): String = {
Review comment:
Yeah, I know. I'm thinking about a better name, while adding stageId/attemptId makes me feel too long for the method name.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27849:
[SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of
sql metrics configurable in UI
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-597673836
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24403/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27849:
[SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of
sql metrics configurable in UI
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-597814411
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27849:
[SPARK-31081][UI][SQL] Make SQLMetrics more readable from UI
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27849: [SPARK-31081][UI][SQL] Make SQLMetrics more readable from UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-596188101
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #27849:
[SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of
sql metrics configurable in UI
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#discussion_r391356590
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala
##########
@@ -116,8 +117,8 @@ object SQLMetrics {
// data size total (min, med, max):
// 100GB (100MB, 1GB, 10GB)
val acc = new SQLMetric(SIZE_METRIC, -1)
- acc.register(sc, name = Some(s"$name total (min, med, max (stageId (attemptId): taskId))"),
- countFailedValues = false)
+ acc.register(sc, name = Some(s"$name total (min, med, ${attachTaskId("max", sc.conf)})"),
Review comment:
I also noted that, thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27849: [SPARK-31081][UI][SQL]
Make the display of stageId/stageAttemptId/taskId of sql metrics
configurable in UI
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-597813488
**[Test build #119674 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119674/testReport)** for PR 27849 at commit [`a41d9c4`](https://github.com/apache/spark/commit/a41d9c4e710539025b003e2c2f5c672f3ffb715b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gatorsmile removed a comment on issue #27849:
[SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of
sql metrics configurable in UI
Posted by GitBox <gi...@apache.org>.
gatorsmile removed a comment on issue #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-598536237
@Ngone51 Let us focus on the high priority tasks. We can improve it by adding a checkbox in Spark 3.1?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on issue #27849: [SPARK-31081][UI][SQL]
Make the display of stageId/stageAttemptId/taskId of sql metrics
configurable in UI
Posted by GitBox <gi...@apache.org>.
sarutak commented on issue #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-599303912
@gengliangwang I'll try to add the checkbox.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27849:
[SPARK-31081][UI][SQL] Make SQLMetrics more readable from UI
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27849: [SPARK-31081][UI][SQL] Make SQLMetrics more readable from UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-596209547
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #27849: [SPARK-31081][UI][SQL]
Make SQLMetrics more readable from UI
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on issue #27849: [SPARK-31081][UI][SQL] Make SQLMetrics more readable from UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-596200702
ping @cloud-fan @gengliangwang @nartal1 Please take a look, thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27849:
[SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of
sql metrics configurable in UI
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-597673816
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27849:
[SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of
sql metrics configurable in UI
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#discussion_r391187701
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala
##########
@@ -233,4 +228,13 @@ object SQLMetrics {
SparkListenerDriverAccumUpdates(executionId.toLong, metrics.map(m => m.id -> m.value)))
}
}
+
+ private def attachTaskId(name: String, conf: SparkConf): String = {
Review comment:
We are attaching `stageId` and `attemptId`, too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27849:
[SPARK-31081][UI][SQL] Make SQLMetrics more readable from UI
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27849: [SPARK-31081][UI][SQL] Make SQLMetrics more readable from UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-596188103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24265/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27849: [SPARK-31081][UI][SQL]
Make the display of stageId/stageAttemptId/taskId of sql metrics
configurable in UI
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-597672780
**[Test build #119674 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119674/testReport)** for PR 27849 at commit [`a41d9c4`](https://github.com/apache/spark/commit/a41d9c4e710539025b003e2c2f5c672f3ffb715b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27849:
[SPARK-31081][UI][SQL] Make SQLMetrics more readable from UI
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27849: [SPARK-31081][UI][SQL] Make SQLMetrics more readable from UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-596188101
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #27849:
[SPARK-31081][UI][SQL] Make metrics in SparkPlanGraphNode more readable
from Spark UI
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #27849: [SPARK-31081][UI][SQL] Make metrics in SparkPlanGraphNode more readable from Spark UI
URL: https://github.com/apache/spark/pull/27849#discussion_r390127370
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala
##########
@@ -126,16 +125,14 @@ object SQLMetrics {
// duration(min, med, max):
// 5s (800ms, 1s, 2s)
val acc = new SQLMetric(TIMING_METRIC, -1)
- acc.register(sc, name = Some(s"$name total (min, med, max (stageId (attemptId): taskId))"),
- countFailedValues = false)
+ acc.register(sc, name = Some(name), countFailedValues = false)
acc
}
def createNanoTimingMetric(sc: SparkContext, name: String): SQLMetric = {
// Same with createTimingMetric, just normalize the unit of time to millisecond.
val acc = new SQLMetric(NS_TIMING_METRIC, -1)
- acc.register(sc, name = Some(s"$name total (min, med, max (stageId (attemptId): taskId))"),
- countFailedValues = false)
+ acc.register(sc, name = Some(s"name"), countFailedValues = false)
Review comment:
TODO: fix "name"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27849:
[SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of
sql metrics configurable in UI
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-597814423
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119674/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on issue #27849: [SPARK-31081][UI][SQL]
Make the display of stageId/stageAttemptId/taskId of sql metrics
configurable in UI
Posted by GitBox <gi...@apache.org>.
tgravescs commented on issue #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-598431075
Sorry, I disagree. I understand that it adds more clutter and am definitely open to improvements, but I think it does bring valuable information that once people realize is there will use it. This entire ui is for debugging purposes so I don't get your argument with that. If a user isn't actively looking to see what went on with their job why would they go to this UI at all.
If you have a better way to map these SQL boxes to the stage it ran in, please let me know?
This also quickly tells you which task was the slowest to aid in debugging further.
Can we leave on by default or don't display and have a checkbox like we have on other tables? (disclaimer - I haven't looked how hard that would be in code to add the checkbox)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #27849: [SPARK-31081][UI][SQL]
Make the display of stageId/stageAttemptId/taskId of sql metrics
configurable in UI
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on issue #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-598507962
> This entire ui is for debugging purposes so I don't get your argument with that. If a user isn't actively looking to see what went on with their job why would they go to this UI at all.
I agree that UI is mostly for debugging purposes. But please also note that this is living UI(rather than SHS UI). And living UI may shutdown at any time before user figure out the root cause (except those long running applications). So, personally, I aways need to restart and insert `Thread.sleep` when I need to debug on living UI in case it shutdown before I figure out the root cause. So, I think user could always turn on `spark.sql.ui.displayTaskInfoForMaxMetric` when they realize they need to restart and debug on living UI.
And if they really want to see those ids at any time, they can just turn it on always.
Or if we can have better way to display ids, then we can turn it on by default. And here's my previous propose to display ids in case you would like it:
<img width="595" alt="ids" src="https://user-images.githubusercontent.com/16397174/76581609-f3a75a00-650e-11ea-89de-2edb947fc17d.png">
> I really hope in the future with all the things being reverted that I see we copy the original authors and reviewers.
Sorry, this is my bad. I'll take care in the future.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27849:
[SPARK-31081][UI][SQL] Make SQLMetrics more readable from UI
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27849: [SPARK-31081][UI][SQL] Make SQLMetrics more readable from UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-596187989
**[Test build #119534 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119534/testReport)** for PR 27849 at commit [`6f61906`](https://github.com/apache/spark/commit/6f61906e8b840d8204941f94a8e9f0ed896bb4c1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27849:
[SPARK-31081][UI][SQL] Make SQLMetrics more readable from UI
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27849: [SPARK-31081][UI][SQL] Make SQLMetrics more readable from UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-596209547
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27849:
[SPARK-31081][UI][SQL] Make SQLMetrics more readable from UI
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27849: [SPARK-31081][UI][SQL] Make SQLMetrics more readable from UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-596188103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24265/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on issue #27849: [SPARK-31081][UI][SQL]
Make the display of stageId/stageAttemptId/taskId of sql metrics
configurable in UI
Posted by GitBox <gi...@apache.org>.
tgravescs commented on issue #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-598431804
BTW thanks @gengliangwang for adding me, it would have been nice if I was added right away since I added this feature that you are now trying to take back out.
I really hope in the future with all the things being reverted that I see we copy the original authors and reviewers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27849:
[SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of
sql metrics configurable in UI
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#discussion_r391186535
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
##########
@@ -135,6 +135,14 @@ object StaticSQLConf {
.intConf
.createWithDefault(1000)
+ val DISPLAY_TASK_ID_FOR_MAX_METRIC =
+ buildStaticConf("spark.sql.ui.displayTaskInfoForMaxMetric")
+ .doc("If turn on, Spark will display stageId-stageAttemptId-taskId of the max metrics to " +
+ "tell where the max value comes from. It's useful to help debug job quicker.")
+ .version("3.0.0")
Review comment:
@Ngone51 .
Since SPARK-31081 is filed as an `Improvement`, this should be `3.1.0`.
If you want to consider this as a bug or regression at 3.0.0, you had better change JIRA first.
cc @rxin since he is the release manager of 3.0.0 (also cc @gatorsmile ).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on issue #27849:
[SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of
sql metrics configurable in UI
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on issue #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-598061822
cc @tgravescs as well
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #27849:
[SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of
sql metrics configurable in UI
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#discussion_r391433257
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
##########
@@ -135,6 +135,14 @@ object StaticSQLConf {
.intConf
.createWithDefault(1000)
+ val DISPLAY_TASK_ID_FOR_MAX_METRIC =
+ buildStaticConf("spark.sql.ui.displayTaskInfoForMaxMetric")
+ .doc("If turn on, Spark will display stageId-stageAttemptId-taskId of the max metrics to " +
+ "tell where the max value comes from. It's useful to help debug job quicker.")
+ .version("3.0.0")
Review comment:
+1 for making this 3.0. The UI changes in #26843 can be confusing to users after 3.0 release.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27849:
[SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of
sql metrics configurable in UI
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#discussion_r391424866
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
##########
@@ -135,6 +135,14 @@ object StaticSQLConf {
.intConf
.createWithDefault(1000)
+ val DISPLAY_TASK_ID_FOR_MAX_METRIC =
+ buildStaticConf("spark.sql.ui.displayTaskInfoForMaxMetric")
Review comment:
shall we make it an internal conf?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27849:
[SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of
sql metrics configurable in UI
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-597801676
Thank you for narrowing down title, @Ngone51 .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on issue #27849: [SPARK-31081][UI][SQL]
Make the display of stageId/stageAttemptId/taskId of sql metrics
configurable in UI
Posted by GitBox <gi...@apache.org>.
tgravescs commented on issue #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-598753739
the problem with a config is that if you turn it off, especially by default most users won't see it to know its even an option.
I'd much prefer the checkbox type approach (note again I haven't looked to see how complicated is with these boxes).
I'm also fine with the other format (0:0:14) as long as the legend if obvious on what it is. could have a hover there or something as well.
Perhaps I should ask everyone this - when you were looking at the UI, why were you looking at it? What values in there were you using for instance? A lot of people have asked me ok, I did this groupby or join and I see it in that box, but how do I know what stage that ran in..... this at least helps with that question. So how do you do that now? my previous answer was you had to click through a lot of jobs/stages to figure it out, which is very time consuming.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27849:
[SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of
sql metrics configurable in UI
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27849: [SPARK-31081][UI][SQL] Make the display of stageId/stageAttemptId/taskId of sql metrics configurable in UI
URL: https://github.com/apache/spark/pull/27849#issuecomment-597673816
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org