You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/03/28 06:54:30 UTC
[GitHub] [spark] AngersZhuuuu opened a new pull request #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
AngersZhuuuu opened a new pull request #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061
### What changes were proposed in this pull request?
Expose executor metrics in task detail table in StageLevel
### Why are the changes needed?
help developer to check app running status.
### Does this PR introduce any user-facing change?
NO
### How was this patch tested?
MT
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605628524
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25261/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#discussion_r405293901
##########
File path: core/src/main/resources/org/apache/spark/ui/static/stagepage.js
##########
@@ -327,6 +336,13 @@ $(document).ready(function () {
"should be approximately the sum of the peak sizes across all such data structures created " +
"in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
"external sort.");
+ $('#peak_jvm_on_heap_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "top")
+ .attr("title", "Peak Executor JVM on-heap memory usage during this task running");
Review comment:
`tasks` might be proper rather than `this task`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605545966
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120542/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605545302
cc @sarutak
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on issue #28061: [SPARK-31270][CORE]
Expose executor metrics in task detail table in StageLevel
Posted by GitBox <gi...@apache.org>.
sarutak commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605597663
I'll review within a week.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605545964
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on issue #28061: [SPARK-31270][CORE]
Expose executor metrics in task detail table in StageLevel
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605548557
> Maybe,
> `peakJvmHeapMemory` -> `peakJvmOnHeapMemory`?
> `Peak JVM Heap Memory` -> `Peak JVM On Heap Memory`?
Sure, make it more clear
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605642876
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#discussion_r399836877
##########
File path: core/src/main/resources/org/apache/spark/ui/static/stagepage.js
##########
@@ -572,7 +600,7 @@ $(document).ready(function () {
var row1 = createRowMetadataForColumn(
columnKey, taskMetricsResponse[columnKey], 4);
var row2 = createRowMetadataForColumn(
- "shuffleWriteTime", taskMetricsResponse[columnKey], 21);
+ "shuffleWriteTime", taskMetricsResponse[columnKey], 23);
Review comment:
That's the exact reason why I asked that. I've seen several UI patches like this. :)
> Since add new column in Additional Metric, the column index is changed. So need to do this
There exists lots of similar commits already for new develop of Spark web UI. Always, the commits(already reviewed and merged) are the best demo for developers.
> If it's ok, this pr can be a demo for new develope of spark web UI
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605667321
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120556/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605667319
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#discussion_r399734867
##########
File path: core/src/main/resources/org/apache/spark/ui/static/stagepage.js
##########
@@ -572,7 +600,7 @@ $(document).ready(function () {
var row1 = createRowMetadataForColumn(
columnKey, taskMetricsResponse[columnKey], 4);
var row2 = createRowMetadataForColumn(
- "shuffleWriteTime", taskMetricsResponse[columnKey], 21);
+ "shuffleWriteTime", taskMetricsResponse[columnKey], 23);
Review comment:
> If possible, could you try to a
Since add new column in Additional Metric, the column index is changed. So need to do this
To be honest , I spend a lot of time to make sure the column change when add new column, since the spark web ui code changed a lot, since origin pr in `stagepage.js` have not do things like add column.
If it's ok, this pr can be a demo for new develope of spark web UI
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605628521
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on issue #28061: [SPARK-31270][CORE]
Expose executor metrics in task detail table in StageLevel
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-610804127
cc @sarutak @gengliangwang Close it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605406013
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605544925
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605628521
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605406013
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605545964
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#discussion_r399893110
##########
File path: core/src/main/resources/org/apache/spark/ui/static/stagepage.js
##########
@@ -572,7 +600,7 @@ $(document).ready(function () {
var row1 = createRowMetadataForColumn(
columnKey, taskMetricsResponse[columnKey], 4);
var row2 = createRowMetadataForColumn(
- "shuffleWriteTime", taskMetricsResponse[columnKey], 21);
+ "shuffleWriteTime", taskMetricsResponse[columnKey], 23);
Review comment:
> There exists lots of similar commits already for new develop of Spark web UI. Always, the commits(already reviewed and merged) are the best demo for developers.
Each page's JS script isn't the same, and I have find that for stage page, don't have merged pr about add new additional column.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605628402
**[Test build #120555 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120555/testReport)** for PR 28061 at commit [`48a355d`](https://github.com/apache/spark/commit/48a355d19536274966c4f619af53905adf38e3a8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605647422
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25262/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on issue #28061: [SPARK-31270][CORE]
Expose executor metrics in task detail table in StageLevel
Posted by GitBox <gi...@apache.org>.
sarutak commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-610779977
O.K, I understand the motivation.
I confirmed that those peak metrics are calculated for each task so the fundamental idea might be fine even though over-wrapping or consecutive tasks are affected each other like `GC Time`.
What do you think, @dongjoon-hyun , @gengliangwang ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605667321
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120556/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on issue #28061: [SPARK-31270][CORE]
Expose executor metrics in task detail table in StageLevel
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-610796096
> @AngersZhuuuu Sorry to reply late.
> Actually, I am slightly -1 on the changes since it is not straightforward enough. Even @sarutak has to think about the meaning of the metrics.
>
> How about just displaying the max on/off heap memory in the executor page?
how about this ? https://github.com/apache/spark/pull/28036
https://github.com/apache/spark/pull/28036#issuecomment-610725617
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605544928
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25248/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on issue #28061: [SPARK-31270][CORE]
Expose executor metrics in task detail table in StageLevel
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-609606587
> So I wonder it's not helpful that those metrics are displayed for each task.
Such as Stage Executor Metrics, it show each executors's peak memory during this stage, but there is still a problem that during this stage, other stage's task will run in same executor too when executor's core num is bigger than 1.
We just show each task's running status about JVM, as an SRE, we need to know task running status as many as possible. Any one of these metrics can help the user find problems and provide help.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605628524
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25261/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605647216
**[Test build #120556 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120556/testReport)** for PR 28061 at commit [`bbea4f9`](https://github.com/apache/spark/commit/bbea4f9f929db25d76eb5b137f7f0ae6eb5ea4e4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605647414
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28061: [SPARK-31270][CORE]
Expose executor metrics in task detail table in StageLevel
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605544831
**[Test build #120542 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120542/testReport)** for PR 28061 at commit [`7807901`](https://github.com/apache/spark/commit/7807901ab636c0f582c4d70729c5caa62176fd1e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-610797332
>how about this ? #28036
#28036 (comment)
That's way better!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605545966
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120542/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605642881
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120555/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605544831
**[Test build #120542 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120542/testReport)** for PR 28061 at commit [`7807901`](https://github.com/apache/spark/commit/7807901ab636c0f582c4d70729c5caa62176fd1e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-610794328
@AngersZhuuuu Sorry to reply late.
Actually, I am slightly -1 on the changes since it is not straightforward enough. Even @sarutak has to think about the meaning of the metrics.
How about just displaying the max on/off heap memory in the executor page?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605647414
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605667319
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605406102
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605647422
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25262/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on issue #28061: [SPARK-31270][CORE]
Expose executor metrics in task detail table in StageLevel
Posted by GitBox <gi...@apache.org>.
sarutak commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-610803719
I also prefer that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-610801357
@sarutak I prefer the approach in https://github.com/apache/spark/pull/28036 .
What do you think? If so, shall we close this one?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on issue #28061: [SPARK-31270][CORE]
Expose executor metrics in task detail table in StageLevel
Posted by GitBox <gi...@apache.org>.
sarutak commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-609501342
Additional comments:
* Could you explain more about manual test in the description so that we can understand what and how you tested?
* With this change, appearance of UI changes so this change should be user facing.
* Please update screen shots when you update the code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on issue #28061: [SPARK-31270][CORE]
Expose executor metrics in task detail table in StageLevel
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-610797883
> That's way better!
ok, I will focus on that pr and follow your advise
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28061: [SPARK-31270][CORE]
Expose executor metrics in task detail table in StageLevel
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605642718
**[Test build #120555 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120555/testReport)** for PR 28061 at commit [`48a355d`](https://github.com/apache/spark/commit/48a355d19536274966c4f619af53905adf38e3a8).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28061: [SPARK-31270][CORE]
Expose executor metrics in task detail table in StageLevel
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605666984
**[Test build #120556 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120556/testReport)** for PR 28061 at commit [`bbea4f9`](https://github.com/apache/spark/commit/bbea4f9f929db25d76eb5b137f7f0ae6eb5ea4e4).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605642876
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#discussion_r405294705
##########
File path: core/src/main/resources/org/apache/spark/ui/static/stagepage.js
##########
@@ -327,6 +336,13 @@ $(document).ready(function () {
"should be approximately the sum of the peak sizes across all such data structures created " +
"in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
"external sort.");
+ $('#peak_jvm_on_heap_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "top")
+ .attr("title", "Peak Executor JVM on-heap memory usage during this task running");
+ $('#peak_jvm_off_heap_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "top")
+ .attr("title", "Peak Executor JVM off-heap memory usage during this task running");
Review comment:
Same comment as [here](https://github.com/apache/spark/pull/28061/files#diff-c59528e611faa69b0d84b0ab25b2b2d7R341).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605544928
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25248/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu closed pull request #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu closed pull request #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28061: [SPARK-31270][CORE]
Expose executor metrics in task detail table in StageLevel
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605647216
**[Test build #120556 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120556/testReport)** for PR 28061 at commit [`bbea4f9`](https://github.com/apache/spark/commit/bbea4f9f929db25d76eb5b137f7f0ae6eb5ea4e4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605681520
Thank you, @sarutak !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605544925
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#discussion_r403767340
##########
File path: core/src/main/scala/org/apache/spark/status/LiveEntity.scala
##########
@@ -129,13 +129,19 @@ private class LiveTask(
// checked when calculating indexed values when writing to the store (see [[TaskDataWrapper]]).
private var metrics: v1.TaskMetrics = createMetrics(default = -1L)
+ val executorMetrics = new ExecutorMetrics()
+
var errorMessage: Option[String] = None
/**
* Update the metrics for the task and return the difference between the previous and new
* values.
*/
- def updateMetrics(metrics: TaskMetrics): v1.TaskMetrics = {
+ def updateMetrics(metrics: TaskMetrics,
+ executorMetrics: Option[ExecutorMetrics] = None): v1.TaskMetrics = {
+ executorMetrics.foreach {
+ this.executorMetrics.compareAndUpdatePeakValues
+ }
Review comment:
How about `executorMetrics.foreach(this.executorMetrics.compareAndUpdatePeakValues)`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28061: [SPARK-31270][CORE]
Expose executor metrics in task detail table in StageLevel
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605628402
**[Test build #120555 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120555/testReport)** for PR 28061 at commit [`48a355d`](https://github.com/apache/spark/commit/48a355d19536274966c4f619af53905adf38e3a8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang edited a comment on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
gengliangwang edited a comment on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-610794328
@AngersZhuuuu Sorry to reply late.
Actually, I am slightly -1 on the changes since it is not straightforward enough. Even @sarutak has to think about the meaning of the metrics. The existing task metrics are the sum of values from executors, while the new one is the max of values from executors.
How about just displaying the max on/off heap memory in the executor page?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605406102
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#discussion_r399732247
##########
File path: core/src/main/resources/org/apache/spark/ui/static/stagepage.js
##########
@@ -572,7 +600,7 @@ $(document).ready(function () {
var row1 = createRowMetadataForColumn(
columnKey, taskMetricsResponse[columnKey], 4);
var row2 = createRowMetadataForColumn(
- "shuffleWriteTime", taskMetricsResponse[columnKey], 21);
+ "shuffleWriteTime", taskMetricsResponse[columnKey], 23);
Review comment:
Hi, @AngersZhuuuu . If possible, could you try to avoid this change?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#discussion_r405311697
##########
File path: core/src/main/resources/org/apache/spark/ui/static/stagepage.js
##########
@@ -572,7 +600,7 @@ $(document).ready(function () {
var row1 = createRowMetadataForColumn(
columnKey, taskMetricsResponse[columnKey], 4);
var row2 = createRowMetadataForColumn(
- "shuffleWriteTime", taskMetricsResponse[columnKey], 21);
+ "shuffleWriteTime", taskMetricsResponse[columnKey], 23);
Review comment:
@AngersZhuuuu I found following issues.
1. When a job performs shuffle, StagePage for shuffle write stage has a problem.
Even if you visit StagePage where the corresponding stage performs shuffle write and check `Shuffle Write Time`,
the metrics is not shown.
<img width="1426" alt="スクリーンショット 2020-04-08 16 14 30" src="https://user-images.githubusercontent.com/4736016/78756030-136a5a80-79b5-11ea-90f3-917e31839588.png">
2. Web UI will be hung-up
When you visit a StagePage where the corresponding stage performs shuffle read and reload, UI will be hung-up.
<img width="1438" alt="hungup" src="https://user-images.githubusercontent.com/4736016/78756200-5f1d0400-79b5-11ea-967a-6992fd4b5a4f.png">
Those issues are caused because column indices are not changed properly.
To avoid such issues, please keep it simple. I believe column indices need not to be changed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28061: [SPARK-31270][CORE]
Expose executor metrics in task detail table in StageLevel
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605545956
**[Test build #120542 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120542/testReport)** for PR 28061 at commit [`7807901`](https://github.com/apache/spark/commit/7807901ab636c0f582c4d70729c5caa62176fd1e).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605544617
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28061:
[SPARK-31270][CORE] Expose executor metrics in task detail table in
StageLevel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28061: [SPARK-31270][CORE] Expose executor metrics in task detail table in StageLevel
URL: https://github.com/apache/spark/pull/28061#issuecomment-605642881
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120555/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org