You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/07/01 03:52:28 UTC
[GitHub] [spark] edwinalu opened a new pull request #23340: [SPARK-23431][CORE] Expose the new executor memory metrics at the stage level
edwinalu opened a new pull request #23340:
URL: https://github.com/apache/spark/pull/23340
## What changes were proposed in this pull request?
Collect and show the new executor memory metrics for each stage, to provide more information on how memory is used per stage. Peak values for metrics are show for each stage. For executor summaries for each stage, the peak values per executor are also shown.
## How was this patch tested?
Added new unit tests.
Please review http://spark.apache.org/contributing.html before opening a pull request.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itamarst commented on pull request #23340: [SPARK-23431][CORE] Expose the new executor memory metrics at the stage level
Posted by GitBox <gi...@apache.org>.
itamarst commented on pull request #23340:
URL: https://github.com/apache/spark/pull/23340#issuecomment-653129600
@imback82 I'll be taking this up if you don't (on behalf of @stackedsax), but since I am not a Spark contributor at the moment it'd probably be a _lot_ faster and easier if you did it. So please do look into it—thank you!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] stackedsax commented on pull request #23340: [SPARK-23431][CORE] Expose the new executor memory metrics at the stage level
Posted by GitBox <gi...@apache.org>.
stackedsax commented on pull request #23340:
URL: https://github.com/apache/spark/pull/23340#issuecomment-652176572
@edwinalu thanks for getting this started so long ago. Looks like there are some conflicts after so much time has passed. Do you have time/interest to continue working on this or would you like some help getting this sorted out?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] stackedsax commented on pull request #23340: [SPARK-23431][CORE] Expose the new executor memory metrics at the stage level
Posted by GitBox <gi...@apache.org>.
stackedsax commented on pull request #23340:
URL: https://github.com/apache/spark/pull/23340#issuecomment-648466419
@squito @rezasafi any chance either of you could review this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #23340: [SPARK-23431][CORE] Expose the new executor memory metrics at the stage level
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #23340:
URL: https://github.com/apache/spark/pull/23340#issuecomment-730726078
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #23340: [SPARK-23431][CORE] Expose the new executor memory metrics at the stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #23340:
URL: https://github.com/apache/spark/pull/23340#issuecomment-531895234
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] edwinalu commented on pull request #23340: [SPARK-23431][CORE] Expose the new executor memory metrics at the stage level
Posted by GitBox <gi...@apache.org>.
edwinalu commented on pull request #23340:
URL: https://github.com/apache/spark/pull/23340#issuecomment-652443118
@stackedsax , I don't have time to continue working on this unfortunately. If you or someone else is able to help with adding stage level metrics, that would be great. Please let me know if there's any questions about the original PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #23340: [SPARK-23431][CORE] Expose the new executor memory metrics at the stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #23340:
URL: https://github.com/apache/spark/pull/23340#issuecomment-652174521
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] stackedsax commented on pull request #23340: [SPARK-23431][CORE] Expose the new executor memory metrics at the stage level
Posted by GitBox <gi...@apache.org>.
stackedsax commented on pull request #23340:
URL: https://github.com/apache/spark/pull/23340#issuecomment-672196580
@imback82 Looks like #29020 is merged into 3.1.0. Should this issue get closed now, or is it being left open for someone to do the same in 2.x versions of Spark?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] imback82 commented on pull request #23340: [SPARK-23431][CORE] Expose the new executor memory metrics at the stage level
Posted by GitBox <gi...@apache.org>.
imback82 commented on pull request #23340:
URL: https://github.com/apache/spark/pull/23340#issuecomment-672217163
This can be closed now. We can use changes in #29020 if backporting to 2.4 is needed. (@dongjoon-hyun / @gengliangwang can confirm, but I don't think this will be backported to 2.4.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itamarst commented on pull request #23340: [SPARK-23431][CORE] Expose the new executor memory metrics at the stage level
Posted by GitBox <gi...@apache.org>.
itamarst commented on pull request #23340:
URL: https://github.com/apache/spark/pull/23340#issuecomment-634067377
Ping—any chance someone could review this patch? These are useful metrics.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #23340: [SPARK-23431][CORE] Expose the new executor memory metrics at the stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #23340:
URL: https://github.com/apache/spark/pull/23340#issuecomment-652174521
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #23340: [SPARK-23431][CORE] Expose the new executor memory metrics at the stage level
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #23340:
URL: https://github.com/apache/spark/pull/23340#issuecomment-652174792
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] imback82 commented on pull request #23340: [SPARK-23431][CORE] Expose the new executor memory metrics at the stage level
Posted by GitBox <gi...@apache.org>.
imback82 commented on pull request #23340:
URL: https://github.com/apache/spark/pull/23340#issuecomment-652777567
@stackedsax, do you plan to work on this? Otherwise, I can start looking into this next week. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #23340: [SPARK-23431][CORE] Expose the new executor memory metrics at the stage level
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #23340:
URL: https://github.com/apache/spark/pull/23340
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] imback82 commented on pull request #23340: [SPARK-23431][CORE] Expose the new executor memory metrics at the stage level
Posted by GitBox <gi...@apache.org>.
imback82 commented on pull request #23340:
URL: https://github.com/apache/spark/pull/23340#issuecomment-654654972
I created a PR here: #29020. I will also ask a reviewing committer to put @edwinalu as a co-author if that's the practice when a PR is "inherited". Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #23340: [SPARK-23431][CORE] Expose the new executor memory metrics at the stage level
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #23340:
URL: https://github.com/apache/spark/pull/23340#issuecomment-652173340
This is reopened according to the community request. I also removed the label, `Stale`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] imback82 commented on pull request #23340: [SPARK-23431][CORE] Expose the new executor memory metrics at the stage level
Posted by GitBox <gi...@apache.org>.
imback82 commented on pull request #23340:
URL: https://github.com/apache/spark/pull/23340#issuecomment-653314080
Thanks @itamarst. I started looking into this and will update this thread.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org