You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/08/11 02:51:31 UTC

[GitHub] [spark] AngersZhuuuu opened a new pull request #33696: [WIP][SPARK-36437][CORE] Add non-unified memory peak metrics

AngersZhuuuu opened a new pull request #33696:
URL: https://github.com/apache/spark/pull/33696


   ### What changes were proposed in this pull request?
   Add non-unified memory metrics to have evidence  for change config of spark.memory.fraction
   
   
   ### Why are the changes needed?
   
   
   ### Does this PR introduce _any_ user-facing change?
   
   
   ### How was this patch tested?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33696: [SPARK-36437][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #33696:
URL: https://github.com/apache/spark/pull/33696#discussion_r687469984



##########
File path: core/src/main/scala/org/apache/spark/metrics/ExecutorMetricType.scala
##########
@@ -177,6 +177,16 @@ case object OnHeapUnifiedMemory extends MemoryManagerExecutorMetricType(
 case object OffHeapUnifiedMemory extends MemoryManagerExecutorMetricType(
   (m => m.offHeapExecutionMemoryUsed + m.offHeapStorageMemoryUsed))
 
+case object OnHeapUnUnifiedMemory extends MemoryManagerExecutorMetricType(

Review comment:
       > where do we calculate the peak value? the framework?
   
   Yea. 
   
   ```
   case object JVMHeapMemory extends SingleValueExecutorMetricType {
     override private[spark] def getMetricValue(memoryManager: MemoryManager): Long = {
       ManagementFactory.getMemoryMXBean.getHeapMemoryUsage().getUsed()
     }
   }
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on pull request #33696: [SPARK-36473][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33696:
URL: https://github.com/apache/spark/pull/33696#issuecomment-913420702


   ping @cloud-fan 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33696: [WIP][SPARK-36437][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33696:
URL: https://github.com/apache/spark/pull/33696#issuecomment-896508090


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142303/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33696: [WIP][SPARK-36437][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33696:
URL: https://github.com/apache/spark/pull/33696#issuecomment-896466961


   **[Test build #142303 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142303/testReport)** for PR 33696 at commit [`a935be0`](https://github.com/apache/spark/commit/a935be03e1a05f8ae18ec45f00583f72ac19257b).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #33696: [SPARK-36437][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33696:
URL: https://github.com/apache/spark/pull/33696#issuecomment-896573281


   **[Test build #142313 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142313/testReport)** for PR 33696 at commit [`b33ad90`](https://github.com/apache/spark/commit/b33ad90675910357f976266099883f8c7067441e).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #33696: [SPARK-36437][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #33696:
URL: https://github.com/apache/spark/pull/33696#discussion_r686646248



##########
File path: core/src/main/scala/org/apache/spark/metrics/ExecutorMetricType.scala
##########
@@ -177,6 +177,16 @@ case object OnHeapUnifiedMemory extends MemoryManagerExecutorMetricType(
 case object OffHeapUnifiedMemory extends MemoryManagerExecutorMetricType(
   (m => m.offHeapExecutionMemoryUsed + m.offHeapStorageMemoryUsed))
 
+case object OnHeapUnUnifiedMemory extends MemoryManagerExecutorMetricType(
+  (m => ManagementFactory.getMemoryMXBean.getHeapMemoryUsage().getUsed()
+    - m.onHeapExecutionMemoryUsed - m.onHeapStorageMemoryUsed)
+)
+
+case object OffHeapUnUnifiedMemory extends MemoryManagerExecutorMetricType(
+  (m => ManagementFactory.getMemoryMXBean.getNonHeapMemoryUsage().getUsed()

Review comment:
       are you sure this include off-heap memory?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33696: [WIP][SPARK-36437][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33696:
URL: https://github.com/apache/spark/pull/33696#issuecomment-896606668


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46821/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33696: [WIP][SPARK-36437][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33696:
URL: https://github.com/apache/spark/pull/33696#issuecomment-896528145


   **[Test build #142307 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142307/testReport)** for PR 33696 at commit [`67b807e`](https://github.com/apache/spark/commit/67b807e3f1696220126f15d366223d0c0abac42b).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33696: [WIP][SPARK-36437][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33696:
URL: https://github.com/apache/spark/pull/33696#issuecomment-896573281


   **[Test build #142313 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142313/testReport)** for PR 33696 at commit [`b33ad90`](https://github.com/apache/spark/commit/b33ad90675910357f976266099883f8c7067441e).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33696: [SPARK-36437][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33696:
URL: https://github.com/apache/spark/pull/33696#issuecomment-896739064


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142313/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33696: [SPARK-36437][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33696:
URL: https://github.com/apache/spark/pull/33696#issuecomment-896504400






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33696: [WIP][SPARK-36437][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33696:
URL: https://github.com/apache/spark/pull/33696#issuecomment-896507398


   **[Test build #142303 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142303/testReport)** for PR 33696 at commit [`a935be0`](https://github.com/apache/spark/commit/a935be03e1a05f8ae18ec45f00583f72ac19257b).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] github-actions[bot] closed pull request #33696: [SPARK-36473][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #33696:
URL: https://github.com/apache/spark/pull/33696


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #33696: [SPARK-36437][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #33696:
URL: https://github.com/apache/spark/pull/33696#discussion_r686647927



##########
File path: core/src/main/scala/org/apache/spark/metrics/ExecutorMetricType.scala
##########
@@ -177,6 +177,16 @@ case object OnHeapUnifiedMemory extends MemoryManagerExecutorMetricType(
 case object OffHeapUnifiedMemory extends MemoryManagerExecutorMetricType(
   (m => m.offHeapExecutionMemoryUsed + m.offHeapStorageMemoryUsed))
 
+case object OnHeapUnUnifiedMemory extends MemoryManagerExecutorMetricType(

Review comment:
       why do we need this? `JVMHeapMemory` reports it already and people can easily calculate it by themselves.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33696: [SPARK-36473][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33696:
URL: https://github.com/apache/spark/pull/33696#issuecomment-896739064


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142313/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33696: [WIP][SPARK-36437][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33696:
URL: https://github.com/apache/spark/pull/33696#issuecomment-896606628


   Kubernetes integration test unable to build dist.
   
   exiting with code: 1
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46821/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #33696: [WIP][SPARK-36437][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33696:
URL: https://github.com/apache/spark/pull/33696#issuecomment-896484513


   **[Test build #142307 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142307/testReport)** for PR 33696 at commit [`67b807e`](https://github.com/apache/spark/commit/67b807e3f1696220126f15d366223d0c0abac42b).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #33696: [SPARK-36437][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #33696:
URL: https://github.com/apache/spark/pull/33696#discussion_r686647184



##########
File path: core/src/main/scala/org/apache/spark/metrics/ExecutorMetricType.scala
##########
@@ -177,6 +177,16 @@ case object OnHeapUnifiedMemory extends MemoryManagerExecutorMetricType(
 case object OffHeapUnifiedMemory extends MemoryManagerExecutorMetricType(
   (m => m.offHeapExecutionMemoryUsed + m.offHeapStorageMemoryUsed))
 
+case object OnHeapUnUnifiedMemory extends MemoryManagerExecutorMetricType(
+  (m => ManagementFactory.getMemoryMXBean.getHeapMemoryUsage().getUsed()

Review comment:
       is there any document to prove this can return the used JVM heap memory?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33696: [WIP][SPARK-36437][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33696:
URL: https://github.com/apache/spark/pull/33696#issuecomment-896504400


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46810/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33696: [SPARK-36437][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #33696:
URL: https://github.com/apache/spark/pull/33696#discussion_r686657719



##########
File path: core/src/main/scala/org/apache/spark/metrics/ExecutorMetricType.scala
##########
@@ -177,6 +177,16 @@ case object OnHeapUnifiedMemory extends MemoryManagerExecutorMetricType(
 case object OffHeapUnifiedMemory extends MemoryManagerExecutorMetricType(
   (m => m.offHeapExecutionMemoryUsed + m.offHeapStorageMemoryUsed))
 
+case object OnHeapUnUnifiedMemory extends MemoryManagerExecutorMetricType(

Review comment:
       > why do we need this? `JVMHeapMemory` reports it already and people can easily calculate it by themselves.
   
   Always we use event log/restful API to get the peakMemoryMetrics. what we can get is the peakJVMOnHeap/peakOnHeapStorageMemory/peakOnHeapExecution memory during a period of time.
   But we can use the final status of peakJVMOnHeap/peakOnHeapStorageMemory/peakOnHeapExecution  to calculate  and get the peak unmanaged memory usage.
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33696: [WIP][SPARK-36437][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33696:
URL: https://github.com/apache/spark/pull/33696#issuecomment-896484513


   **[Test build #142307 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142307/testReport)** for PR 33696 at commit [`67b807e`](https://github.com/apache/spark/commit/67b807e3f1696220126f15d366223d0c0abac42b).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #33696: [SPARK-36437][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #33696:
URL: https://github.com/apache/spark/pull/33696#discussion_r687452058



##########
File path: core/src/main/scala/org/apache/spark/metrics/ExecutorMetricType.scala
##########
@@ -177,6 +177,16 @@ case object OnHeapUnifiedMemory extends MemoryManagerExecutorMetricType(
 case object OffHeapUnifiedMemory extends MemoryManagerExecutorMetricType(
   (m => m.offHeapExecutionMemoryUsed + m.offHeapStorageMemoryUsed))
 
+case object OnHeapUnUnifiedMemory extends MemoryManagerExecutorMetricType(

Review comment:
       where do we calculate the peak value? the framework?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33696: [SPARK-36437][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33696:
URL: https://github.com/apache/spark/pull/33696#issuecomment-896737719


   **[Test build #142313 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142313/testReport)** for PR 33696 at commit [`b33ad90`](https://github.com/apache/spark/commit/b33ad90675910357f976266099883f8c7067441e).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33696: [WIP][SPARK-36437][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33696:
URL: https://github.com/apache/spark/pull/33696#issuecomment-896513931


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46815/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #33696: [WIP][SPARK-36437][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33696:
URL: https://github.com/apache/spark/pull/33696#issuecomment-896466961


   **[Test build #142303 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142303/testReport)** for PR 33696 at commit [`a935be0`](https://github.com/apache/spark/commit/a935be03e1a05f8ae18ec45f00583f72ac19257b).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33696: [WIP][SPARK-36437][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33696:
URL: https://github.com/apache/spark/pull/33696#issuecomment-896513964


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46815/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33696: [WIP][SPARK-36437][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33696:
URL: https://github.com/apache/spark/pull/33696#issuecomment-896501041


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46810/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33696: [WIP][SPARK-36437][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33696:
URL: https://github.com/apache/spark/pull/33696#issuecomment-896529385


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142307/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33696: [WIP][SPARK-36437][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33696:
URL: https://github.com/apache/spark/pull/33696#issuecomment-896481018


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46810/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] github-actions[bot] commented on pull request #33696: [SPARK-36473][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #33696:
URL: https://github.com/apache/spark/pull/33696#issuecomment-995315755


   We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
   If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33696: [SPARK-36437][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #33696:
URL: https://github.com/apache/spark/pull/33696#discussion_r688313188



##########
File path: core/src/main/scala/org/apache/spark/metrics/ExecutorMetricType.scala
##########
@@ -177,6 +177,16 @@ case object OnHeapUnifiedMemory extends MemoryManagerExecutorMetricType(
 case object OffHeapUnifiedMemory extends MemoryManagerExecutorMetricType(
   (m => m.offHeapExecutionMemoryUsed + m.offHeapStorageMemoryUsed))
 
+case object OnHeapUnUnifiedMemory extends MemoryManagerExecutorMetricType(

Review comment:
       Any more suggestion? 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33696: [WIP][SPARK-36437][CORE] Add non-unified memory peak metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33696:
URL: https://github.com/apache/spark/pull/33696#issuecomment-896499785


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46815/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org