[jira] [Commented] (SPARK-48789) Deprecate peakExecutionMemory metrics

["Josh Rosen (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-48789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17863402#comment-17863402 ] 

Josh Rosen commented on SPARK-48789:
------------------------------------

Agreed: the existing `peakExecutionMemory` metric is fairly inconsistent in its coverage and misses many important sources of allocation. It was originally added while the MemoryManager abstractions were being developed and was never fully unified. It also predated support for off-heap memory. For all of these reasons, I'm supportive of deprecating and replacing it.

> Deprecate peakExecutionMemory metrics
> -------------------------------------
>
>                 Key: SPARK-48789
>                 URL: https://issues.apache.org/jira/browse/SPARK-48789
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 4.0.0
>            Reporter: Ziqi Liu
>            Priority: Major
>
> Currently we have a `peakExecutionMemory` in `TaskMetrics`([code|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala#L109-L114]), as described in the code comments:
> {code:java}
> /**
>  * Peak memory used by internal data structures created during shuffles, aggregations and
>  * joins. The value of this accumulator should be approximately the sum of the peak sizes
>  * across all such data structures created in this task. For SQL jobs, this only tracks all
>  * unsafe operators and ExternalSort.
>  */ {code}
> The naming is confusing since this does not really reflect the whole execution memory, thus it's barely useful.
>  
> As a followup for https://issues.apache.org/jira/browse/SPARK-48628 and https://issues.apache.org/jira/browse/SPARK-48788
> once we land both, we should have a good replacement about task execution memory metrics, then let's consider deprecate this one, hopefully in Spark 4.0



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org