You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Eugene Koifman (Jira)" <ji...@apache.org> on 2021/07/14 15:50:00 UTC
[jira] [Reopened] (SPARK-35639) Add metrics about coalesced
partitions to CustomShuffleReader in AQE
[ https://issues.apache.org/jira/browse/SPARK-35639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eugene Koifman reopened SPARK-35639:
------------------------------------
[~cloud_fan] this is not done yet.
The original PR that actually adds metrics is [https://github.com/apache/spark/pull/32776] and is still open.
I was asked to split it into 2 so only the 2nd part got merged so far: [https://github.com/apache/spark/pull/32872.]
I'll rebase and update [https://github.com/apache/spark/pull/32776] that will actually add metrics.
> Add metrics about coalesced partitions to CustomShuffleReader in AQE
> --------------------------------------------------------------------
>
> Key: SPARK-35639
> URL: https://issues.apache.org/jira/browse/SPARK-35639
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Eugene Koifman
> Assignee: Eugene Koifman
> Priority: Major
> Fix For: 3.2.0
>
>
> {{CustomShuffleReaderExec}} reports "number of skewed partitions" and "number of skewed partition splits".
> It would be useful to also report "number of partitions to coalesce" and "number of coalesced partitions" and include this in string rendering of the SparkPlan node so that it looks like this
> {code:java}
> (12) CustomShuffleReader
> Input [2]: [a#23, b#24]
> Arguments: coalesced 3 partitions into 1 and split 2 skewed partitions into 4
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org