You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "liuzqt (via GitHub)" <gi...@apache.org> on 2025/03/14 00:22:11 UTC
[PR] [SPARK-51505][SQL] Log empty partition number metrics in AQE coalesce [spark]
liuzqt opened a new pull request, #50273:
URL: https://github.com/apache/spark/pull/50273
### What changes were proposed in this pull request?
Log empty partition number metrics in AQE coalesce
### Why are the changes needed?
There're cases where shuffle is highly skewed and many partitions are empty (probably due to small NDV) and only a few partitons contains data and are very large. AQE coalesce here basically did nothing by eliminate the empty partitions. AQE coalesce metrics might look confusing and user might think it wrongly coalesce to large partitions.
We'd better log empty partition number in the metrics.
### Does this PR introduce _any_ user-facing change?
Added `numEmptyPartitions` metrics to AQEShuffleReadExec when there is coalescing.
### How was this patch tested?
New test case.
### Was this patch authored or co-authored using generative AI tooling?
NO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51505][SQL] Log empty partition number metrics in AQE coalesce [spark]
Posted by "maryannxue (via GitHub)" <gi...@apache.org>.
maryannxue commented on code in PR #50273:
URL: https://github.com/apache/spark/pull/50273#discussion_r2003821231
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AQEShuffleReadExec.scala:
##########
@@ -37,10 +37,13 @@ import org.apache.spark.sql.vectorized.ColumnarBatch
* node during canonicalization.
* @param partitionSpecs The partition specs that defines the arrangement, requires at least one
* partition.
+ * @param numEmptyPartitions Number of empty partitions. This is only present when there is
+ * coalescing
*/
case class AQEShuffleReadExec private(
child: SparkPlan,
- partitionSpecs: Seq[ShufflePartitionSpec]) extends UnaryExecNode {
+ partitionSpecs: Seq[ShufflePartitionSpec],
+ numEmptyPartitions: Option[Int] = None) extends UnaryExecNode {
Review Comment:
nit: make this `numEmptyPartitions: Int = 0`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51505][SQL] Log empty partition number metrics in AQE coalesce [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #50273:
URL: https://github.com/apache/spark/pull/50273#discussion_r2010173181
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AQEShuffleReadExec.scala:
##########
@@ -200,7 +200,14 @@ case class AQEShuffleReadExec private(
val numCoalescedPartitionsMetric = metrics("numCoalescedPartitions")
val x = partitionSpecs.count(isCoalescedSpec)
numCoalescedPartitionsMetric.set(x)
- driverAccumUpdates += numCoalescedPartitionsMetric.id -> x
+ val numEmptyPartitionsMetric = metrics("numEmptyPartitions")
+ val y = child match {
+ case s: ShuffleQueryStageExec =>
Review Comment:
We may not reach it for canonicalized plans or other unknown cases. Shall we be conservative and add a `case _ => 0`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51505][SQL] Log empty partition number metrics in AQE coalesce [spark]
Posted by "liuzqt (via GitHub)" <gi...@apache.org>.
liuzqt commented on code in PR #50273:
URL: https://github.com/apache/spark/pull/50273#discussion_r2004339184
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AQEShuffleReadExec.scala:
##########
@@ -37,10 +37,13 @@ import org.apache.spark.sql.vectorized.ColumnarBatch
* node during canonicalization.
* @param partitionSpecs The partition specs that defines the arrangement, requires at least one
* partition.
+ * @param numEmptyPartitions Number of empty partitions. This is only present when there is
+ * coalescing
*/
case class AQEShuffleReadExec private(
child: SparkPlan,
- partitionSpecs: Seq[ShufflePartitionSpec]) extends UnaryExecNode {
+ partitionSpecs: Seq[ShufflePartitionSpec],
+ numEmptyPartitions: Option[Int] = None) extends UnaryExecNode {
Review Comment:
Do you think we need to distinguish `None`(for non-coalesce `AQEShuffleReadExec`) and `0`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51505][SQL] Log empty partition number metrics in AQE coalesce [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #50273:
URL: https://github.com/apache/spark/pull/50273#issuecomment-2731840772
can you provide some screenshots before and after this change? Let's see all the metrics available today and how this new metric adds value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51505][SQL] Log empty partition number metrics in AQE coalesce [spark]
Posted by "liuzqt (via GitHub)" <gi...@apache.org>.
liuzqt commented on PR #50273:
URL: https://github.com/apache/spark/pull/50273#issuecomment-2734945883
> can you provide some screenshots before and after this change? Let's see all the metrics available today and how this new metric adds value.
Added Spark UI screenshot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51505][SQL] Log empty partition number metrics in AQE coalesce [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #50273: [SPARK-51505][SQL] Log empty partition number metrics in AQE coalesce
URL: https://github.com/apache/spark/pull/50273
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51505][SQL] Log empty partition number metrics in AQE coalesce [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #50273:
URL: https://github.com/apache/spark/pull/50273#discussion_r2007069742
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AQEShuffleReadExec.scala:
##########
@@ -200,7 +203,11 @@ case class AQEShuffleReadExec private(
val numCoalescedPartitionsMetric = metrics("numCoalescedPartitions")
val x = partitionSpecs.count(isCoalescedSpec)
numCoalescedPartitionsMetric.set(x)
- driverAccumUpdates += numCoalescedPartitionsMetric.id -> x
+ val numEmptyPartitionsMetric = metrics("numEmptyPartitions")
Review Comment:
can we simply calculate the metric value here by looking at the underlying shuffle stage? e.g.
```
child match {
case s: ShuffleMapStage =>
s.mapStats.map(stats => stats.bytesByPartitionId.count(_ == 0)).getOrElse(0)
case => 0
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51505][SQL] Log empty partition number metrics in AQE coalesce [spark]
Posted by "liuzqt (via GitHub)" <gi...@apache.org>.
liuzqt commented on code in PR #50273:
URL: https://github.com/apache/spark/pull/50273#discussion_r2008176849
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AQEShuffleReadExec.scala:
##########
@@ -200,7 +203,11 @@ case class AQEShuffleReadExec private(
val numCoalescedPartitionsMetric = metrics("numCoalescedPartitions")
val x = partitionSpecs.count(isCoalescedSpec)
numCoalescedPartitionsMetric.set(x)
- driverAccumUpdates += numCoalescedPartitionsMetric.id -> x
+ val numEmptyPartitionsMetric = metrics("numEmptyPartitions")
Review Comment:
Good point! refactored
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51505][SQL] Log empty partition number metrics in AQE coalesce [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #50273:
URL: https://github.com/apache/spark/pull/50273#discussion_r2007070496
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AQEShuffleReadExec.scala:
##########
@@ -200,7 +203,11 @@ case class AQEShuffleReadExec private(
val numCoalescedPartitionsMetric = metrics("numCoalescedPartitions")
val x = partitionSpecs.count(isCoalescedSpec)
numCoalescedPartitionsMetric.set(x)
- driverAccumUpdates += numCoalescedPartitionsMetric.id -> x
+ val numEmptyPartitionsMetric = metrics("numEmptyPartitions")
Review Comment:
Populating the new field `numEmptyPartitions` for `AQEShuffleReadExec` looks very cumbersome and needs invasive code changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51505][SQL] Log empty partition number metrics in AQE coalesce [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #50273:
URL: https://github.com/apache/spark/pull/50273#issuecomment-2750688379
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51505][SQL] Log empty partition number metrics in AQE coalesce [spark]
Posted by "liuzqt (via GitHub)" <gi...@apache.org>.
liuzqt commented on PR #50273:
URL: https://github.com/apache/spark/pull/50273#issuecomment-2722992293
@maryannxue @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org