You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "liuzqt (via GitHub)" <gi...@apache.org> on 2025/07/22 01:03:34 UTC
[PR] [SPARK-51505][SQL] Always show empty partition number metrics in AQEShuffleReadExec [spark]
liuzqt opened a new pull request, #51608:
URL: https://github.com/apache/spark/pull/51608
### What changes were proposed in this pull request?
Always show empty partition number metrics in AQEShuffleReadExec
### Why are the changes needed?
Even when there is no coaclescing, it could be still confusing: imaging we have shuffle skewness and each non-empty partition is large so no coalescing happen, but there are many empty partitions.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Update UT
### Was this patch authored or co-authored using generative AI tooling?
NO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51505][SQL] Always show empty partition number metrics in AQEShuffleReadExec [spark]
Posted by "the-sakthi (via GitHub)" <gi...@apache.org>.
the-sakthi commented on code in PR #51608:
URL: https://github.com/apache/spark/pull/51608#discussion_r2226662563
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala:
##########
@@ -1092,7 +1092,7 @@ class AdaptiveQueryExecSuite
}
}
- test("metrics of the shuffle read") {
+ test("AABCmetrics of the shuffle read") {
Review Comment:
"AABCmetrics" -- any specific meaning to this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51505][SQL] Always show empty partition number metrics in AQEShuffleReadExec [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #51608: [SPARK-51505][SQL] Always show empty partition number metrics in AQEShuffleReadExec
URL: https://github.com/apache/spark/pull/51608
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51505][SQL] Always show empty partition number metrics in AQEShuffleReadExec [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #51608:
URL: https://github.com/apache/spark/pull/51608#issuecomment-3101633073
The test failure in `AdaptiveQueryExecSuite` seems to be related
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51505][SQL] Always show empty partition number metrics in AQEShuffleReadExec [spark]
Posted by "liuzqt (via GitHub)" <gi...@apache.org>.
liuzqt commented on code in PR #51608:
URL: https://github.com/apache/spark/pull/51608#discussion_r2227098082
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala:
##########
@@ -1092,7 +1092,7 @@ class AdaptiveQueryExecSuite
}
}
- test("metrics of the shuffle read") {
+ test("AABCmetrics of the shuffle read") {
Review Comment:
Sorry some random inputs slip into my commits...cleaned up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51505][SQL] Always show empty partition number metrics in AQEShuffleReadExec [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #51608:
URL: https://github.com/apache/spark/pull/51608#issuecomment-3111910944
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51505][SQL] Always show empty partition number metrics in AQEShuffleReadExec [spark]
Posted by "liuzqt (via GitHub)" <gi...@apache.org>.
liuzqt commented on PR #51608:
URL: https://github.com/apache/spark/pull/51608#issuecomment-3100624598
@cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org