You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "peter-toth (via GitHub)" <gi...@apache.org> on 2023/02/05 10:18:28 UTC
[GitHub] [spark] peter-toth opened a new pull request, #39887: [SPARK-42346][SQL] Rewrite distinct aggregates after subquery merge
peter-toth opened a new pull request, #39887:
URL: https://github.com/apache/spark/pull/39887
### What changes were proposed in this pull request?
Unfortunately https://github.com/apache/spark/pull/32298 introduced a regression from Spark 3.2 to 3.3 as after that change a merged subquery can contain multiple distict type aggregates. Those aggregates need to be rewritten by the `RewriteDistinctAggregates` rule to get the correct results. This PR fixed that.
### Why are the changes needed?
The following query:
```
SELECT
(SELECT count(distinct c1) FROM t1),
(SELECT count(distinct c2) FROM t1)
```
currently fails with:
```
java.lang.IllegalStateException: You hit a query analyzer bug. Please report your query to Spark user mailing list.
at org.apache.spark.sql.execution.SparkStrategies$Aggregation$.apply(SparkStrategies.scala:538)
```
but works again after this PR.
### Does this PR introduce _any_ user-facing change?
Yes, the above query works again.
### How was this patch tested?
Added new UT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #39887: [SPARK-42346][SQL] Rewrite distinct aggregates after subquery merge
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on PR #39887:
URL: https://github.com/apache/spark/pull/39887#issuecomment-1419081359
Merged to master, branch-3.4 and branch-3.3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum closed pull request #39887: [SPARK-42346][SQL] Rewrite distinct aggregates after subquery merge
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum closed pull request #39887: [SPARK-42346][SQL] Rewrite distinct aggregates after subquery merge
URL: https://github.com/apache/spark/pull/39887
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a diff in pull request #39887: [SPARK-42346][SQL] Rewrite distinct aggregates after subquery merge
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #39887:
URL: https://github.com/apache/spark/pull/39887#discussion_r1096671489
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -2598,4 +2598,29 @@ class SubquerySuite extends QueryTest
Row("aa"))
}
}
+
+ test("SPARK-42346: Rewrite distinct aggregates after merging subqueries") {
+ withTempView("t1") {
+ Seq((1, 2), (3, 4)).toDF("c1", "c2").createOrReplaceTempView("t1")
+
+ checkAnswer(sql(
+ """
+ |SELECT
+ | (SELECT count(distinct c1) FROM t1),
+ | (SELECT count(distinct c2) FROM t1)
+ |""".stripMargin),
+ Row(2, 2))
Review Comment:
It seems `MergeScalarSubqueries` can not improve this case?
Spark 3.2:
```
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- Project [Subquery subquery#185, [id=#223] AS scalarsubquery()#198L, Subquery subquery#186, [id=#242] AS scalarsubquery()#199L]
: :- Subquery subquery#185, [id=#223]
: : +- AdaptiveSparkPlan isFinalPlan=false
: : +- HashAggregate(keys=[], functions=[count(distinct c1#153)], output=[count(DISTINCT c1)#193L])
: : +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=221]
: : +- HashAggregate(keys=[], functions=[partial_count(distinct c1#153)], output=[count#202L])
: : +- HashAggregate(keys=[c1#153], functions=[], output=[c1#153])
: : +- Exchange hashpartitioning(c1#153, 200), ENSURE_REQUIREMENTS, [plan_id=217]
: : +- HashAggregate(keys=[c1#153], functions=[], output=[c1#153])
: : +- FileScan parquet default.t1[c1#153] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/Users/yumwang/Downloads/spark-3.2.3-bin-hadoop2.7/spark-warehous..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<c1:int>
: +- Subquery subquery#186, [id=#242]
: +- AdaptiveSparkPlan isFinalPlan=false
: +- HashAggregate(keys=[], functions=[count(distinct c2#197)], output=[count(DISTINCT c2)#195L])
: +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=240]
: +- HashAggregate(keys=[], functions=[partial_count(distinct c2#197)], output=[count#206L])
: +- HashAggregate(keys=[c2#197], functions=[], output=[c2#197])
: +- Exchange hashpartitioning(c2#197, 200), ENSURE_REQUIREMENTS, [plan_id=236]
: +- HashAggregate(keys=[c2#197], functions=[], output=[c2#197])
: +- FileScan parquet default.t1[c2#197] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/Users/yumwang/Downloads/spark-3.2.3-bin-hadoop2.7/spark-warehous..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<c2:int>
+- Scan OneRowRelation[]
```
After `MergeScalarSubqueries` and this PR:
```
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- Project [Subquery subquery#0, [id=#36].count(DISTINCT c1) AS scalarsubquery()#11L, Subquery subquery#1, [id=#63].count(DISTINCT c2) AS scalarsubquery()#12L]
: :- Subquery subquery#0, [id=#36]
: : +- AdaptiveSparkPlan isFinalPlan=false
: : +- Project [named_struct(count(DISTINCT c1), count(DISTINCT c1)#5L, count(DISTINCT c2), count(DISTINCT c2)#8L) AS mergedValue#17]
: : +- HashAggregate(keys=[], functions=[count(spark_catalog.default.t1.c1#19), count(c2#20)], output=[count(DISTINCT c1)#5L, count(DISTINCT c2)#8L])
: : +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=33]
: : +- HashAggregate(keys=[], functions=[partial_count(spark_catalog.default.t1.c1#19) FILTER (WHERE (gid#18 = 1)), partial_count(c2#20) FILTER (WHERE (gid#18 = 2))], output=[count#23L, count#24L])
: : +- HashAggregate(keys=[spark_catalog.default.t1.c1#19, c2#20, gid#18], functions=[], output=[spark_catalog.default.t1.c1#19, c2#20, gid#18])
: : +- Exchange hashpartitioning(spark_catalog.default.t1.c1#19, c2#20, gid#18, 5), ENSURE_REQUIREMENTS, [plan_id=29]
: : +- HashAggregate(keys=[spark_catalog.default.t1.c1#19, c2#20, gid#18], functions=[], output=[spark_catalog.default.t1.c1#19, c2#20, gid#18])
: : +- Expand [[c1#2, null, 1], [null, c2#3, 2]], [spark_catalog.default.t1.c1#19, c2#20, gid#18]
: : +- FileScan parquet spark_catalog.default.t1[c1#2,c2#3] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/Users/yumwang/opensource/spark/spark-warehouse/org.apache.spark...., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<c1:int,c2:int>
: +- Subquery subquery#1, [id=#63]
: +- AdaptiveSparkPlan isFinalPlan=false
: +- Project [named_struct(count(DISTINCT c1), count(DISTINCT c1)#5L, count(DISTINCT c2), count(DISTINCT c2)#8L) AS mergedValue#17]
: +- HashAggregate(keys=[], functions=[count(spark_catalog.default.t1.c1#19), count(c2#20)], output=[count(DISTINCT c1)#5L, count(DISTINCT c2)#8L])
: +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=60]
: +- HashAggregate(keys=[], functions=[partial_count(spark_catalog.default.t1.c1#19) FILTER (WHERE (gid#18 = 1)), partial_count(c2#20) FILTER (WHERE (gid#18 = 2))], output=[count#23L, count#24L])
: +- HashAggregate(keys=[spark_catalog.default.t1.c1#19, c2#20, gid#18], functions=[], output=[spark_catalog.default.t1.c1#19, c2#20, gid#18])
: +- Exchange hashpartitioning(spark_catalog.default.t1.c1#19, c2#20, gid#18, 5), ENSURE_REQUIREMENTS, [plan_id=56]
: +- HashAggregate(keys=[spark_catalog.default.t1.c1#19, c2#20, gid#18], functions=[], output=[spark_catalog.default.t1.c1#19, c2#20, gid#18])
: +- Expand [[c1#2, null, 1], [null, c2#3, 2]], [spark_catalog.default.t1.c1#19, c2#20, gid#18]
: +- FileScan parquet spark_catalog.default.t1[c1#2,c2#3] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/Users/yumwang/opensource/spark/spark-warehouse/org.apache.spark...., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<c1:int,c2:int>
+- Scan OneRowRelation[]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #39887: [SPARK-42346][SQL] Rewrite distinct aggregates after subquery merge
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #39887:
URL: https://github.com/apache/spark/pull/39887#issuecomment-1419035721
late LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] peter-toth commented on pull request #39887: [SPARK-42346][SQL] Rewrite distinct aggregates after subquery merge
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #39887:
URL: https://github.com/apache/spark/pull/39887#issuecomment-1419102325
Thanks for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] peter-toth commented on a diff in pull request #39887: [SPARK-42346][SQL] Rewrite distinct aggregates after subquery merge
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #39887:
URL: https://github.com/apache/spark/pull/39887#discussion_r1096707745
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -2598,4 +2598,29 @@ class SubquerySuite extends QueryTest
Row("aa"))
}
}
+
+ test("SPARK-42346: Rewrite distinct aggregates after merging subqueries") {
+ withTempView("t1") {
+ Seq((1, 2), (3, 4)).toDF("c1", "c2").createOrReplaceTempView("t1")
+
+ checkAnswer(sql(
+ """
+ |SELECT
+ | (SELECT count(distinct c1) FROM t1),
+ | (SELECT count(distinct c2) FROM t1)
+ |""".stripMargin),
+ Row(2, 2))
Review Comment:
No, the 2 plans differ as expected. After `MergeScalarSubqueries` the 2 subqueries became identical and both contain `count(DISTINCT c1)` and `count(DISTINCT c2)` in `mergedValue`.
The final physical plan with the reuse makes the improvement more visible:
```
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(1) Project [Subquery subquery#251, [id=#117].count(DISTINCT c1) AS scalarsubquery()#259L, ReusedSubquery Subquery subquery#251, [id=#117].count(DISTINCT c2) AS scalarsubquery()#260L]
: :- Subquery subquery#251, [id=#117]
: : +- AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(3) Project [named_struct(count(DISTINCT c1), count(DISTINCT c1)#254L, count(DISTINCT c2), count(DISTINCT c2)#256L) AS mergedValue#265]
+- *(3) HashAggregate(keys=[], functions=[count(t1.c1#267), count(c2#268)], output=[count(DISTINCT c1)#254L, count(DISTINCT c2)#256L])
+- ShuffleQueryStage 1
+- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=228]
+- *(2) HashAggregate(keys=[], functions=[partial_count(t1.c1#267) FILTER (WHERE (gid#266 = 1)), partial_count(c2#268) FILTER (WHERE (gid#266 = 2))], output=[count#271L, count#272L])
+- *(2) HashAggregate(keys=[t1.c1#267, c2#268, gid#266], functions=[], output=[t1.c1#267, c2#268, gid#266])
+- AQEShuffleRead coalesced
+- ShuffleQueryStage 0
+- Exchange hashpartitioning(t1.c1#267, c2#268, gid#266, 5), ENSURE_REQUIREMENTS, [plan_id=193]
+- *(1) HashAggregate(keys=[t1.c1#267, c2#268, gid#266], functions=[], output=[t1.c1#267, c2#268, gid#266])
+- *(1) Expand [[c1#247, null, 1], [null, c2#248, 2]], [t1.c1#267, c2#268, gid#266]
+- *(1) Project [_1#242 AS c1#247, _2#243 AS c2#248]
+- *(1) LocalTableScan [_1#242, _2#243]
+- == Initial Plan ==
Project [named_struct(count(DISTINCT c1), count(DISTINCT c1)#254L, count(DISTINCT c2), count(DISTINCT c2)#256L) AS mergedValue#265]
+- HashAggregate(keys=[], functions=[count(t1.c1#267), count(c2#268)], output=[count(DISTINCT c1)#254L, count(DISTINCT c2)#256L])
+- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=114]
+- HashAggregate(keys=[], functions=[partial_count(t1.c1#267) FILTER (WHERE (gid#266 = 1)), partial_count(c2#268) FILTER (WHERE (gid#266 = 2))], output=[count#271L, count#272L])
+- HashAggregate(keys=[t1.c1#267, c2#268, gid#266], functions=[], output=[t1.c1#267, c2#268, gid#266])
+- Exchange hashpartitioning(t1.c1#267, c2#268, gid#266, 5), ENSURE_REQUIREMENTS, [plan_id=110]
+- HashAggregate(keys=[t1.c1#267, c2#268, gid#266], functions=[], output=[t1.c1#267, c2#268, gid#266])
+- Expand [[c1#247, null, 1], [null, c2#248, 2]], [t1.c1#267, c2#268, gid#266]
+- Project [_1#242 AS c1#247, _2#243 AS c2#248]
+- LocalTableScan [_1#242, _2#243]
: +- ReusedSubquery Subquery subquery#251, [id=#117]
+- *(1) Scan OneRowRelation[]
+- == Initial Plan ==
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a diff in pull request #39887: [SPARK-42346][SQL] Rewrite distinct aggregates after subquery merge
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #39887:
URL: https://github.com/apache/spark/pull/39887#discussion_r1096671489
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -2598,4 +2598,29 @@ class SubquerySuite extends QueryTest
Row("aa"))
}
}
+
+ test("SPARK-42346: Rewrite distinct aggregates after merging subqueries") {
+ withTempView("t1") {
+ Seq((1, 2), (3, 4)).toDF("c1", "c2").createOrReplaceTempView("t1")
+
+ checkAnswer(sql(
+ """
+ |SELECT
+ | (SELECT count(distinct c1) FROM t1),
+ | (SELECT count(distinct c2) FROM t1)
+ |""".stripMargin),
+ Row(2, 2))
Review Comment:
It seems `MergeScalarSubqueries` can not improve this case:
Spark 3.2:
```
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- Project [Subquery subquery#185, [id=#223] AS scalarsubquery()#198L, Subquery subquery#186, [id=#242] AS scalarsubquery()#199L]
: :- Subquery subquery#185, [id=#223]
: : +- AdaptiveSparkPlan isFinalPlan=false
: : +- HashAggregate(keys=[], functions=[count(distinct c1#153)], output=[count(DISTINCT c1)#193L])
: : +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=221]
: : +- HashAggregate(keys=[], functions=[partial_count(distinct c1#153)], output=[count#202L])
: : +- HashAggregate(keys=[c1#153], functions=[], output=[c1#153])
: : +- Exchange hashpartitioning(c1#153, 200), ENSURE_REQUIREMENTS, [plan_id=217]
: : +- HashAggregate(keys=[c1#153], functions=[], output=[c1#153])
: : +- FileScan parquet default.t1[c1#153] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/Users/yumwang/Downloads/spark-3.2.3-bin-hadoop2.7/spark-warehous..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<c1:int>
: +- Subquery subquery#186, [id=#242]
: +- AdaptiveSparkPlan isFinalPlan=false
: +- HashAggregate(keys=[], functions=[count(distinct c2#197)], output=[count(DISTINCT c2)#195L])
: +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=240]
: +- HashAggregate(keys=[], functions=[partial_count(distinct c2#197)], output=[count#206L])
: +- HashAggregate(keys=[c2#197], functions=[], output=[c2#197])
: +- Exchange hashpartitioning(c2#197, 200), ENSURE_REQUIREMENTS, [plan_id=236]
: +- HashAggregate(keys=[c2#197], functions=[], output=[c2#197])
: +- FileScan parquet default.t1[c2#197] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/Users/yumwang/Downloads/spark-3.2.3-bin-hadoop2.7/spark-warehous..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<c2:int>
+- Scan OneRowRelation[]
```
After `MergeScalarSubqueries` and this PR:
```
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- Project [Subquery subquery#0, [id=#36].count(DISTINCT c1) AS scalarsubquery()#11L, Subquery subquery#1, [id=#63].count(DISTINCT c2) AS scalarsubquery()#12L]
: :- Subquery subquery#0, [id=#36]
: : +- AdaptiveSparkPlan isFinalPlan=false
: : +- Project [named_struct(count(DISTINCT c1), count(DISTINCT c1)#5L, count(DISTINCT c2), count(DISTINCT c2)#8L) AS mergedValue#17]
: : +- HashAggregate(keys=[], functions=[count(spark_catalog.default.t1.c1#19), count(c2#20)], output=[count(DISTINCT c1)#5L, count(DISTINCT c2)#8L])
: : +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=33]
: : +- HashAggregate(keys=[], functions=[partial_count(spark_catalog.default.t1.c1#19) FILTER (WHERE (gid#18 = 1)), partial_count(c2#20) FILTER (WHERE (gid#18 = 2))], output=[count#23L, count#24L])
: : +- HashAggregate(keys=[spark_catalog.default.t1.c1#19, c2#20, gid#18], functions=[], output=[spark_catalog.default.t1.c1#19, c2#20, gid#18])
: : +- Exchange hashpartitioning(spark_catalog.default.t1.c1#19, c2#20, gid#18, 5), ENSURE_REQUIREMENTS, [plan_id=29]
: : +- HashAggregate(keys=[spark_catalog.default.t1.c1#19, c2#20, gid#18], functions=[], output=[spark_catalog.default.t1.c1#19, c2#20, gid#18])
: : +- Expand [[c1#2, null, 1], [null, c2#3, 2]], [spark_catalog.default.t1.c1#19, c2#20, gid#18]
: : +- FileScan parquet spark_catalog.default.t1[c1#2,c2#3] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/Users/yumwang/opensource/spark/spark-warehouse/org.apache.spark...., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<c1:int,c2:int>
: +- Subquery subquery#1, [id=#63]
: +- AdaptiveSparkPlan isFinalPlan=false
: +- Project [named_struct(count(DISTINCT c1), count(DISTINCT c1)#5L, count(DISTINCT c2), count(DISTINCT c2)#8L) AS mergedValue#17]
: +- HashAggregate(keys=[], functions=[count(spark_catalog.default.t1.c1#19), count(c2#20)], output=[count(DISTINCT c1)#5L, count(DISTINCT c2)#8L])
: +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=60]
: +- HashAggregate(keys=[], functions=[partial_count(spark_catalog.default.t1.c1#19) FILTER (WHERE (gid#18 = 1)), partial_count(c2#20) FILTER (WHERE (gid#18 = 2))], output=[count#23L, count#24L])
: +- HashAggregate(keys=[spark_catalog.default.t1.c1#19, c2#20, gid#18], functions=[], output=[spark_catalog.default.t1.c1#19, c2#20, gid#18])
: +- Exchange hashpartitioning(spark_catalog.default.t1.c1#19, c2#20, gid#18, 5), ENSURE_REQUIREMENTS, [plan_id=56]
: +- HashAggregate(keys=[spark_catalog.default.t1.c1#19, c2#20, gid#18], functions=[], output=[spark_catalog.default.t1.c1#19, c2#20, gid#18])
: +- Expand [[c1#2, null, 1], [null, c2#3, 2]], [spark_catalog.default.t1.c1#19, c2#20, gid#18]
: +- FileScan parquet spark_catalog.default.t1[c1#2,c2#3] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/Users/yumwang/opensource/spark/spark-warehouse/org.apache.spark...., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<c1:int,c2:int>
+- Scan OneRowRelation[]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] peter-toth commented on pull request #39887: [SPARK-42346][SQL] Rewrite distinct aggregates after subquery merge
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #39887:
URL: https://github.com/apache/spark/pull/39887#issuecomment-1417361582
cc @cloud-fan, @viirya, @wangyum
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] RobinL commented on pull request #39887: [SPARK-42346][SQL] Rewrite distinct aggregates after subquery merge
Posted by "RobinL (via GitHub)" <gi...@apache.org>.
RobinL commented on PR #39887:
URL: https://github.com/apache/spark/pull/39887#issuecomment-1417413523
Thanks so much @peter-toth! Amazing to have a fix so quickly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org