You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/10/30 03:27:30 UTC
[GitHub] [spark] huaxingao opened a new pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
huaxingao opened a new pull request #34445:
URL: https://github.com/apache/spark/pull/34445
### What changes were proposed in this pull request?
lift the restriction for aggregate push down for parquet and orc if group by is partition col
### Why are the changes needed?
to complete the file source aggregate push down work
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
new tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-972118667
**[Test build #145344 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145344/testReport)** for PR 34445 at commit [`b561d09`](https://github.com/apache/spark/commit/b561d09b48b4cd4ad9996d2c0fb0f16378f77ee5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-972166078
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145344/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34445:
URL: https://github.com/apache/spark/pull/34445#discussion_r750838217
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileSourceAggregatePushDownSuite.scala
##########
@@ -261,6 +261,65 @@ trait FileSourceAggregatePushDownSuite
}
}
+ test("aggregate with partition group by can be pushed down") {
+ withTempPath { dir =>
+ spark.range(10).selectExpr("id", "id % 3 as p")
+ .write.partitionBy("p").format(format).save(dir.getCanonicalPath)
+ withTempView("tmp") {
+ spark.read.format(format).load(dir.getCanonicalPath).createOrReplaceTempView("tmp");
+ Seq("false", "true").foreach { enableVectorizedReader =>
+ withSQLConf(aggPushDownEnabledKey -> "true",
+ vectorizedReaderEnabledKey -> enableVectorizedReader) {
+ val df = sql("SELECT count(*), count(id), p, max(id), p, count(p), max(id)," +
+ " min(id), p FROM tmp group by p")
+ df.queryExecution.optimizedPlan.collect {
+ case _: DataSourceV2ScanRelation =>
+ val expected_plan_fragment =
+ "PushedAggregation: [COUNT(*), COUNT(id), MAX(id), COUNT(p), MIN(id)], " +
+ "PushedFilters: [], PushedGroupBy: [p]"
+ checkKeywordsExistsInExplain(df, expected_plan_fragment)
+ }
+ checkAnswer(df, Seq(Row(3, 3, 1, 7, 1, 3, 7, 1, 1), Row(3, 3, 2, 8, 2, 3, 8, 2, 2),
+ Row(4, 4, 0, 9, 0, 4, 9, 0, 0)))
+ }
+ }
+ }
+ }
+ }
+
+ test("aggregate with multi partition group by columns can be pushed down") {
+ withTempPath { dir =>
+ Seq((10, 1, 2, 5, 6), (2, 1, 2, 5, 6), (3, 2, 1, 4, 8), (4, 2, 1, 4, 9),
+ (5, 2, 1, 5, 8), (6, 2, 1, 4, 8), (1, 1, 2, 5, 6), (4, 1, 2, 5, 6),
+ (3, 2, 2, 9, 10), (-4, 2, 2, 9, 10), (6, 2, 2, 9, 10))
+ .toDF("value", "p1", "p2", "p3", "p4")
+ .write
+ .partitionBy("p2", "p1", "p4", "p3")
+ .format(format)
+ .save(dir.getCanonicalPath)
+ withTempView("tmp") {
+ spark.read.format(format).load(dir.getCanonicalPath).createOrReplaceTempView("tmp");
+ Seq("false", "true").foreach { enableVectorizedReader =>
+ withSQLConf(aggPushDownEnabledKey -> "true",
Review comment:
here too. We should make sure `aggPushDownEnabledKey` won't change results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-962520215
**[Test build #144957 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144957/testReport)** for PR 34445 at commit [`0e655a8`](https://github.com/apache/spark/commit/0e655a8dedf278d52a661abf836d2e061412c03e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-955138077
**[Test build #144773 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144773/testReport)** for PR 34445 at commit [`9ea1c2d`](https://github.com/apache/spark/commit/9ea1c2d5b34abe9994f9e0ceedeea837436c8788).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-955150621
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49242/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-971016562
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49764/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-957975899
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-958012816
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49325/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34445:
URL: https://github.com/apache/spark/pull/34445#discussion_r741361243
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala
##########
@@ -457,17 +457,22 @@ object OrcUtils extends Logging {
}
}
+ // if there are group by columns, we will build result row first,
+ // and then append group by columns values (partition columns values) to the result row.
+ val schemaWithoutGroupby =
Review comment:
nit: `schemaWithoutGroupBy`
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/AggregatePushDownUtils.scala
##########
@@ -138,4 +141,19 @@ object AggregatePushDownUtils {
converter.convert(aggregatesAsRow, columnVectors.toArray)
new ColumnarBatch(columnVectors.asInstanceOf[Array[ColumnVector]], 1)
}
+
+ /**
+ * Return the schema for aggregates only (exclude group by columns)
+ */
+ def getSchemaWithoutGroupingExpression(
+ aggregation: Aggregation,
+ aggSchema: StructType): StructType = {
+ val groupByColNums = aggregation.groupByColumns.length
Review comment:
nit: `groupByColumns`
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetUtils.scala
##########
@@ -195,7 +201,12 @@ object ParquetUtils {
case (_, i) =>
throw new SparkException("Unexpected parquet type name: " + primitiveTypeNames(i))
}
- converter.currentRecord
+
+ if (aggregation.groupByColumns.length > 0) {
+ new JoinedRow(partitionValues, converter.currentRecord)
Review comment:
just curious how we make sure the `partitionValues` is always consistent with `aggregation.groupByColumns`? in terms of ordering.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetUtils.scala
##########
@@ -195,7 +201,12 @@ object ParquetUtils {
case (_, i) =>
throw new SparkException("Unexpected parquet type name: " + primitiveTypeNames(i))
}
- converter.currentRecord
+
+ if (aggregation.groupByColumns.length > 0) {
Review comment:
nit: maybe use `aggregation.groupByColumns.nonEmpty`
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/AggregatePushDownUtils.scala
##########
@@ -138,4 +141,19 @@ object AggregatePushDownUtils {
converter.convert(aggregatesAsRow, columnVectors.toArray)
new ColumnarBatch(columnVectors.asInstanceOf[Array[ColumnVector]], 1)
}
+
+ /**
+ * Return the schema for aggregates only (exclude group by columns)
+ */
+ def getSchemaWithoutGroupingExpression(
Review comment:
nit: maybe `getSchemaWithoutGroupColumns`?
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileSourceAggregatePushDownSuite.scala
##########
@@ -261,6 +261,63 @@ trait FileSourceAggregatePushDownSuite
}
}
+ test("aggregate with partition group by can be pushed down") {
Review comment:
nit: add JIRA number
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-958104543
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49325/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #34445:
URL: https://github.com/apache/spark/pull/34445#discussion_r744174328
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/AggregatePushDownUtils.scala
##########
@@ -138,4 +141,19 @@ object AggregatePushDownUtils {
converter.convert(aggregatesAsRow, columnVectors.toArray)
new ColumnarBatch(columnVectors.asInstanceOf[Array[ColumnVector]], 1)
}
+
+ /**
+ * Return the schema for aggregates only (exclude group by columns)
+ */
+ def getSchemaWithoutGroupingExpression(
+ aggregation: Aggregation,
+ aggSchema: StructType): StructType = {
+ val groupByColNums = aggregation.groupByColumns.length
Review comment:
changed to `numOfGroupByColumns`
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala
##########
@@ -457,17 +457,22 @@ object OrcUtils extends Logging {
}
}
+ // if there are group by columns, we will build result row first,
+ // and then append group by columns values (partition columns values) to the result row.
+ val schemaWithoutGroupby =
Review comment:
changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-971038438
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49764/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34445:
URL: https://github.com/apache/spark/pull/34445#discussion_r750838044
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileSourceAggregatePushDownSuite.scala
##########
@@ -261,6 +261,65 @@ trait FileSourceAggregatePushDownSuite
}
}
+ test("aggregate with partition group by can be pushed down") {
+ withTempPath { dir =>
+ spark.range(10).selectExpr("id", "id % 3 as p")
+ .write.partitionBy("p").format(format).save(dir.getCanonicalPath)
+ withTempView("tmp") {
+ spark.read.format(format).load(dir.getCanonicalPath).createOrReplaceTempView("tmp");
+ Seq("false", "true").foreach { enableVectorizedReader =>
+ withSQLConf(aggPushDownEnabledKey -> "true",
Review comment:
Hmm, can you test both `aggPushDownEnabledKey` as true and false and see if the results are the same?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on a change in pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #34445:
URL: https://github.com/apache/spark/pull/34445#discussion_r740549841
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala
##########
@@ -457,17 +457,22 @@ object OrcUtils extends Logging {
}
}
+ // if there are group by columns, we will build result set row first,
+ // and then append group by columns values (partition col values) to the result set row.
Review comment:
nit: `partition col values` -> `partition columns values`
nit: `result set row ` -> `result row`? What's `result set row`?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala
##########
@@ -490,7 +495,12 @@ object OrcUtils extends Logging {
s"createAggInternalRowFromFooter should not take $x as the aggregate expression")
}
- val orcValuesDeserializer = new OrcDeserializer(aggSchema, (0 until aggSchema.length).toArray)
- orcValuesDeserializer.deserializeFromValues(aggORCValues)
+ val orcValuesDeserializer = new OrcDeserializer(schemaWithoutGroupby,
+ (0 until schemaWithoutGroupby.length).toArray)
+ if (aggregation.groupByColumns.length > 0) {
+ new JoinedRow(partitionValues, orcValuesDeserializer.deserializeFromValues(aggORCValues))
Review comment:
nit: `orcValuesDeserializer.deserializeFromValues(aggORCValues)` can be called only once before the if-else branch.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileSourceAggregatePushDownSuite.scala
##########
@@ -261,6 +261,63 @@ trait FileSourceAggregatePushDownSuite
}
}
+ test("aggregate with partition group by can be pushed down") {
+ withTempPath { dir =>
+ spark.range(10).selectExpr("id", "id % 3 as p")
+ .write.partitionBy("p").format(format).save(dir.getCanonicalPath)
+ withTempView("tmp") {
+ spark.read.format(format).load(dir.getCanonicalPath).createOrReplaceTempView("tmp");
+ Seq("false", "true").foreach { enableVectorizedReader =>
+ withSQLConf(aggPushDownEnabledKey -> "true",
+ vectorizedReaderEnabledKey -> enableVectorizedReader) {
+ val df = sql("SELECT count(*), count(id), p, max(id), p, p, max(id), min(id), p" +
Review comment:
shall we also add test for aggregate on group-by column? e.g. `count(p)`, `max(p)` etc.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/AggregatePushDownUtils.scala
##########
@@ -138,4 +141,14 @@ object AggregatePushDownUtils {
converter.convert(aggregatesAsRow, columnVectors.toArray)
new ColumnarBatch(columnVectors.asInstanceOf[Array[ColumnVector]], 1)
}
+
+ def aggSchemaWithOutGroupBy(aggregation: Aggregation, aggSchema: StructType): StructType = {
Review comment:
Shall we add some comments for the public method?
Also nit: shall we call it `getSchemaWithoutGroupingExpression`? To be aligned with all aggregate operators, e.g. `BaseAggregateExec.groupingExpressions`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-958044555
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144855/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #34445:
URL: https://github.com/apache/spark/pull/34445#discussion_r741315128
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileSourceAggregatePushDownSuite.scala
##########
@@ -261,6 +261,63 @@ trait FileSourceAggregatePushDownSuite
}
}
+ test("aggregate with partition group by can be pushed down") {
+ withTempPath { dir =>
+ spark.range(10).selectExpr("id", "id % 3 as p")
+ .write.partitionBy("p").format(format).save(dir.getCanonicalPath)
+ withTempView("tmp") {
+ spark.read.format(format).load(dir.getCanonicalPath).createOrReplaceTempView("tmp");
+ Seq("false", "true").foreach { enableVectorizedReader =>
+ withSQLConf(aggPushDownEnabledKey -> "true",
+ vectorizedReaderEnabledKey -> enableVectorizedReader) {
+ val df = sql("SELECT count(*), count(id), p, max(id), p, p, max(id), min(id), p" +
Review comment:
I added `count(p)`, but didn't add `max(p)` since it won't be pushed down
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #34445:
URL: https://github.com/apache/spark/pull/34445#discussion_r744174581
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/AggregatePushDownUtils.scala
##########
@@ -138,4 +141,19 @@ object AggregatePushDownUtils {
converter.convert(aggregatesAsRow, columnVectors.toArray)
new ColumnarBatch(columnVectors.asInstanceOf[Array[ColumnVector]], 1)
}
+
+ /**
+ * Return the schema for aggregates only (exclude group by columns)
+ */
+ def getSchemaWithoutGroupingExpression(
Review comment:
Changed to `getSchemaWithoutGroupingExpression` per this suggestion https://github.com/apache/spark/pull/34445#discussion_r740418455
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-962531300
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49428/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-972427699
Last commit is test-only and GA was passed. Merging to master. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-972397585
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49824/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-958104543
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49325/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-958044555
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144855/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-955138077
**[Test build #144773 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144773/testReport)** for PR 34445 at commit [`9ea1c2d`](https://github.com/apache/spark/commit/9ea1c2d5b34abe9994f9e0ceedeea837436c8788).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-962530060
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49428/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-968253062
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49673/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-972283574
**[Test build #145353 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145353/testReport)** for PR 34445 at commit [`1f45a04`](https://github.com/apache/spark/commit/1f45a04b1bbaa1e0134ccb9d0446023bd95793d1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-958093161
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49325/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-957975899
**[Test build #144855 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144855/testReport)** for PR 34445 at commit [`03c2bd4`](https://github.com/apache/spark/commit/03c2bd470309bf7b6efe4cf3cbddf64aa0ed2fa9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-968243787
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49673/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-971038438
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49764/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-970838351
**[Test build #145294 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145294/testReport)** for PR 34445 at commit [`4eeae6d`](https://github.com/apache/spark/commit/4eeae6d3a2d3ea7c22c579e09a83a88e13f91a70).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-971172253
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145294/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #34445:
URL: https://github.com/apache/spark/pull/34445#discussion_r744174405
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetUtils.scala
##########
@@ -195,7 +201,12 @@ object ParquetUtils {
case (_, i) =>
throw new SparkException("Unexpected parquet type name: " + primitiveTypeNames(i))
}
- converter.currentRecord
+
+ if (aggregation.groupByColumns.length > 0) {
+ new JoinedRow(partitionValues, converter.currentRecord)
Review comment:
I actually forgot to considering ordering. Fixed. Thanks for pointing this out!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #34445:
URL: https://github.com/apache/spark/pull/34445#discussion_r748808003
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/AggregatePushDownUtils.scala
##########
@@ -138,4 +145,39 @@ object AggregatePushDownUtils {
converter.convert(aggregatesAsRow, columnVectors.toArray)
new ColumnarBatch(columnVectors.asInstanceOf[Array[ColumnVector]], 1)
}
+
+ /**
+ * Return the schema for aggregates only (exclude group by columns)
+ */
+ def getSchemaWithoutGroupingExpression(
+ aggregation: Aggregation,
+ aggSchema: StructType): StructType = {
+ val numOfGroupByColumns = aggregation.groupByColumns.length
+ if (numOfGroupByColumns > 0) {
+ new StructType(aggSchema.fields.drop(numOfGroupByColumns))
+ } else {
+ aggSchema
+ }
+ }
+
+ /**
+ * Reorder partition cols if they are not in the same order as group by columns
+ */
+ def reOrderPartitionCol(
+ partitionSchema: StructType,
+ aggregation: Aggregation,
+ partitionValues: InternalRow): InternalRow = {
+ val groupByColNames = aggregation.groupByColumns.map(_.fieldNames.head)
+ var reorderedPartColValues = Array.empty[Any]
+ if (!partitionSchema.names.sameElements(groupByColNames)) {
+ groupByColNames.foreach { col =>
+ val index = partitionSchema.names.indexOf(col)
+ val v = partitionValues.asInstanceOf[GenericInternalRow].values(index)
Review comment:
Seems to me that the `partitionValues` comes from `PartitionPath`, which always contains `GenericInternalRow`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-968233070
**[Test build #145204 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145204/testReport)** for PR 34445 at commit [`4fb313b`](https://github.com/apache/spark/commit/4fb313bd4f69d384fdacd1e2457b7be7772527ce).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-968275868
**[Test build #145204 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145204/testReport)** for PR 34445 at commit [`4fb313b`](https://github.com/apache/spark/commit/4fb313bd4f69d384fdacd1e2457b7be7772527ce).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-971814873
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49815/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-971804686
**[Test build #145344 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145344/testReport)** for PR 34445 at commit [`b561d09`](https://github.com/apache/spark/commit/b561d09b48b4cd4ad9996d2c0fb0f16378f77ee5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-972166078
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145344/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-972526286
**[Test build #145353 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145353/testReport)** for PR 34445 at commit [`1f45a04`](https://github.com/apache/spark/commit/1f45a04b1bbaa1e0134ccb9d0446023bd95793d1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-955171464
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144773/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-958043159
**[Test build #144855 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144855/testReport)** for PR 34445 at commit [`03c2bd4`](https://github.com/apache/spark/commit/03c2bd470309bf7b6efe4cf3cbddf64aa0ed2fa9).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
huaxingao commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-957969098
@c21 Thanks a lot for review! I have addressed the comments. Please take one more look when you have time. Thanks!
also cc @sunchao @viirya
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34445:
URL: https://github.com/apache/spark/pull/34445#discussion_r750840506
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetUtils.scala
##########
@@ -250,8 +261,7 @@ object ParquetUtils {
schemaName = "count(" + count.column.fieldNames.head + ")"
rowCount += block.getRowCount
var isPartitionCol = false
- if (partitionSchema.fields.map(PartitioningUtils.getColName(_, isCaseSensitive))
- .toSet.contains(count.column.fieldNames.head)) {
+ if (partitionSchema.fields.map(_.name).toSet.contains(count.column.fieldNames.head)) {
Review comment:
Don't need check case sensitivity now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-968280144
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145204/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-968280144
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145204/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-970838351
**[Test build #145294 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145294/testReport)** for PR 34445 at commit [`4eeae6d`](https://github.com/apache/spark/commit/4eeae6d3a2d3ea7c22c579e09a83a88e13f91a70).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34445:
URL: https://github.com/apache/spark/pull/34445#discussion_r750839789
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/orc/OrcPartitionReaderFactory.scala
##########
@@ -86,7 +86,7 @@ case class OrcPartitionReaderFactory(
val filePath = new Path(new URI(file.filePath))
if (aggregation.nonEmpty) {
- return buildReaderWithAggregates(filePath, conf)
+ return buildReaderWithAggregates(file, conf)
}
Review comment:
Seems `filePath` can be created after the `if` block:
```scala
if (aggregation.nonEmpty) {
return buildReaderWithAggregates(file, conf)
}
val filePath = new Path(new URI(file.filePath))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-968254075
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49673/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-955171464
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144773/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-955150621
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49242/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-972418502
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49824/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
huaxingao commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-972435473
Thank you all
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34445:
URL: https://github.com/apache/spark/pull/34445#discussion_r751498421
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileSourceAggregatePushDownSuite.scala
##########
@@ -261,6 +261,67 @@ trait FileSourceAggregatePushDownSuite
}
}
+ test("aggregate with partition group by can be pushed down") {
+ withTempPath { dir =>
+ spark.range(10).selectExpr("id", "id % 3 as P")
+ .write.partitionBy("p").format(format).save(dir.getCanonicalPath)
+ withTempView("tmp") {
+ spark.read.format(format).load(dir.getCanonicalPath).createOrReplaceTempView("tmp");
+ val query = "SELECT count(*), count(id), p, max(id), p, count(p), max(id)," +
+ " min(id), p FROM tmp group by p"
+ val expected = sql(query).collect
+ Seq("false", "true").foreach { enableVectorizedReader =>
+ withSQLConf(aggPushDownEnabledKey -> "true",
+ vectorizedReaderEnabledKey -> enableVectorizedReader) {
+ val df = sql(query)
+ df.queryExecution.optimizedPlan.collect {
+ case _: DataSourceV2ScanRelation =>
+ val expected_plan_fragment =
+ "PushedAggregation: [COUNT(*), COUNT(id), MAX(id), COUNT(p), MIN(id)], " +
+ "PushedFilters: [], PushedGroupBy: [p]"
+ checkKeywordsExistsInExplain(df, expected_plan_fragment)
+ }
+ checkAnswer(df, expected)
+ }
+ }
+ }
+ }
+ }
+
+ test("aggregate with multi partition group by columns can be pushed down") {
+ withTempPath { dir =>
+ Seq((10, 1, 2, 5, 6), (2, 1, 2, 5, 6), (3, 2, 1, 4, 8), (4, 2, 1, 4, 9),
+ (5, 2, 1, 5, 8), (6, 2, 1, 4, 8), (1, 1, 2, 5, 6), (4, 1, 2, 5, 6),
+ (3, 2, 2, 9, 10), (-4, 2, 2, 9, 10), (6, 2, 2, 9, 10))
+ .toDF("value", "p1", "p2", "p3", "p4")
+ .write
+ .partitionBy("p2", "p1", "p4", "p3")
+ .format(format)
+ .save(dir.getCanonicalPath)
+
+ withTempView("tmp") {
+ spark.read.format(format).load(dir.getCanonicalPath).createOrReplaceTempView("tmp")
+ val query = "SELECT count(*), count(value), max(value), min(value)," +
+ " p4, p2, p3, p1 FROM tmp GROUP BY p1, p2, p3, p4"
+ val expected = sql(query).collect
Review comment:
Hmm, if we enable aggregate push down later one day, this test might be ignorantly missed to its original goal. Should we explicitly set `aggPushDownEnabledKey` as false and collect the expected results?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on a change in pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #34445:
URL: https://github.com/apache/spark/pull/34445#discussion_r740549841
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala
##########
@@ -457,17 +457,22 @@ object OrcUtils extends Logging {
}
}
+ // if there are group by columns, we will build result set row first,
+ // and then append group by columns values (partition col values) to the result set row.
Review comment:
nit: `partition col values` -> `partition columns values`
nit: `result set row ` -> `result row`? What's `result set row`?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala
##########
@@ -490,7 +495,12 @@ object OrcUtils extends Logging {
s"createAggInternalRowFromFooter should not take $x as the aggregate expression")
}
- val orcValuesDeserializer = new OrcDeserializer(aggSchema, (0 until aggSchema.length).toArray)
- orcValuesDeserializer.deserializeFromValues(aggORCValues)
+ val orcValuesDeserializer = new OrcDeserializer(schemaWithoutGroupby,
+ (0 until schemaWithoutGroupby.length).toArray)
+ if (aggregation.groupByColumns.length > 0) {
+ new JoinedRow(partitionValues, orcValuesDeserializer.deserializeFromValues(aggORCValues))
Review comment:
nit: `orcValuesDeserializer.deserializeFromValues(aggORCValues)` can be called only once before the if-else branch.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileSourceAggregatePushDownSuite.scala
##########
@@ -261,6 +261,63 @@ trait FileSourceAggregatePushDownSuite
}
}
+ test("aggregate with partition group by can be pushed down") {
+ withTempPath { dir =>
+ spark.range(10).selectExpr("id", "id % 3 as p")
+ .write.partitionBy("p").format(format).save(dir.getCanonicalPath)
+ withTempView("tmp") {
+ spark.read.format(format).load(dir.getCanonicalPath).createOrReplaceTempView("tmp");
+ Seq("false", "true").foreach { enableVectorizedReader =>
+ withSQLConf(aggPushDownEnabledKey -> "true",
+ vectorizedReaderEnabledKey -> enableVectorizedReader) {
+ val df = sql("SELECT count(*), count(id), p, max(id), p, p, max(id), min(id), p" +
Review comment:
shall we also add test for aggregate on group-by column? e.g. `count(p)`, `max(p)` etc.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/AggregatePushDownUtils.scala
##########
@@ -138,4 +141,14 @@ object AggregatePushDownUtils {
converter.convert(aggregatesAsRow, columnVectors.toArray)
new ColumnarBatch(columnVectors.asInstanceOf[Array[ColumnVector]], 1)
}
+
+ def aggSchemaWithOutGroupBy(aggregation: Aggregation, aggSchema: StructType): StructType = {
Review comment:
Shall we add some comments for the public method?
Also nit: shall we call it `getSchemaWithoutGroupingExpression`? To be aligned with all aggregate operators, e.g. `BaseAggregateExec.groupingExpressions`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #34445:
URL: https://github.com/apache/spark/pull/34445#discussion_r741315128
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileSourceAggregatePushDownSuite.scala
##########
@@ -261,6 +261,63 @@ trait FileSourceAggregatePushDownSuite
}
}
+ test("aggregate with partition group by can be pushed down") {
+ withTempPath { dir =>
+ spark.range(10).selectExpr("id", "id % 3 as p")
+ .write.partitionBy("p").format(format).save(dir.getCanonicalPath)
+ withTempView("tmp") {
+ spark.read.format(format).load(dir.getCanonicalPath).createOrReplaceTempView("tmp");
+ Seq("false", "true").foreach { enableVectorizedReader =>
+ withSQLConf(aggPushDownEnabledKey -> "true",
+ vectorizedReaderEnabledKey -> enableVectorizedReader) {
+ val df = sql("SELECT count(*), count(id), p, max(id), p, p, max(id), min(id), p" +
Review comment:
I added `count(p)`, but didn't add `max(p)` since it won't be pushed down
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-955170745
**[Test build #144773 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144773/testReport)** for PR 34445 at commit [`9ea1c2d`](https://github.com/apache/spark/commit/9ea1c2d5b34abe9994f9e0ceedeea837436c8788).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #34445:
URL: https://github.com/apache/spark/pull/34445#discussion_r751465710
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetUtils.scala
##########
@@ -250,8 +261,7 @@ object ParquetUtils {
schemaName = "count(" + count.column.fieldNames.head + ")"
rowCount += block.getRowCount
var isPartitionCol = false
- if (partitionSchema.fields.map(PartitioningUtils.getColName(_, isCaseSensitive))
- .toSet.contains(count.column.fieldNames.head)) {
+ if (partitionSchema.fields.map(_.name).toSet.contains(count.column.fieldNames.head)) {
Review comment:
seems to me no need to check case sensitivity because I have normalized aggregates and group by columns in `V2ScanRelationPushDown`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-957975899
**[Test build #144855 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144855/testReport)** for PR 34445 at commit [`03c2bd4`](https://github.com/apache/spark/commit/03c2bd470309bf7b6efe4cf3cbddf64aa0ed2fa9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #34445:
URL: https://github.com/apache/spark/pull/34445#discussion_r744174402
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileSourceAggregatePushDownSuite.scala
##########
@@ -261,6 +261,63 @@ trait FileSourceAggregatePushDownSuite
}
}
+ test("aggregate with partition group by can be pushed down") {
Review comment:
I want it to be consistent with the rest of the tests, so didn't add a jira number
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #34445:
URL: https://github.com/apache/spark/pull/34445#discussion_r744174351
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetUtils.scala
##########
@@ -195,7 +201,12 @@ object ParquetUtils {
case (_, i) =>
throw new SparkException("Unexpected parquet type name: " + primitiveTypeNames(i))
}
- converter.currentRecord
+
+ if (aggregation.groupByColumns.length > 0) {
Review comment:
changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-962547559
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144957/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34445:
URL: https://github.com/apache/spark/pull/34445#discussion_r750839929
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/orc/OrcPartitionReaderFactory.scala
##########
@@ -130,7 +130,7 @@ case class OrcPartitionReaderFactory(
val filePath = new Path(new URI(file.filePath))
if (aggregation.nonEmpty) {
- return buildColumnarReaderWithAggregates(filePath, conf)
+ return buildColumnarReaderWithAggregates(file, conf)
}
Review comment:
ditto.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-971172253
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145294/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-968233070
**[Test build #145204 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145204/testReport)** for PR 34445 at commit [`4fb313b`](https://github.com/apache/spark/commit/4fb313bd4f69d384fdacd1e2457b7be7772527ce).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-970907608
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49764/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-971804686
**[Test build #145344 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145344/testReport)** for PR 34445 at commit [`b561d09`](https://github.com/apache/spark/commit/b561d09b48b4cd4ad9996d2c0fb0f16378f77ee5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-962531300
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49428/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-962547559
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144957/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-955142480
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49242/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34445:
URL: https://github.com/apache/spark/pull/34445#discussion_r748584416
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/AggregatePushDownUtils.scala
##########
@@ -138,4 +145,39 @@ object AggregatePushDownUtils {
converter.convert(aggregatesAsRow, columnVectors.toArray)
new ColumnarBatch(columnVectors.asInstanceOf[Array[ColumnVector]], 1)
}
+
+ /**
+ * Return the schema for aggregates only (exclude group by columns)
+ */
+ def getSchemaWithoutGroupingExpression(
+ aggregation: Aggregation,
+ aggSchema: StructType): StructType = {
+ val numOfGroupByColumns = aggregation.groupByColumns.length
+ if (numOfGroupByColumns > 0) {
+ new StructType(aggSchema.fields.drop(numOfGroupByColumns))
+ } else {
+ aggSchema
+ }
+ }
+
+ /**
+ * Reorder partition cols if they are not in the same order as group by columns
+ */
+ def reOrderPartitionCol(
+ partitionSchema: StructType,
+ aggregation: Aggregation,
+ partitionValues: InternalRow): InternalRow = {
+ val groupByColNames = aggregation.groupByColumns.map(_.fieldNames.head)
+ var reorderedPartColValues = Array.empty[Any]
+ if (!partitionSchema.names.sameElements(groupByColNames)) {
+ groupByColNames.foreach { col =>
+ val index = partitionSchema.names.indexOf(col)
+ val v = partitionValues.asInstanceOf[GenericInternalRow].values(index)
Review comment:
just curious: is this always guaranteed to be `GenericInternalRow`?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/AggregatePushDownUtils.scala
##########
@@ -138,4 +145,39 @@ object AggregatePushDownUtils {
converter.convert(aggregatesAsRow, columnVectors.toArray)
new ColumnarBatch(columnVectors.asInstanceOf[Array[ColumnVector]], 1)
}
+
+ /**
+ * Return the schema for aggregates only (exclude group by columns)
+ */
+ def getSchemaWithoutGroupingExpression(
+ aggregation: Aggregation,
+ aggSchema: StructType): StructType = {
+ val numOfGroupByColumns = aggregation.groupByColumns.length
+ if (numOfGroupByColumns > 0) {
+ new StructType(aggSchema.fields.drop(numOfGroupByColumns))
+ } else {
+ aggSchema
+ }
+ }
+
+ /**
+ * Reorder partition cols if they are not in the same order as group by columns
+ */
+ def reOrderPartitionCol(
+ partitionSchema: StructType,
+ aggregation: Aggregation,
+ partitionValues: InternalRow): InternalRow = {
+ val groupByColNames = aggregation.groupByColumns.map(_.fieldNames.head)
Review comment:
should we add an assertion here checking that `aggregation.groupByColumns` has the same length as `partitionSchema` and `partitionValues`?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetPartitionReaderFactory.scala
##########
@@ -134,10 +134,14 @@ case class ParquetPartitionReaderFactory(
private var hasNext = true
private lazy val row: InternalRow = {
val footer = getFooter(file)
+
+ val partitionValues = AggregatePushDownUtils.reOrderPartitionCol(
Review comment:
can we call this in `ParquetUtils.createAggInternalRowFromFooter`? it seems we always call `AggregatePushDownUtils.reOrderPartitionCol` then followed by `ParquetUtils.createAggInternalRowFromFooter`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
huaxingao commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-957969098
@c21 Thanks a lot for review! I have addressed the comments. Please take one more look when you have time. Thanks!
also cc @sunchao @viirya
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-957975899
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on a change in pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #34445:
URL: https://github.com/apache/spark/pull/34445#discussion_r740549841
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala
##########
@@ -457,17 +457,22 @@ object OrcUtils extends Logging {
}
}
+ // if there are group by columns, we will build result set row first,
+ // and then append group by columns values (partition col values) to the result set row.
Review comment:
nit: `partition col values` -> `partition columns values`
nit: `result set row ` -> `result row`? What's `result set row`?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala
##########
@@ -490,7 +495,12 @@ object OrcUtils extends Logging {
s"createAggInternalRowFromFooter should not take $x as the aggregate expression")
}
- val orcValuesDeserializer = new OrcDeserializer(aggSchema, (0 until aggSchema.length).toArray)
- orcValuesDeserializer.deserializeFromValues(aggORCValues)
+ val orcValuesDeserializer = new OrcDeserializer(schemaWithoutGroupby,
+ (0 until schemaWithoutGroupby.length).toArray)
+ if (aggregation.groupByColumns.length > 0) {
+ new JoinedRow(partitionValues, orcValuesDeserializer.deserializeFromValues(aggORCValues))
Review comment:
nit: `orcValuesDeserializer.deserializeFromValues(aggORCValues)` can be called only once before the if-else branch.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileSourceAggregatePushDownSuite.scala
##########
@@ -261,6 +261,63 @@ trait FileSourceAggregatePushDownSuite
}
}
+ test("aggregate with partition group by can be pushed down") {
+ withTempPath { dir =>
+ spark.range(10).selectExpr("id", "id % 3 as p")
+ .write.partitionBy("p").format(format).save(dir.getCanonicalPath)
+ withTempView("tmp") {
+ spark.read.format(format).load(dir.getCanonicalPath).createOrReplaceTempView("tmp");
+ Seq("false", "true").foreach { enableVectorizedReader =>
+ withSQLConf(aggPushDownEnabledKey -> "true",
+ vectorizedReaderEnabledKey -> enableVectorizedReader) {
+ val df = sql("SELECT count(*), count(id), p, max(id), p, p, max(id), min(id), p" +
Review comment:
shall we also add test for aggregate on group-by column? e.g. `count(p)`, `max(p)` etc.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/AggregatePushDownUtils.scala
##########
@@ -138,4 +141,14 @@ object AggregatePushDownUtils {
converter.convert(aggregatesAsRow, columnVectors.toArray)
new ColumnarBatch(columnVectors.asInstanceOf[Array[ColumnVector]], 1)
}
+
+ def aggSchemaWithOutGroupBy(aggregation: Aggregation, aggSchema: StructType): StructType = {
Review comment:
Shall we add some comments for the public method?
Also nit: shall we call it `getSchemaWithoutGroupingExpression`? To be aligned with all aggregate operators, e.g. `BaseAggregateExec.groupingExpressions`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-958044555
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-968254075
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49673/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-972530335
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145353/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-971161629
**[Test build #145294 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145294/testReport)** for PR 34445 at commit [`4eeae6d`](https://github.com/apache/spark/commit/4eeae6d3a2d3ea7c22c579e09a83a88e13f91a70).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34445:
URL: https://github.com/apache/spark/pull/34445#discussion_r750528925
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/AggregatePushDownUtils.scala
##########
@@ -81,19 +81,26 @@ object AggregatePushDownUtils {
}
}
- if (aggregation.groupByColumns.nonEmpty || dataFilters.nonEmpty) {
+ if (dataFilters.nonEmpty) {
// Parquet/ORC footer has max/min/count for columns
// e.g. SELECT COUNT(col1) FROM t
// but footer doesn't have max/min/count for a column if max/min/count
// are combined with filter or group by
// e.g. SELECT COUNT(col1) FROM t WHERE col2 = 8
// SELECT COUNT(col1) FROM t GROUP BY col2
// However, if the filter is on partition column, max/min/count can still be pushed down
- // Todo: add support if groupby column is partition col
- // (https://issues.apache.org/jira/browse/SPARK-36646)
return None
}
+ if (aggregation.groupByColumns.nonEmpty &&
Review comment:
nit: maybe add some comments explaining the reasoning why we have this check and only support the case when group by columns is the same as partition columns. What if the number of group by columns is smaller than that of partition columns?
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileSourceAggregatePushDownSuite.scala
##########
@@ -261,6 +261,65 @@ trait FileSourceAggregatePushDownSuite
}
}
+ test("aggregate with partition group by can be pushed down") {
+ withTempPath { dir =>
+ spark.range(10).selectExpr("id", "id % 3 as p")
+ .write.partitionBy("p").format(format).save(dir.getCanonicalPath)
+ withTempView("tmp") {
+ spark.read.format(format).load(dir.getCanonicalPath).createOrReplaceTempView("tmp");
+ Seq("false", "true").foreach { enableVectorizedReader =>
+ withSQLConf(aggPushDownEnabledKey -> "true",
+ vectorizedReaderEnabledKey -> enableVectorizedReader) {
+ val df = sql("SELECT count(*), count(id), p, max(id), p, count(p), max(id)," +
+ " min(id), p FROM tmp group by p")
+ df.queryExecution.optimizedPlan.collect {
+ case _: DataSourceV2ScanRelation =>
+ val expected_plan_fragment =
+ "PushedAggregation: [COUNT(*), COUNT(id), MAX(id), COUNT(p), MIN(id)], " +
+ "PushedFilters: [], PushedGroupBy: [p]"
+ checkKeywordsExistsInExplain(df, expected_plan_fragment)
+ }
+ checkAnswer(df, Seq(Row(3, 3, 1, 7, 1, 3, 7, 1, 1), Row(3, 3, 2, 8, 2, 3, 8, 2, 2),
+ Row(4, 4, 0, 9, 0, 4, 9, 0, 0)))
+ }
+ }
+ }
+ }
+ }
+
+ test("aggregate with multi partition group by columns can be pushed down") {
+ withTempPath { dir =>
+ Seq((10, 1, 2, 5, 6), (2, 1, 2, 5, 6), (3, 2, 1, 4, 8), (4, 2, 1, 4, 9),
+ (5, 2, 1, 5, 8), (6, 2, 1, 4, 8), (1, 1, 2, 5, 6), (4, 1, 2, 5, 6),
+ (3, 2, 2, 9, 10), (-4, 2, 2, 9, 10), (6, 2, 2, 9, 10))
+ .toDF("value", "p1", "p2", "p3", "p4")
+ .write
+ .partitionBy("p2", "p1", "p4", "p3")
+ .format(format)
+ .save(dir.getCanonicalPath)
+ withTempView("tmp") {
+ spark.read.format(format).load(dir.getCanonicalPath).createOrReplaceTempView("tmp");
+ Seq("false", "true").foreach { enableVectorizedReader =>
+ withSQLConf(aggPushDownEnabledKey -> "true",
+ vectorizedReaderEnabledKey -> enableVectorizedReader) {
+ val df = sql("SELECT count(*), count(value), max(value), min(value)," +
+ " p4, p2, p3, p1 FROM tmp GROUP BY p1, p2, p3, p4")
+ df.queryExecution.optimizedPlan.collect {
+ case _: DataSourceV2ScanRelation =>
+ val expected_plan_fragment =
+ "PushedAggregation: [COUNT(*), COUNT(value), MAX(value), MIN(value)]," +
+ " PushedFilters: [], PushedGroupBy: [p1, p2, p3, p4]"
+ // checkKeywordsExistsInExplain(df, expected_plan_fragment)
Review comment:
nit: remove this?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/AggregatePushDownUtils.scala
##########
@@ -81,19 +81,26 @@ object AggregatePushDownUtils {
}
}
- if (aggregation.groupByColumns.nonEmpty || dataFilters.nonEmpty) {
+ if (dataFilters.nonEmpty) {
// Parquet/ORC footer has max/min/count for columns
// e.g. SELECT COUNT(col1) FROM t
// but footer doesn't have max/min/count for a column if max/min/count
// are combined with filter or group by
// e.g. SELECT COUNT(col1) FROM t WHERE col2 = 8
// SELECT COUNT(col1) FROM t GROUP BY col2
// However, if the filter is on partition column, max/min/count can still be pushed down
- // Todo: add support if groupby column is partition col
- // (https://issues.apache.org/jira/browse/SPARK-36646)
return None
}
+ if (aggregation.groupByColumns.nonEmpty &&
+ partitionNames.size != aggregation.groupByColumns.length) {
+ return None
+ }
+ aggregation.groupByColumns.foreach { col =>
Review comment:
nit: maybe also add some comments here - it's not that easy to understand and can help the maintenance of this code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34445:
URL: https://github.com/apache/spark/pull/34445#discussion_r750532587
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/AggregatePushDownUtils.scala
##########
@@ -138,4 +145,44 @@ object AggregatePushDownUtils {
converter.convert(aggregatesAsRow, columnVectors.toArray)
new ColumnarBatch(columnVectors.asInstanceOf[Array[ColumnVector]], 1)
}
+
+ /**
+ * Return the schema for aggregates only (exclude group by columns)
+ */
+ def getSchemaWithoutGroupingExpression(
+ aggregation: Aggregation,
+ aggSchema: StructType): StructType = {
Review comment:
nit: maybe swap the order of `aggSchema` and `aggregation` here, as we're modifying the schema here with the info from aggregation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-962524148
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49428/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-962547294
**[Test build #144957 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144957/testReport)** for PR 34445 at commit [`0e655a8`](https://github.com/apache/spark/commit/0e655a8dedf278d52a661abf836d2e061412c03e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-962520215
**[Test build #144957 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144957/testReport)** for PR 34445 at commit [`0e655a8`](https://github.com/apache/spark/commit/0e655a8dedf278d52a661abf836d2e061412c03e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
huaxingao commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-957969098
@c21 Thanks a lot for review! I have addressed the comments. Please take one more look when you have time. Thanks!
also cc @sunchao @viirya
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34445:
URL: https://github.com/apache/spark/pull/34445#discussion_r741361243
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala
##########
@@ -457,17 +457,22 @@ object OrcUtils extends Logging {
}
}
+ // if there are group by columns, we will build result row first,
+ // and then append group by columns values (partition columns values) to the result row.
+ val schemaWithoutGroupby =
Review comment:
nit: `schemaWithoutGroupBy`
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/AggregatePushDownUtils.scala
##########
@@ -138,4 +141,19 @@ object AggregatePushDownUtils {
converter.convert(aggregatesAsRow, columnVectors.toArray)
new ColumnarBatch(columnVectors.asInstanceOf[Array[ColumnVector]], 1)
}
+
+ /**
+ * Return the schema for aggregates only (exclude group by columns)
+ */
+ def getSchemaWithoutGroupingExpression(
+ aggregation: Aggregation,
+ aggSchema: StructType): StructType = {
+ val groupByColNums = aggregation.groupByColumns.length
Review comment:
nit: `groupByColumns`
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetUtils.scala
##########
@@ -195,7 +201,12 @@ object ParquetUtils {
case (_, i) =>
throw new SparkException("Unexpected parquet type name: " + primitiveTypeNames(i))
}
- converter.currentRecord
+
+ if (aggregation.groupByColumns.length > 0) {
+ new JoinedRow(partitionValues, converter.currentRecord)
Review comment:
just curious how we make sure the `partitionValues` is always consistent with `aggregation.groupByColumns`? in terms of ordering.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetUtils.scala
##########
@@ -195,7 +201,12 @@ object ParquetUtils {
case (_, i) =>
throw new SparkException("Unexpected parquet type name: " + primitiveTypeNames(i))
}
- converter.currentRecord
+
+ if (aggregation.groupByColumns.length > 0) {
Review comment:
nit: maybe use `aggregation.groupByColumns.nonEmpty`
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/AggregatePushDownUtils.scala
##########
@@ -138,4 +141,19 @@ object AggregatePushDownUtils {
converter.convert(aggregatesAsRow, columnVectors.toArray)
new ColumnarBatch(columnVectors.asInstanceOf[Array[ColumnVector]], 1)
}
+
+ /**
+ * Return the schema for aggregates only (exclude group by columns)
+ */
+ def getSchemaWithoutGroupingExpression(
Review comment:
nit: maybe `getSchemaWithoutGroupColumns`?
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileSourceAggregatePushDownSuite.scala
##########
@@ -261,6 +261,63 @@ trait FileSourceAggregatePushDownSuite
}
}
+ test("aggregate with partition group by can be pushed down") {
Review comment:
nit: add JIRA number
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-958044555
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #34445:
URL: https://github.com/apache/spark/pull/34445#discussion_r741315128
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileSourceAggregatePushDownSuite.scala
##########
@@ -261,6 +261,63 @@ trait FileSourceAggregatePushDownSuite
}
}
+ test("aggregate with partition group by can be pushed down") {
+ withTempPath { dir =>
+ spark.range(10).selectExpr("id", "id % 3 as p")
+ .write.partitionBy("p").format(format).save(dir.getCanonicalPath)
+ withTempView("tmp") {
+ spark.read.format(format).load(dir.getCanonicalPath).createOrReplaceTempView("tmp");
+ Seq("false", "true").foreach { enableVectorizedReader =>
+ withSQLConf(aggPushDownEnabledKey -> "true",
+ vectorizedReaderEnabledKey -> enableVectorizedReader) {
+ val df = sql("SELECT count(*), count(id), p, max(id), p, p, max(id), min(id), p" +
Review comment:
I added `count(p)`, but didn't add `max(p)` since it won't be pushed down
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-972283574
**[Test build #145353 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145353/testReport)** for PR 34445 at commit [`1f45a04`](https://github.com/apache/spark/commit/1f45a04b1bbaa1e0134ccb9d0446023bd95793d1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-972530335
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145353/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-971814873
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49815/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-972363870
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49824/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-972418502
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49824/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya closed pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
viirya closed pull request #34445:
URL: https://github.com/apache/spark/pull/34445
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34445: [SPARK-36646][SQL] Push down group by partition column for aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34445:
URL: https://github.com/apache/spark/pull/34445#issuecomment-955149965
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49242/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org