You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/12/03 11:26:41 UTC
[GitHub] [spark] beliefer opened a new pull request #34799: [SPARK-37527] Translate more standard aggregate functions for pushdown
beliefer opened a new pull request #34799:
URL: https://github.com/apache/spark/pull/34799
### What changes were proposed in this pull request?
Currently, Spark aggregate pushdown will translate some standard aggregate functions, so that compile these functions to adapt specify database.
After this job, users could override `JdbcDialect.compileAggregate` to implement some aggregate functions supported by some database.
### Why are the changes needed?
Make the implement of *Dialect could extends the aggregate functions by override `JdbcDialect.compileAggregate`.
### Does this PR introduce _any_ user-facing change?
Yes. Users could pushdown more aggregate functions.
### How was this patch tested?
Exists tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-987896282
**[Test build #145989 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145989/testReport)** for PR 34799 at commit [`b9c7d96`](https://github.com/apache/spark/commit/b9c7d9645c6e30752bc786fbf1519ce0fd44df0d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #34799:
URL: https://github.com/apache/spark/pull/34799#discussion_r764617946
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala
##########
@@ -208,6 +208,9 @@ abstract class JdbcDialect extends Serializable with Logging{
case max: Max =>
if (max.column.fieldNames.length != 1) return None
Some(s"MAX(${quoteIdentifier(max.column.fieldNames.head)})")
+ case avg: Average =>
Review comment:
In Spark, partial aggregate output for avg is a sequence of (sum, count). If we want to push down partial aggregate of avg to data source, should we also use sum and count?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-987852115
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50459/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34799: [SPARK-37527] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-985486139
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50375/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34799: [SPARK-37527] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-985568012
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50376/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-986742159
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50425/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-987852115
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50459/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-987943424
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50463/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-987968208
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50465/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34799: [SPARK-37527] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-985541015
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50376/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-986905374
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145951/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-986670737
**[Test build #145950 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145950/testReport)** for PR 34799 at commit [`e012e74`](https://github.com/apache/spark/commit/e012e74d6624deda07117e44ea6bee44010f62e1).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34799: [SPARK-37527] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-985455827
**[Test build #145900 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145900/testReport)** for PR 34799 at commit [`82469e4`](https://github.com/apache/spark/commit/82469e4c0050499133606f811d154f11cfd29b3d).
* This patch **fails Java style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `public final class AnyOrSome implements AggregateFunc `
* `public final class Corr implements AggregateFunc `
* `public final class CovarPop implements AggregateFunc `
* `public final class CovarSamp implements AggregateFunc `
* `public final class Every implements AggregateFunc `
* `public final class StddevPop implements AggregateFunc `
* `public final class StddevSamp implements AggregateFunc `
* `public final class VarPop implements AggregateFunc `
* `public final class VarSamp implements AggregateFunc `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34799: [SPARK-37527] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-985524582
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50375/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-987828290
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50459/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-987943458
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50463/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-987943458
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50463/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34799: [SPARK-37527] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-985702721
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145901/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34799:
URL: https://github.com/apache/spark/pull/34799#discussion_r762826918
##########
File path: sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/aggregate/CovarPop.java
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.expressions.aggregate;
+
+import org.apache.spark.annotation.Evolving;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+
+/**
+ * An aggregate function that returns the population covariance of a set of number pairs.
+ *
+ * @since 3.3.0
+ */
+@Evolving
+public final class CovarPop implements AggregateFunc {
+ private final NamedReference left;
+ private final NamedReference right;
Review comment:
or we can use `left` and `right` consistently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34799:
URL: https://github.com/apache/spark/pull/34799#discussion_r762826565
##########
File path: sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/aggregate/Every.java
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.expressions.aggregate;
+
+import org.apache.spark.annotation.Evolving;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+
+/**
+ * An aggregate function that returns true if all values of `column` are true.
+ *
+ * @since 3.3.0
+ */
+@Evolving
+public final class Every implements AggregateFunc {
Review comment:
ditto, please check if we can really see it in the physical plan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-987700869
**[Test build #145983 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145983/testReport)** for PR 34799 at commit [`d92eecf`](https://github.com/apache/spark/commit/d92eecfcebbeb59768f3832c8e06f86ec42fee5b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-987942485
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145983/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-987942485
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145983/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer closed pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
beliefer closed pull request #34799:
URL: https://github.com/apache/spark/pull/34799
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-987990625
**[Test build #145984 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145984/testReport)** for PR 34799 at commit [`651f40b`](https://github.com/apache/spark/commit/651f40be11669ce3c68946da00bffce4715f494d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-987700869
**[Test build #145983 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145983/testReport)** for PR 34799 at commit [`d92eecf`](https://github.com/apache/spark/commit/d92eecfcebbeb59768f3832c8e06f86ec42fee5b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-987898083
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50463/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #34799:
URL: https://github.com/apache/spark/pull/34799#discussion_r764626922
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala
##########
@@ -208,6 +208,9 @@ abstract class JdbcDialect extends Serializable with Logging{
case max: Max =>
if (max.column.fieldNames.length != 1) return None
Some(s"MAX(${quoteIdentifier(max.column.fieldNames.head)})")
+ case avg: Average =>
Review comment:
But for final aggregate, we uses `First` to replace `Average`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34799: [SPARK-37527] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-985701029
**[Test build #145901 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145901/testReport)** for PR 34799 at commit [`96d2c21`](https://github.com/apache/spark/commit/96d2c217145f3dfd4894cc5b4ab4ebb2a844ba71).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34799: [SPARK-37527] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-985702721
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145901/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-986699028
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50425/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-986667319
**[Test build #145951 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145951/testReport)** for PR 34799 at commit [`6907b3a`](https://github.com/apache/spark/commit/6907b3ac0f90506a77ea1c89f44acd4c3f44df17).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34799: [SPARK-37527] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-985448000
**[Test build #145900 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145900/testReport)** for PR 34799 at commit [`82469e4`](https://github.com/apache/spark/commit/82469e4c0050499133606f811d154f11cfd29b3d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34799:
URL: https://github.com/apache/spark/pull/34799#discussion_r762824074
##########
File path: sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/aggregate/CovarPop.java
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.expressions.aggregate;
+
+import org.apache.spark.annotation.Evolving;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+
+/**
+ * An aggregate function that returns the population covariance of a set of number pairs.
+ *
+ * @since 3.3.0
+ */
+@Evolving
+public final class CovarPop implements AggregateFunc {
+ private final NamedReference left;
+ private final NamedReference right;
Review comment:
why not `x` and `y`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34799:
URL: https://github.com/apache/spark/pull/34799#discussion_r762822914
##########
File path: sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/aggregate/AnyOrSome.java
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.expressions.aggregate;
+
+import org.apache.spark.annotation.Evolving;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+
+/**
+ * An aggregate function that returns true if at least one value of `column` is true.
+ *
+ * @since 3.3.0
+ */
+@Evolving
+public final class AnyOrSome implements AggregateFunc {
Review comment:
ANY/SOME will be replaced by MAX/MIN in spark, so we can't really hit in at runtime, and no need to add data source push down API for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34799: [SPARK-37527] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-985448000
**[Test build #145900 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145900/testReport)** for PR 34799 at commit [`82469e4`](https://github.com/apache/spark/commit/82469e4c0050499133606f811d154f11cfd29b3d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-987738846
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50458/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-987800698
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50458/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #34799:
URL: https://github.com/apache/spark/pull/34799#discussion_r764527964
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala
##########
@@ -208,6 +208,9 @@ abstract class JdbcDialect extends Serializable with Logging{
case max: Max =>
if (max.column.fieldNames.length != 1) return None
Some(s"MAX(${quoteIdentifier(max.column.fieldNames.head)})")
+ case avg: Average =>
Review comment:
You means pushdown `sum/count` to data source ? Why not use `avg` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-987995400
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-988182436
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145989/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #34799:
URL: https://github.com/apache/spark/pull/34799#discussion_r764626922
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala
##########
@@ -208,6 +208,9 @@ abstract class JdbcDialect extends Serializable with Logging{
case max: Max =>
if (max.column.fieldNames.length != 1) return None
Some(s"MAX(${quoteIdentifier(max.column.fieldNames.head)})")
+ case avg: Average =>
Review comment:
But this PR uses `First` to replace `Average`. So we no need to pay attention to `sum` and `count`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34799: [SPARK-37527] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-985517428
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50375/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34799: [SPARK-37527] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-985484314
**[Test build #145901 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145901/testReport)** for PR 34799 at commit [`96d2c21`](https://github.com/apache/spark/commit/96d2c217145f3dfd4894cc5b4ab4ebb2a844ba71).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34799:
URL: https://github.com/apache/spark/pull/34799#discussion_r762823780
##########
File path: sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/aggregate/Corr.java
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.expressions.aggregate;
+
+import org.apache.spark.annotation.Evolving;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+
+/**
+ * An aggregate function that returns Pearson coefficient of correlation between
+ * a set of number pairs.
+ *
+ * @since 3.3.0
+ */
+@Evolving
+public final class Corr implements AggregateFunc {
+ private final NamedReference x;
+ private final NamedReference y;
+
+ public Corr(NamedReference left, NamedReference right) {
+ this.x = left;
+ this.y = right;
+ }
+
+ public NamedReference getX() { return x; }
Review comment:
let's follow other aggregate functions and use scala style. Just name it `x`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-1015980639
https://github.com/apache/spark/pull/35101 merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-986664228
**[Test build #145950 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145950/testReport)** for PR 34799 at commit [`e012e74`](https://github.com/apache/spark/commit/e012e74d6624deda07117e44ea6bee44010f62e1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-987750255
**[Test build #145984 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145984/testReport)** for PR 34799 at commit [`651f40b`](https://github.com/apache/spark/commit/651f40be11669ce3c68946da00bffce4715f494d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-986905374
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145951/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-987750255
**[Test build #145984 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145984/testReport)** for PR 34799 at commit [`651f40b`](https://github.com/apache/spark/commit/651f40be11669ce3c68946da00bffce4715f494d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #34799:
URL: https://github.com/apache/spark/pull/34799#discussion_r762836995
##########
File path: sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/aggregate/Corr.java
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.expressions.aggregate;
+
+import org.apache.spark.annotation.Evolving;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+
+/**
+ * An aggregate function that returns Pearson coefficient of correlation between
+ * a set of number pairs.
+ *
+ * @since 3.3.0
+ */
+@Evolving
+public final class Corr implements AggregateFunc {
+ private final NamedReference x;
+ private final NamedReference y;
+
+ public Corr(NamedReference left, NamedReference right) {
+ this.x = left;
+ this.y = right;
+ }
+
+ public NamedReference getX() { return x; }
Review comment:
Thank you for the reminder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-986742159
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50425/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-988568065
ping @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-986365490
ping @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34799: [SPARK-37527] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-985484314
**[Test build #145901 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145901/testReport)** for PR 34799 at commit [`96d2c21`](https://github.com/apache/spark/commit/96d2c217145f3dfd4894cc5b4ab4ebb2a844ba71).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-987790172
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50459/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-988182436
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145989/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-987800698
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50458/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #34799:
URL: https://github.com/apache/spark/pull/34799#discussion_r764626922
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala
##########
@@ -208,6 +208,9 @@ abstract class JdbcDialect extends Serializable with Logging{
case max: Max =>
if (max.column.fieldNames.length != 1) return None
Some(s"MAX(${quoteIdentifier(max.column.fieldNames.head)})")
+ case avg: Average =>
Review comment:
But for final aggregate, we uses `First` to replace `Average`. We need discard the partial mode and just select final aggregate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-987935071
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50465/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #34799:
URL: https://github.com/apache/spark/pull/34799#discussion_r764178701
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala
##########
@@ -208,6 +208,9 @@ abstract class JdbcDialect extends Serializable with Logging{
case max: Max =>
if (max.column.fieldNames.length != 1) return None
Some(s"MAX(${quoteIdentifier(max.column.fieldNames.head)})")
+ case avg: Average =>
Review comment:
Is it right to push down `avg`? Should we use `sum` and `count` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-987777916
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50458/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34799: [SPARK-37527] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-985455889
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145900/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34799: [SPARK-37527] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-985510152
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50376/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #34799:
URL: https://github.com/apache/spark/pull/34799#discussion_r762899954
##########
File path: sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/aggregate/CovarPop.java
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.expressions.aggregate;
+
+import org.apache.spark.annotation.Evolving;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+
+/**
+ * An aggregate function that returns the population covariance of a set of number pairs.
+ *
+ * @since 3.3.0
+ */
+@Evolving
+public final class CovarPop implements AggregateFunc {
+ private final NamedReference left;
+ private final NamedReference right;
Review comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-986667319
**[Test build #145951 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145951/testReport)** for PR 34799 at commit [`6907b3a`](https://github.com/apache/spark/commit/6907b3ac0f90506a77ea1c89f44acd4c3f44df17).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-986699057
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145950/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-986699057
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145950/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-986664228
**[Test build #145950 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145950/testReport)** for PR 34799 at commit [`e012e74`](https://github.com/apache/spark/commit/e012e74d6624deda07117e44ea6bee44010f62e1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-986903693
**[Test build #145951 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145951/testReport)** for PR 34799 at commit [`6907b3a`](https://github.com/apache/spark/commit/6907b3ac0f90506a77ea1c89f44acd4c3f44df17).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-986727713
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50425/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34799: [SPARK-37527] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-985455889
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145900/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34799: [SPARK-37527] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-985524582
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50375/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34799: [SPARK-37527] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-985568012
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50376/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-987928708
**[Test build #145983 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145983/testReport)** for PR 34799 at commit [`d92eecf`](https://github.com/apache/spark/commit/d92eecfcebbeb59768f3832c8e06f86ec42fee5b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `public final class Average implements AggregateFunc `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-987995402
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-987896282
**[Test build #145989 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145989/testReport)** for PR 34799 at commit [`b9c7d96`](https://github.com/apache/spark/commit/b9c7d9645c6e30752bc786fbf1519ce0fd44df0d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34799:
URL: https://github.com/apache/spark/pull/34799#issuecomment-988157883
**[Test build #145989 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145989/testReport)** for PR 34799 at commit [`b9c7d96`](https://github.com/apache/spark/commit/b9c7d9645c6e30752bc786fbf1519ce0fd44df0d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #34799: [SPARK-37527][SQL] Translate more standard aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #34799:
URL: https://github.com/apache/spark/pull/34799#discussion_r764626922
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala
##########
@@ -208,6 +208,9 @@ abstract class JdbcDialect extends Serializable with Logging{
case max: Max =>
if (max.column.fieldNames.length != 1) return None
Some(s"MAX(${quoteIdentifier(max.column.fieldNames.head)})")
+ case avg: Average =>
Review comment:
But for final aggregate, it just need the value of avg(na matter it is partial aggregate or pushdown aggregate).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org