You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/10/21 08:06:37 UTC
[GitHub] [spark] beliefer opened a new pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
beliefer opened a new pull request #34352:
URL: https://github.com/apache/spark/pull/34352
### What changes were proposed in this pull request?
Spark SQL not supports to create function of `Aggregator` yet and deprecated `UserDefinedAggregateFunction`.
If we want remove `UserDefinedAggregateFunction`, Spark SQL should provide a new option.
### Why are the changes needed?
We need to provide a new way to create user defined aggregate function so as remove `UserDefinedAggregateFunction` in future.
### Does this PR introduce _any_ user-facing change?
Yes. Users will create user defined aggregate function by implement `Aggregator`.
### How was this patch tested?
New tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-972440972
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49828/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-972648563
**[Test build #145383 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145383/testReport)** for PR 34352 at commit [`9bc0597`](https://github.com/apache/spark/commit/9bc0597fc60f441b4b80e1538cdb21b04f4ba684).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-972740486
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49856/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-948430392
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48969/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-948392347
**[Test build #144497 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144497/testReport)** for PR 34352 at commit [`89fc1c0`](https://github.com/apache/spark/commit/89fc1c0ac2d529972193e9feed1f35e8a93da00a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-1000251896
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50989/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-1000459914
**[Test build #146528 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146528/testReport)** for PR 34352 at commit [`f9a50e2`](https://github.com/apache/spark/commit/f9a50e2a7dd17bbdfee5b3a0eea9680ca87977e3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-972721024
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49856/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-972648563
**[Test build #145383 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145383/testReport)** for PR 34352 at commit [`9bc0597`](https://github.com/apache/spark/commit/9bc0597fc60f441b4b80e1538cdb21b04f4ba684).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-948392347
**[Test build #144497 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144497/testReport)** for PR 34352 at commit [`89fc1c0`](https://github.com/apache/spark/commit/89fc1c0ac2d529972193e9feed1f35e8a93da00a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-1000289366
**[Test build #146528 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146528/testReport)** for PR 34352 at commit [`f9a50e2`](https://github.com/apache/spark/commit/f9a50e2a7dd17bbdfee5b3a0eea9680ca87977e3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-1000475253
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146528/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-1000391409
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/51004/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-972488007
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49828/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-972874241
**[Test build #145393 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145393/testReport)** for PR 34352 at commit [`61d9614`](https://github.com/apache/spark/commit/61d9614bbf90a46be0900d23d3c8d565750d7c4c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-972491636
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145357/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-972786751
**[Test build #145387 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145387/testReport)** for PR 34352 at commit [`a60af8b`](https://github.com/apache/spark/commit/a60af8b0fb34453fd0fda0536b26fea521b46b8d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-1000391431
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/51004/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-1000390509
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146513/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-948475766
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48969/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-972687231
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49856/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-972471375
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49828/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-972488007
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49828/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-1000475253
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146528/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-1000289366
**[Test build #146528 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146528/testReport)** for PR 34352 at commit [`f9a50e2`](https://github.com/apache/spark/commit/f9a50e2a7dd17bbdfee5b3a0eea9680ca87977e3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-1000351098
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/51004/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34352:
URL: https://github.com/apache/spark/pull/34352#discussion_r733473172
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/internal/BaseSessionStateBuilder.scala
##########
@@ -410,6 +415,35 @@ class SparkUDFExpressionBuilder extends FunctionExpressionBuilder {
name, expr.inputTypes.size.toString, input.size)
}
expr
+ } else if (classOf[Aggregator[_, _, _]].isAssignableFrom(clazz)) {
+ val aggregator = clazz.getConstructor().newInstance().asInstanceOf[Aggregator[Any, Any, Any]]
+
+ // Construct the input encoder
+ val mirror = universe.runtimeMirror(clazz.getClassLoader)
+ val classType = mirror.classSymbol(clazz)
+ val baseClassType = universe.typeOf[Aggregator[_, _, _]].typeSymbol.asClass
+ val baseType = universe.internal.thisType(classType).baseType(baseClassType)
+ val tpe = baseType.typeArgs.head
+ val cls = mirror.runtimeClass(tpe)
Review comment:
Is there any reference for the code above? The code looks obscure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-948486976
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144497/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-948486976
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144497/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-972421833
**[Test build #145357 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145357/testReport)** for PR 34352 at commit [`2348081`](https://github.com/apache/spark/commit/234808187c68e9bb89bea04b1456f38f7bbf4cb2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-972740486
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49856/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-972787853
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145387/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-972858540
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49860/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-972969155
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49866/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-948485088
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48969/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-972491636
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145357/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-972868165
**[Test build #145383 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145383/testReport)** for PR 34352 at commit [`9bc0597`](https://github.com/apache/spark/commit/9bc0597fc60f441b4b80e1538cdb21b04f4ba684).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-972871371
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #34352:
URL: https://github.com/apache/spark/pull/34352#discussion_r752041419
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala
##########
@@ -50,6 +51,20 @@ private case class FunctionResult(f1: String, f2: String)
private case class LocalDateInstantType(date: LocalDate, instant: Instant)
private case class TimestampInstantType(t: Timestamp, instant: Instant)
+class MyDoubleAverage extends Aggregator[jlDouble, (Double, Long), jlDouble] {
Review comment:
1. the type parameter will be `IN`, we need find another way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-972817057
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49860/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-972910763
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49866/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-973156024
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145393/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34352:
URL: https://github.com/apache/spark/pull/34352#discussion_r775302106
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala
##########
@@ -848,6 +867,37 @@ class UDFSuite extends QueryTest with SharedSparkSession {
}
}
+ test("SPARK-37018: Spark SQL should support create function with Aggregator") {
+ val avgFuncClass = "org.apache.spark.sql.MyDoubleAverage"
+ val avgFunction = "my_avg"
+ val sumFuncClass = "org.apache.spark.sql.MyDoubleSum"
+ val sumFunction = "my_sum"
+ withTempDatabase { dbName =>
+ withUserDefinedFunction(
+ s"default.$avgFunction" -> false,
+ s"default.$sumFunction" -> false,
+ s"$dbName.$avgFunction" -> false,
+ s"$dbName.$sumFunction" -> false,
+ avgFunction -> true,
+ sumFunction -> true) {
Review comment:
sorry but a nit:
```suggestion
s"default.$avgFunction" -> false,
s"default.$sumFunction" -> false,
s"$dbName.$avgFunction" -> false,
s"$dbName.$sumFunction" -> false,
avgFunction -> true,
sumFunction -> true) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-972787822
**[Test build #145387 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145387/testReport)** for PR 34352 at commit [`a60af8b`](https://github.com/apache/spark/commit/a60af8b0fb34453fd0fda0536b26fea521b46b8d).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-972787853
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145387/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-972978809
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49866/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-972978809
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49866/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-972871371
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-948486712
**[Test build #144497 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144497/testReport)** for PR 34352 at commit [`89fc1c0`](https://github.com/apache/spark/commit/89fc1c0ac2d529972193e9feed1f35e8a93da00a).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34352:
URL: https://github.com/apache/spark/pull/34352#discussion_r733473500
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/internal/BaseSessionStateBuilder.scala
##########
@@ -410,6 +415,35 @@ class SparkUDFExpressionBuilder extends FunctionExpressionBuilder {
name, expr.inputTypes.size.toString, input.size)
}
expr
+ } else if (classOf[Aggregator[_, _, _]].isAssignableFrom(clazz)) {
+ val aggregator = clazz.getConstructor().newInstance().asInstanceOf[Aggregator[Any, Any, Any]]
+
+ // Construct the input encoder
+ val mirror = universe.runtimeMirror(clazz.getClassLoader)
+ val classType = mirror.classSymbol(clazz)
+ val baseClassType = universe.typeOf[Aggregator[_, _, _]].typeSymbol.asClass
+ val baseType = universe.internal.thisType(classType).baseType(baseClassType)
+ val tpe = baseType.typeArgs.head
+ val cls = mirror.runtimeClass(tpe)
+ val serializer = ScalaReflection.serializerForType(tpe)
+ val deserializer = ScalaReflection.deserializerForType(tpe)
+ val inputEncoder = new ExpressionEncoder[Any](
+ serializer,
+ deserializer,
+ ClassTag(cls))
Review comment:
```suggestion
val inputEncoder = new ExpressionEncoder[Any](serializer, deserializer, ClassTag(cls))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-1000251896
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50989/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-1000182314
**[Test build #146513 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146513/testReport)** for PR 34352 at commit [`e742552`](https://github.com/apache/spark/commit/e7425524f586a54585b7a986bef660e7303dfaae).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34352:
URL: https://github.com/apache/spark/pull/34352#discussion_r733476604
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala
##########
@@ -50,6 +51,20 @@ private case class FunctionResult(f1: String, f2: String)
private case class LocalDateInstantType(date: LocalDate, instant: Instant)
private case class TimestampInstantType(t: Timestamp, instant: Instant)
+class MyDoubleAverage extends Aggregator[jlDouble, (Double, Long), jlDouble] {
Review comment:
There are a few more stuff I'd like to test:
1. What if the class has type parameters? e.g. `class TypedCount[IN](val f: IN => Any) extends Aggregator[IN, Long, Long]`. What happens if we register it as a SQL UDAF?
2. Let's test negative cases, e.g., wrong num of parameters, wrong parameter types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-972491492
**[Test build #145357 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145357/testReport)** for PR 34352 at commit [`2348081`](https://github.com/apache/spark/commit/234808187c68e9bb89bea04b1456f38f7bbf4cb2).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-973156024
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145393/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-1000182314
**[Test build #146513 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146513/testReport)** for PR 34352 at commit [`e742552`](https://github.com/apache/spark/commit/e7425524f586a54585b7a986bef660e7303dfaae).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-1000249360
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50989/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-1000390509
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146513/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-1000391431
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/51004/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-972786751
**[Test build #145387 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145387/testReport)** for PR 34352 at commit [`a60af8b`](https://github.com/apache/spark/commit/a60af8b0fb34453fd0fda0536b26fea521b46b8d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-972421833
**[Test build #145357 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145357/testReport)** for PR 34352 at commit [`2348081`](https://github.com/apache/spark/commit/234808187c68e9bb89bea04b1456f38f7bbf4cb2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-972874241
**[Test build #145393 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145393/testReport)** for PR 34352 at commit [`61d9614`](https://github.com/apache/spark/commit/61d9614bbf90a46be0900d23d3c8d565750d7c4c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-1000206707
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50989/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-1000372386
**[Test build #146513 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146513/testReport)** for PR 34352 at commit [`e742552`](https://github.com/apache/spark/commit/e7425524f586a54585b7a986bef660e7303dfaae).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-948485088
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48969/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34352: [SPARK-37018][SQL] Spark SQL should support create function with Aggregator
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34352:
URL: https://github.com/apache/spark/pull/34352#issuecomment-973154318
**[Test build #145393 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145393/testReport)** for PR 34352 at commit [`61d9614`](https://github.com/apache/spark/commit/61d9614bbf90a46be0900d23d3c8d565750d7c4c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org