You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/06/06 06:31:56 UTC
[GitHub] [spark] beliefer opened a new pull request, #36773: [SPARK-39385][SQL] Translate linear regression aggregate functions for pushdown
beliefer opened a new pull request, #36773:
URL: https://github.com/apache/spark/pull/36773
### What changes were proposed in this pull request?
Spark supports a lot of linear regression aggregate functions now.
Because `REGR_AVGX`, `REGR_AVGY`, `REGR_COUNT`, `REGR_SXX` and `REGR_SXY` are replaced to other expression in runtime, This PR will only translate `REGR_INTERCEPT`, `REGR_R2`, `REGR_SLOPE`, `REGR_SXY` for pushdown.
### Why are the changes needed?
Make the implement of *Dialect could compile `REGR_INTERCEPT`, `REGR_R2`, `REGR_SLOPE`, `REGR_SXY`.
### Does this PR introduce _any_ user-facing change?
'No'.
New feature.
### How was this patch tested?
New tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #36773: [SPARK-39385][SQL] Translate linear regression aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #36773:
URL: https://github.com/apache/spark/pull/36773#discussion_r902357088
##########
sql/core/src/main/scala/org/apache/spark/sql/jdbc/H2Dialect.scala:
##########
@@ -60,18 +60,27 @@ private[sql] object H2Dialect extends JdbcDialect {
assert(f.children().length == 1)
val distinct = if (f.isDistinct) "DISTINCT " else ""
Some(s"STDDEV_SAMP($distinct${f.children().head})")
- case f: GeneralAggregateFunc if f.name() == "COVAR_POP" =>
+ case f: GeneralAggregateFunc if f.name() == "COVAR_POP" && f.isDistinct == false =>
Review Comment:
```suggestion
case f: GeneralAggregateFunc if f.name() == "COVAR_POP" && !f.isDistinct =>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #36773: [SPARK-39385][SQL] Translate linear regression aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
beliefer commented on PR #36773:
URL: https://github.com/apache/spark/pull/36773#issuecomment-1178484804
@cloud-fan @huaxingao Thank you for you review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #36773: [SPARK-39385][SQL] Translate linear regression aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #36773:
URL: https://github.com/apache/spark/pull/36773#discussion_r902356266
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala:
##########
@@ -750,6 +750,22 @@ object DataSourceStrategy
PushableColumnWithoutNestedColumn(right), _) =>
Some(new GeneralAggregateFunc("CORR", agg.isDistinct,
Array(FieldReference.column(left), FieldReference.column(right))))
+ case aggregate.RegrIntercept(PushableColumnWithoutNestedColumn(left),
+ PushableColumnWithoutNestedColumn(right)) =>
+ Some(new GeneralAggregateFunc("REGR_INTERCEPT", agg.isDistinct,
+ Array(FieldReference.column(left), FieldReference.column(right))))
+ case aggregate.RegrR2(PushableColumnWithoutNestedColumn(left),
Review Comment:
not only this PR, but why can't we use `PushableExpression`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #36773: [SPARK-39385][SQL] Translate linear regression aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
beliefer commented on PR #36773:
URL: https://github.com/apache/spark/pull/36773#issuecomment-1147223345
ping @huaxingao cc @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #36773: [SPARK-39385][SQL] Translate linear regression aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #36773:
URL: https://github.com/apache/spark/pull/36773#discussion_r916608215
##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCV2Suite.scala:
##########
@@ -1685,6 +1709,42 @@ class JDBCV2Suite extends QueryTest with SharedSparkSession with ExplainSuiteHel
checkAnswer(df2, Seq(Row(1d), Row(1d), Row(null)))
}
+ test("scan with aggregate push-down: linear regression functions with filter and group by") {
Review Comment:
@beliefer mind taking a look please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a diff in pull request #36773: [SPARK-39385][SQL] Translate linear regression aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
srowen commented on code in PR #36773:
URL: https://github.com/apache/spark/pull/36773#discussion_r890245878
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala:
##########
@@ -750,6 +750,22 @@ object DataSourceStrategy
PushableColumnWithoutNestedColumn(right), _) =>
Some(new GeneralAggregateFunc("CORR", agg.isDistinct,
Array(FieldReference.column(left), FieldReference.column(right))))
+ case aggregate.RegrIntercept(PushableColumnWithoutNestedColumn(left),
Review Comment:
Not required, but it'd be nice to find a way to fix the indents here - should not line up with 'case' in the continuation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #36773: [SPARK-39385][SQL] Translate linear regression aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
beliefer commented on PR #36773:
URL: https://github.com/apache/spark/pull/36773#issuecomment-1168614462
ping @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #36773: [SPARK-39385][SQL] Translate linear regression aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #36773:
URL: https://github.com/apache/spark/pull/36773#discussion_r916607535
##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCV2Suite.scala:
##########
@@ -1685,6 +1709,42 @@ class JDBCV2Suite extends QueryTest with SharedSparkSession with ExplainSuiteHel
checkAnswer(df2, Seq(Row(1d), Row(1d), Row(null)))
}
+ test("scan with aggregate push-down: linear regression functions with filter and group by") {
Review Comment:
This also seems to be failing with ANSI mode on:
```
2022-07-08T01:56:48.3914077Z [0m[[0m[0minfo[0m] [0m[0m[31m- scan with aggregate push-down: linear regression functions with filter and group by *** FAILED *** (350 milliseconds)[0m[0m
2022-07-08T01:56:48.3915454Z [0m[[0m[0minfo[0m] [0m[0m[31m org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 416.0 failed 1 times, most recent failure: Lost task 0.0 in stage 416.0 (TID 379) (localhost executor driver): org.apache.spark.SparkArithmeticException: [DIVIDE_BY_ZERO] Division by zero. Use `try_divide` to tolerate divisor being 0 and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" (except for ANSI interval type) to bypass this error.[0m[0m
2022-07-08T01:56:48.3916839Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.errors.QueryExecutionErrors$.divideByZeroError(QueryExecutionErrors.scala:184)[0m[0m
2022-07-08T01:56:48.3917602Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.errors.QueryExecutionErrors.divideByZeroError(QueryExecutionErrors.scala)[0m[0m
2022-07-08T01:56:48.3918487Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.hashAgg_doAggregateWithKeysOutput_0$(Unknown Source)[0m[0m
2022-07-08T01:56:48.3919355Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.processNext(Unknown Source)[0m[0m
2022-07-08T01:56:48.3920199Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)[0m[0m
2022-07-08T01:56:48.3921079Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:760)[0m[0m
2022-07-08T01:56:48.3921693Z [0m[[0m[0minfo[0m] [0m[0m[31m at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)[0m[0m
2022-07-08T01:56:48.3922208Z [0m[[0m[0minfo[0m] [0m[0m[31m at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)[0m[0m
2022-07-08T01:56:48.3922715Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1917)[0m[0m
2022-07-08T01:56:48.3923231Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.rdd.RDD.$anonfun$count$1(RDD.scala:1268)[0m[0m
2022-07-08T01:56:48.3923746Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.rdd.RDD.$anonfun$count$1$adapted(RDD.scala:1268)[0m[0m
2022-07-08T01:56:48.3924290Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2262)[0m[0m
2022-07-08T01:56:48.3924862Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92)[0m[0m
2022-07-08T01:56:48.3925414Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.scheduler.Task.run(Task.scala:139)[0m[0m
2022-07-08T01:56:48.3925982Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)[0m[0m
2022-07-08T01:56:48.3926545Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1490)[0m[0m
2022-07-08T01:56:48.3927090Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)[0m[0m
2022-07-08T01:56:48.3927701Z [0m[[0m[0minfo[0m] [0m[0m[31m at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[0m[0m
2022-07-08T01:56:48.3928327Z [0m[[0m[0minfo[0m] [0m[0m[31m at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[0m[0m
2022-07-08T01:56:48.3928833Z [0m[[0m[0minfo[0m] [0m[0m[31m at java.lang.Thread.run(Thread.java:750)[0m[0m
2022-07-08T01:56:48.3929196Z [0m[[0m[0minfo[0m] [0m[0m[31m[0m[0m
2022-07-08T01:56:48.3929548Z [0m[[0m[0minfo[0m] [0m[0m[31mDriver stacktrace:[0m[0m
2022-07-08T01:56:48.3930152Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2706)[0m[0m
2022-07-08T01:56:48.3930839Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2642)[0m[0m
2022-07-08T01:56:48.3931492Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2641)[0m[0m
2022-07-08T01:56:48.3932111Z [0m[[0m[0minfo[0m] [0m[0m[31m at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)[0m[0m
2022-07-08T01:56:48.3932717Z [0m[[0m[0minfo[0m] [0m[0m[31m at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)[0m[0m
2022-07-08T01:56:48.3933306Z [0m[[0m[0minfo[0m] [0m[0m[31m at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)[0m[0m
2022-07-08T01:56:48.3933969Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2641)[0m[0m
2022-07-08T01:56:48.3937642Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1189)[0m[0m
2022-07-08T01:56:48.3941965Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1189)[0m[0m
2022-07-08T01:56:48.3943119Z [0m[[0m[0minfo[0m] [0m[0m[31m at scala.Option.foreach(Option.scala:407)[0m[0m
2022-07-08T01:56:48.3943813Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1189)[0m[0m
2022-07-08T01:56:48.3944644Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2897)[0m[0m
2022-07-08T01:56:48.3945466Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2836)[0m[0m
2022-07-08T01:56:48.3946265Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2825)[0m[0m
2022-07-08T01:56:48.3947177Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)[0m[0m
2022-07-08T01:56:48.3947776Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:952)[0m[0m
2022-07-08T01:56:48.4330080Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.SparkContext.runJob(SparkContext.scala:2222)[0m[0m
2022-07-08T01:56:48.4330873Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.SparkContext.runJob(SparkContext.scala:2243)[0m[0m
2022-07-08T01:56:48.4331499Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.SparkContext.runJob(SparkContext.scala:2262)[0m[0m
2022-07-08T01:56:48.4332552Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.SparkContext.runJob(SparkContext.scala:2287)[0m[0m
2022-07-08T01:56:48.4333184Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.rdd.RDD.count(RDD.scala:1268)[0m[0m
2022-07-08T01:56:48.4333923Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.QueryTest$.$anonfun$getErrorMessageInCheckAnswer$1(QueryTest.scala:265)[0m[0m
2022-07-08T01:56:48.4334537Z [0m[[0m[0minfo[0m] [0m[0m[31m at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)[0m[0m
2022-07-08T01:56:48.4335504Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:171)[0m[0m
2022-07-08T01:56:48.4336158Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.QueryTest$.getErrorMessageInCheckAnswer(QueryTest.scala:265)[0m[0m
2022-07-08T01:56:48.4336802Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.QueryTest$.checkAnswer(QueryTest.scala:242)[0m[0m
2022-07-08T01:56:48.4337377Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.QueryTest.checkAnswer(QueryTest.scala:150)[0m[0m
2022-07-08T01:56:48.4337959Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.jdbc.JDBCV2Suite.$anonfun$new$204(JDBCV2Suite.scala:1745)[0m[0m
2022-07-08T01:56:48.4338601Z [0m[[0m[0minfo[0m] [0m[0m[31m at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)[0m[0m
2022-07-08T01:56:48.4339126Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)[0m[0m
2022-07-08T01:56:48.4339645Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)[0m[0m
2022-07-08T01:56:48.4340150Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)[0m[0m
2022-07-08T01:56:48.4340653Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.Transformer.apply(Transformer.scala:22)[0m[0m
2022-07-08T01:56:48.4341312Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.Transformer.apply(Transformer.scala:20)[0m[0m
2022-07-08T01:56:48.4369891Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:190)[0m[0m
2022-07-08T01:56:48.4370698Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:204)[0m[0m
2022-07-08T01:56:48.4371447Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:188)[0m[0m
2022-07-08T01:56:48.4372349Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:200)[0m[0m
2022-07-08T01:56:48.4373008Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)[0m[0m
2022-07-08T01:56:48.4373668Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:200)[0m[0m
2022-07-08T01:56:48.4374361Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:182)[0m[0m
2022-07-08T01:56:48.4375096Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:65)[0m[0m
2022-07-08T01:56:48.4375800Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)[0m[0m
2022-07-08T01:56:48.4376475Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)[0m[0m
2022-07-08T01:56:48.4377124Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:65)[0m[0m
2022-07-08T01:56:48.4377802Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:233)[0m[0m
2022-07-08T01:56:48.4378466Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)[0m[0m
2022-07-08T01:56:48.4379077Z [0m[[0m[0minfo[0m] [0m[0m[31m at scala.collection.immutable.List.foreach(List.scala:431)[0m[0m
2022-07-08T01:56:48.4379698Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)[0m[0m
2022-07-08T01:56:48.4380329Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)[0m[0m
2022-07-08T01:56:48.4380955Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)[0m[0m
2022-07-08T01:56:48.4381607Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:233)[0m[0m
2022-07-08T01:56:48.4382292Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:232)[0m[0m
2022-07-08T01:56:48.4382973Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1563)[0m[0m
2022-07-08T01:56:48.4383559Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.Suite.run(Suite.scala:1112)[0m[0m
2022-07-08T01:56:48.4384087Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.Suite.run$(Suite.scala:1094)[0m[0m
2022-07-08T01:56:48.4384760Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1563)[0m[0m
2022-07-08T01:56:48.4385468Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:237)[0m[0m
2022-07-08T01:56:48.4386094Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.SuperEngine.runImpl(Engine.scala:535)[0m[0m
2022-07-08T01:56:48.4386725Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:237)[0m[0m
2022-07-08T01:56:48.4387381Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:236)[0m[0m
2022-07-08T01:56:48.4388169Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:65)[0m[0m
2022-07-08T01:56:48.4388882Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)[0m[0m
2022-07-08T01:56:48.4389533Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)[0m[0m
2022-07-08T01:56:48.4390219Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)[0m[0m
2022-07-08T01:56:48.4390838Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:65)[0m[0m
2022-07-08T01:56:48.4391511Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:318)[0m[0m
2022-07-08T01:56:48.4392296Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:513)[0m[0m
2022-07-08T01:56:48.4433669Z [0m[[0m[0minfo[0m] [0m[0m[31m at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:413)[0m[0m
2022-07-08T01:56:48.4434367Z [0m[[0m[0minfo[0m] [0m[0m[31m at java.util.concurrent.FutureTask.run(FutureTask.java:266)[0m[0m
2022-07-08T01:56:48.4435084Z [0m[[0m[0minfo[0m] [0m[0m[31m at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[0m[0m
2022-07-08T01:56:48.4435808Z [0m[[0m[0minfo[0m] [0m[0m[31m at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[0m[0m
2022-07-08T01:56:48.4436787Z [0m[[0m[0minfo[0m] [0m[0m[31m at java.lang.Thread.run(Thread.java:750)[0m[0m
2022-07-08T01:56:48.4437780Z [0m[[0m[0minfo[0m] [0m[0m[31m Cause: org.apache.spark.SparkArithmeticException: [DIVIDE_BY_ZERO] Division by zero. Use `try_divide` to tolerate divisor being 0 and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" (except for ANSI interval type) to bypass this error.[0m[0m
2022-07-08T01:56:48.4438954Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.errors.QueryExecutionErrors$.divideByZeroError(QueryExecutionErrors.scala:184)[0m[0m
2022-07-08T01:56:48.4439793Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.errors.QueryExecutionErrors.divideByZeroError(QueryExecutionErrors.scala)[0m[0m
2022-07-08T01:56:48.4441245Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.hashAgg_doAggregateWithKeysOutput_0$(Unknown Source)[0m[0m
2022-07-08T01:56:48.4442235Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.processNext(Unknown Source)[0m[0m
2022-07-08T01:56:48.4443261Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)[0m[0m
2022-07-08T01:56:48.4444051Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:760)[0m[0m
2022-07-08T01:56:48.4444879Z [0m[[0m[0minfo[0m] [0m[0m[31m at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)[0m[0m
2022-07-08T01:56:48.4445474Z [0m[[0m[0minfo[0m] [0m[0m[31m at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)[0m[0m
2022-07-08T01:56:48.4446193Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1917)[0m[0m
2022-07-08T01:56:48.4447254Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.rdd.RDD.$anonfun$count$1(RDD.scala:1268)[0m[0m
2022-07-08T01:56:48.4448107Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.rdd.RDD.$anonfun$count$1$adapted(RDD.scala:1268)[0m[0m
2022-07-08T01:56:48.4449171Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2262)[0m[0m
2022-07-08T01:56:48.4449891Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92)[0m[0m
2022-07-08T01:56:48.4450452Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.scheduler.Task.run(Task.scala:139)[0m[0m
2022-07-08T01:56:48.4451018Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)[0m[0m
2022-07-08T01:56:48.4451581Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1490)[0m[0m
2022-07-08T01:56:48.4452202Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)[0m[0m
2022-07-08T01:56:48.4452818Z [0m[[0m[0minfo[0m] [0m[0m[31m at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[0m[0m
2022-07-08T01:56:48.4453447Z [0m[[0m[0minfo[0m] [0m[0m[31m at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[0m[0m
2022-07-08T01:56:48.4453968Z [0m[[0m[0minfo[0m] [0m[0m[31m at java.lang.Thread.run(Thread.java:750)[0m[0m
```
https://github.com/apache/spark/runs/7244240118?check_suite_focus=true
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a diff in pull request #36773: [SPARK-39385][SQL] Translate linear regression aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
beliefer commented on code in PR #36773:
URL: https://github.com/apache/spark/pull/36773#discussion_r890714758
##########
sql/core/src/main/scala/org/apache/spark/sql/jdbc/H2Dialect.scala:
##########
@@ -72,6 +72,26 @@ private[sql] object H2Dialect extends JdbcDialect {
assert(f.children().length == 2)
val distinct = if (f.isDistinct) "DISTINCT " else ""
Some(s"CORR($distinct${f.children().head}, ${f.children().last})")
+ case f: GeneralAggregateFunc if f.name() == "REGR_INTERCEPT" =>
+ assert(f.children().length == 2)
+ val distinct = if (f.isDistinct) "DISTINCT " else ""
+ Some(s"REGR_INTERCEPT($distinct${f.children().head}, ${f.children().last})")
+ case f: GeneralAggregateFunc if f.name() == "REGR_R2" =>
+ assert(f.children().length == 2)
+ val distinct = if (f.isDistinct) "DISTINCT " else ""
+ Some(s"REGR_R2($distinct${f.children().head}, ${f.children().last})")
+ case f: GeneralAggregateFunc if f.name() == "REGR_SLOPE" =>
+ assert(f.children().length == 2)
+ val distinct = if (f.isDistinct) "DISTINCT " else ""
+ Some(s"REGR_SLOPE($distinct${f.children().head}, ${f.children().last})")
+ case f: GeneralAggregateFunc if f.name() == "REGR_SXX" =>
+ assert(f.children().length == 2)
+ val distinct = if (f.isDistinct) "DISTINCT " else ""
+ Some(s"REGR_SXX($distinct${f.children().head}, ${f.children().last})")
+ case f: GeneralAggregateFunc if f.name() == "REGR_SXY" =>
+ assert(f.children().length == 2)
+ val distinct = if (f.isDistinct) "DISTINCT " else ""
Review Comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #36773: [SPARK-39385][SQL] Translate linear regression aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #36773:
URL: https://github.com/apache/spark/pull/36773#discussion_r913490082
##########
sql/core/src/main/scala/org/apache/spark/sql/jdbc/H2Dialect.scala:
##########
@@ -62,18 +62,27 @@ private[sql] object H2Dialect extends JdbcDialect {
assert(f.children().length == 1)
val distinct = if (f.isDistinct) "DISTINCT " else ""
Some(s"STDDEV_SAMP($distinct${f.children().head})")
- case f: GeneralAggregateFunc if f.name() == "COVAR_POP" =>
+ case f: GeneralAggregateFunc if f.name() == "COVAR_POP" && !f.isDistinct =>
assert(f.children().length == 2)
- val distinct = if (f.isDistinct) "DISTINCT " else ""
Review Comment:
so this is a bug fix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a diff in pull request #36773: [SPARK-39385][SQL] Translate linear regression aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
beliefer commented on code in PR #36773:
URL: https://github.com/apache/spark/pull/36773#discussion_r913565329
##########
sql/core/src/main/scala/org/apache/spark/sql/jdbc/H2Dialect.scala:
##########
@@ -62,18 +62,27 @@ private[sql] object H2Dialect extends JdbcDialect {
assert(f.children().length == 1)
val distinct = if (f.isDistinct) "DISTINCT " else ""
Some(s"STDDEV_SAMP($distinct${f.children().head})")
- case f: GeneralAggregateFunc if f.name() == "COVAR_POP" =>
+ case f: GeneralAggregateFunc if f.name() == "COVAR_POP" && !f.isDistinct =>
assert(f.children().length == 2)
- val distinct = if (f.isDistinct) "DISTINCT " else ""
Review Comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a diff in pull request #36773: [SPARK-39385][SQL] Translate linear regression aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
beliefer commented on code in PR #36773:
URL: https://github.com/apache/spark/pull/36773#discussion_r913478436
##########
sql/core/src/main/scala/org/apache/spark/sql/jdbc/H2Dialect.scala:
##########
@@ -62,18 +62,27 @@ private[sql] object H2Dialect extends JdbcDialect {
assert(f.children().length == 1)
val distinct = if (f.isDistinct) "DISTINCT " else ""
Some(s"STDDEV_SAMP($distinct${f.children().head})")
- case f: GeneralAggregateFunc if f.name() == "COVAR_POP" =>
+ case f: GeneralAggregateFunc if f.name() == "COVAR_POP" && !f.isDistinct =>
assert(f.children().length == 2)
- val distinct = if (f.isDistinct) "DISTINCT " else ""
Review Comment:
No. Just H2 does not supports `COVAR_POP` with distinct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #36773: [SPARK-39385][SQL] Translate linear regression aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #36773:
URL: https://github.com/apache/spark/pull/36773#discussion_r913497712
##########
sql/core/src/main/scala/org/apache/spark/sql/jdbc/H2Dialect.scala:
##########
@@ -62,18 +62,27 @@ private[sql] object H2Dialect extends JdbcDialect {
assert(f.children().length == 1)
val distinct = if (f.isDistinct) "DISTINCT " else ""
Some(s"STDDEV_SAMP($distinct${f.children().head})")
- case f: GeneralAggregateFunc if f.name() == "COVAR_POP" =>
+ case f: GeneralAggregateFunc if f.name() == "COVAR_POP" && !f.isDistinct =>
assert(f.children().length == 2)
- val distinct = if (f.isDistinct) "DISTINCT " else ""
Review Comment:
Can we have a separate PR for this? I think previously the query fails if we push down distinct COVAR_POP to H2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a diff in pull request #36773: [SPARK-39385][SQL] Translate linear regression aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
huaxingao commented on code in PR #36773:
URL: https://github.com/apache/spark/pull/36773#discussion_r890658489
##########
sql/core/src/main/scala/org/apache/spark/sql/jdbc/H2Dialect.scala:
##########
@@ -72,6 +72,26 @@ private[sql] object H2Dialect extends JdbcDialect {
assert(f.children().length == 2)
val distinct = if (f.isDistinct) "DISTINCT " else ""
Some(s"CORR($distinct${f.children().head}, ${f.children().last})")
+ case f: GeneralAggregateFunc if f.name() == "REGR_INTERCEPT" =>
+ assert(f.children().length == 2)
+ val distinct = if (f.isDistinct) "DISTINCT " else ""
+ Some(s"REGR_INTERCEPT($distinct${f.children().head}, ${f.children().last})")
+ case f: GeneralAggregateFunc if f.name() == "REGR_R2" =>
+ assert(f.children().length == 2)
+ val distinct = if (f.isDistinct) "DISTINCT " else ""
+ Some(s"REGR_R2($distinct${f.children().head}, ${f.children().last})")
+ case f: GeneralAggregateFunc if f.name() == "REGR_SLOPE" =>
+ assert(f.children().length == 2)
+ val distinct = if (f.isDistinct) "DISTINCT " else ""
+ Some(s"REGR_SLOPE($distinct${f.children().head}, ${f.children().last})")
+ case f: GeneralAggregateFunc if f.name() == "REGR_SXX" =>
Review Comment:
Do we ever hit this path? In `translateAggregate` you don't have `REGR_SXX` and in the PR description it is said that "...... REGR_SXX and REGR_SXY are replaced to other expression in runtime"? Actually `REGR_SXY` is not converted to other expression in runtime, right?
##########
sql/core/src/main/scala/org/apache/spark/sql/jdbc/H2Dialect.scala:
##########
@@ -72,6 +72,26 @@ private[sql] object H2Dialect extends JdbcDialect {
assert(f.children().length == 2)
val distinct = if (f.isDistinct) "DISTINCT " else ""
Some(s"CORR($distinct${f.children().head}, ${f.children().last})")
+ case f: GeneralAggregateFunc if f.name() == "REGR_INTERCEPT" =>
+ assert(f.children().length == 2)
+ val distinct = if (f.isDistinct) "DISTINCT " else ""
+ Some(s"REGR_INTERCEPT($distinct${f.children().head}, ${f.children().last})")
+ case f: GeneralAggregateFunc if f.name() == "REGR_R2" =>
+ assert(f.children().length == 2)
+ val distinct = if (f.isDistinct) "DISTINCT " else ""
+ Some(s"REGR_R2($distinct${f.children().head}, ${f.children().last})")
+ case f: GeneralAggregateFunc if f.name() == "REGR_SLOPE" =>
+ assert(f.children().length == 2)
+ val distinct = if (f.isDistinct) "DISTINCT " else ""
+ Some(s"REGR_SLOPE($distinct${f.children().head}, ${f.children().last})")
+ case f: GeneralAggregateFunc if f.name() == "REGR_SXX" =>
+ assert(f.children().length == 2)
+ val distinct = if (f.isDistinct) "DISTINCT " else ""
+ Some(s"REGR_SXX($distinct${f.children().head}, ${f.children().last})")
+ case f: GeneralAggregateFunc if f.name() == "REGR_SXY" =>
+ assert(f.children().length == 2)
+ val distinct = if (f.isDistinct) "DISTINCT " else ""
Review Comment:
Can we test `DISTINCT` too?
##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCV2Suite.scala:
##########
@@ -1111,6 +1111,28 @@ class JDBCV2Suite extends QueryTest with SharedSparkSession with ExplainSuiteHel
checkAnswer(df, Seq(Row(1d), Row(1d), Row(null)))
}
+ test("scan with aggregate push-down: linear regression functions with filter and group by") {
+ val df = sql(
+ """
+ |SELECT
+ | REGR_INTERCEPT(bonus, bonus),
+ | REGR_R2(bonus, bonus),
+ | REGR_SLOPE(bonus, bonus),
+ | REGR_SXY(bonus, bonus)
+ |FROM h2.test.employee where dept > 0 group by DePt""".stripMargin)
Review Comment:
nit: capitalize the sql keywords `where` and `group by`?
I just noticed that not all the sql keywords in this test suite are capitalized. Probably open a separate PR to fix them all?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a diff in pull request #36773: [SPARK-39385][SQL] Translate linear regression aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
beliefer commented on code in PR #36773:
URL: https://github.com/apache/spark/pull/36773#discussion_r890715857
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala:
##########
@@ -750,6 +750,22 @@ object DataSourceStrategy
PushableColumnWithoutNestedColumn(right), _) =>
Some(new GeneralAggregateFunc("CORR", agg.isDistinct,
Array(FieldReference.column(left), FieldReference.column(right))))
+ case aggregate.RegrIntercept(PushableColumnWithoutNestedColumn(left),
Review Comment:
two indents ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #36773: [SPARK-39385][SQL] Translate linear regression aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #36773:
URL: https://github.com/apache/spark/pull/36773#discussion_r916752602
##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCV2Suite.scala:
##########
@@ -1685,6 +1709,42 @@ class JDBCV2Suite extends QueryTest with SharedSparkSession with ExplainSuiteHel
checkAnswer(df2, Seq(Row(1d), Row(1d), Row(null)))
}
+ test("scan with aggregate push-down: linear regression functions with filter and group by") {
Review Comment:
Made a PR: https://github.com/apache/spark/pull/37137
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #36773: [SPARK-39385][SQL] Translate linear regression aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #36773: [SPARK-39385][SQL] Translate linear regression aggregate functions for pushdown
URL: https://github.com/apache/spark/pull/36773
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a diff in pull request #36773: [SPARK-39385][SQL] Translate linear regression aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
beliefer commented on code in PR #36773:
URL: https://github.com/apache/spark/pull/36773#discussion_r890696454
##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCV2Suite.scala:
##########
@@ -1111,6 +1111,28 @@ class JDBCV2Suite extends QueryTest with SharedSparkSession with ExplainSuiteHel
checkAnswer(df, Seq(Row(1d), Row(1d), Row(null)))
}
+ test("scan with aggregate push-down: linear regression functions with filter and group by") {
+ val df = sql(
+ """
+ |SELECT
+ | REGR_INTERCEPT(bonus, bonus),
+ | REGR_R2(bonus, bonus),
+ | REGR_SLOPE(bonus, bonus),
+ | REGR_SXY(bonus, bonus)
+ |FROM h2.test.employee where dept > 0 group by DePt""".stripMargin)
Review Comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a diff in pull request #36773: [SPARK-39385][SQL] Translate linear regression aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
beliefer commented on code in PR #36773:
URL: https://github.com/apache/spark/pull/36773#discussion_r890694554
##########
sql/core/src/main/scala/org/apache/spark/sql/jdbc/H2Dialect.scala:
##########
@@ -72,6 +72,26 @@ private[sql] object H2Dialect extends JdbcDialect {
assert(f.children().length == 2)
val distinct = if (f.isDistinct) "DISTINCT " else ""
Some(s"CORR($distinct${f.children().head}, ${f.children().last})")
+ case f: GeneralAggregateFunc if f.name() == "REGR_INTERCEPT" =>
+ assert(f.children().length == 2)
+ val distinct = if (f.isDistinct) "DISTINCT " else ""
+ Some(s"REGR_INTERCEPT($distinct${f.children().head}, ${f.children().last})")
+ case f: GeneralAggregateFunc if f.name() == "REGR_R2" =>
+ assert(f.children().length == 2)
+ val distinct = if (f.isDistinct) "DISTINCT " else ""
+ Some(s"REGR_R2($distinct${f.children().head}, ${f.children().last})")
+ case f: GeneralAggregateFunc if f.name() == "REGR_SLOPE" =>
+ assert(f.children().length == 2)
+ val distinct = if (f.isDistinct) "DISTINCT " else ""
+ Some(s"REGR_SLOPE($distinct${f.children().head}, ${f.children().last})")
+ case f: GeneralAggregateFunc if f.name() == "REGR_SXX" =>
Review Comment:
Thank you. I forgot to remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #36773: [SPARK-39385][SQL] Translate linear regression aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on PR #36773:
URL: https://github.com/apache/spark/pull/36773#issuecomment-1177762210
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #36773: [SPARK-39385][SQL] Translate linear regression aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #36773:
URL: https://github.com/apache/spark/pull/36773#discussion_r916569488
##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCV2Suite.scala:
##########
@@ -1685,6 +1709,42 @@ class JDBCV2Suite extends QueryTest with SharedSparkSession with ExplainSuiteHel
checkAnswer(df2, Seq(Row(1d), Row(1d), Row(null)))
}
+ test("scan with aggregate push-down: linear regression functions with filter and group by") {
Review Comment:
Hm, this test seems flaky:
```
org.scalatest.exceptions.TestFailedException: "== Parsed Logical Plan ==
'Aggregate ['DePt], [unresolvedalias('REGR_INTERCEPT('bonus, 'bonus), None), unresolvedalias('REGR_R2('bonus, 'bonus), None), unresolvedalias('REGR_SLOPE('bonus, 'bonus), None), unresolvedalias('REGR_SXY('bonus, 'bonus), None)]
+- 'Filter ('dept > 0)
+- 'UnresolvedRelation [h2, test, employee], [], false
== Analyzed Logical Plan ==
regr_intercept(bonus, bonus): double, regr_r2(bonus, bonus): double, regr_slope(bonus, bonus): double, regr_sxy(bonus, bonus): double
Aggregate [DePt#x], [regr_intercept(bonus#x, bonus#x) AS regr_intercept(bonus, bonus)#x, regr_r2(bonus#x, bonus#x) AS regr_r2(bonus, bonus)#x, regr_slope(bonus#x, bonus#x) AS regr_slope(bonus, bonus)#x, regr_sxy(bonus#x, bonus#x) AS regr_sxy(bonus, bonus)#x]
+- Filter (dept#x > 0)
+- SubqueryAlias h2.test.employee
+- RelationV2[DEPT#x, NAME#x, SALARY#x, BONUS#x, IS_MANAGER#x] h2.test.employee test.employee
== Optimized Logical Plan ==
Project [REGR_INTERCEPT(BONUS, BONUS)#x AS regr_intercept(bonus#x, bonus#x)#x AS regr_intercept(bonus, bonus)#x, REGR_R2(BONUS, BONUS)#x AS regr_r2(bonus#x, bonus#x)#x AS regr_r2(bonus, bonus)#x, REGR_SLOPE(BONUS, BONUS)#x AS regr_slope(bonus#x, bonus#x)#x AS regr_slope(bonus, bonus)#x, REGR_SXY(BONUS, BONUS)#x AS regr_sxy(bonus#x, bonus#x)#x AS regr_sxy(bonus, bonus)#x]
+- RelationV2[DEPT#x, REGR_INTERCEPT(BONUS, BONUS)#x, REGR_R2(BONUS, BONUS)#x, REGR_SLOPE(BONUS, BONUS)#x, REGR_SXY(BONUS, BONUS)#x] test.employee
== Physical Plan ==
*(1) Project [REGR_INTERCEPT(BONUS, BONUS)#x AS regr_intercept(bonus#x, bonus#x)#x AS regr_intercept(bonus, bonus)#x, REGR_R2(BONUS, BONUS)#x AS regr_r2(bonus#x, bonus#x)#x AS regr_r2(bonus, bonus)#x, REGR_SLOPE(BONUS, BONUS)#x AS regr_slope(bonus#x, bonus#x)#x AS regr_slope(bonus, bonus)#x, REGR_SXY(BONUS, BONUS)#x AS regr_sxy(bonus#x, bonus#x)#x AS regr_sxy(bonus, bonus)#x]
+- *(1) Scan org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCScan$$anon$1@6cbb46ad [DEPT#x,REGR_INTERCEPT(BONUS, BONUS)#x,REGR_R2(BONUS, BONUS)#x,REGR_SLOPE(BONUS, BONUS)#x,REGR_SXY(BONUS, BONUS)#x] PushedAggregates: [REGR_INTERCEPT(BONUS, BONUS), REGR_R2(BONUS, BONUS), REGR_SLOPE(BONUS, BONUS), REGR_SXY(BONUS, BONUS)], PushedFilters: [DEPT IS NOT NULL, DEPT > 0], PushedGroupByExpressions: [DEPT], ReadSchema: struct<DEPT:int,REGR_INTERCEPT(BONUS, BONUS):double,REGR_R2(BONUS, BONUS):double,REGR_SLOPE(BONUS, BONUS):double,REGR_SXY(BONUS, BONUS):double>
" did not contain " PushedAggregates: [REGR_INTERCEPT(BONUS, BONUS), REGR_R2(BONUS, BONUS), REGR_SLOPE(BONUS, BONUS), REGR_SXY(BONUS, B..., PushedFilters: [DEPT IS NOT NULL, DEPT > 0], PushedGroupByExpressions: [DEPT], "
```
https://github.com/apache/spark/runs/7247205123
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a diff in pull request #36773: [SPARK-39385][SQL] Translate linear regression aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
beliefer commented on code in PR #36773:
URL: https://github.com/apache/spark/pull/36773#discussion_r913493163
##########
sql/core/src/main/scala/org/apache/spark/sql/jdbc/H2Dialect.scala:
##########
@@ -62,18 +62,27 @@ private[sql] object H2Dialect extends JdbcDialect {
assert(f.children().length == 1)
val distinct = if (f.isDistinct) "DISTINCT " else ""
Some(s"STDDEV_SAMP($distinct${f.children().head})")
- case f: GeneralAggregateFunc if f.name() == "COVAR_POP" =>
+ case f: GeneralAggregateFunc if f.name() == "COVAR_POP" && !f.isDistinct =>
assert(f.children().length == 2)
- val distinct = if (f.isDistinct) "DISTINCT " else ""
Review Comment:
Yes. remove no need code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a diff in pull request #36773: [SPARK-39385][SQL] Translate linear regression aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
beliefer commented on code in PR #36773:
URL: https://github.com/apache/spark/pull/36773#discussion_r913478436
##########
sql/core/src/main/scala/org/apache/spark/sql/jdbc/H2Dialect.scala:
##########
@@ -62,18 +62,27 @@ private[sql] object H2Dialect extends JdbcDialect {
assert(f.children().length == 1)
val distinct = if (f.isDistinct) "DISTINCT " else ""
Some(s"STDDEV_SAMP($distinct${f.children().head})")
- case f: GeneralAggregateFunc if f.name() == "COVAR_POP" =>
+ case f: GeneralAggregateFunc if f.name() == "COVAR_POP" && !f.isDistinct =>
assert(f.children().length == 2)
- val distinct = if (f.isDistinct) "DISTINCT " else ""
Review Comment:
No. Just H2 not supports `COVAR_POP` with distinct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #36773: [SPARK-39385][SQL] Translate linear regression aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #36773:
URL: https://github.com/apache/spark/pull/36773#discussion_r913497712
##########
sql/core/src/main/scala/org/apache/spark/sql/jdbc/H2Dialect.scala:
##########
@@ -62,18 +62,27 @@ private[sql] object H2Dialect extends JdbcDialect {
assert(f.children().length == 1)
val distinct = if (f.isDistinct) "DISTINCT " else ""
Some(s"STDDEV_SAMP($distinct${f.children().head})")
- case f: GeneralAggregateFunc if f.name() == "COVAR_POP" =>
+ case f: GeneralAggregateFunc if f.name() == "COVAR_POP" && !f.isDistinct =>
assert(f.children().length == 2)
- val distinct = if (f.isDistinct) "DISTINCT " else ""
Review Comment:
Can we have a separate PR for this? I think previously the query fails if the push down distinct COVAR_POP to H2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #36773: [SPARK-39385][SQL] Translate linear regression aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #36773:
URL: https://github.com/apache/spark/pull/36773#discussion_r913426052
##########
sql/core/src/main/scala/org/apache/spark/sql/jdbc/H2Dialect.scala:
##########
@@ -62,18 +62,27 @@ private[sql] object H2Dialect extends JdbcDialect {
assert(f.children().length == 1)
val distinct = if (f.isDistinct) "DISTINCT " else ""
Some(s"STDDEV_SAMP($distinct${f.children().head})")
- case f: GeneralAggregateFunc if f.name() == "COVAR_POP" =>
+ case f: GeneralAggregateFunc if f.name() == "COVAR_POP" && !f.isDistinct =>
assert(f.children().length == 2)
- val distinct = if (f.isDistinct) "DISTINCT " else ""
Review Comment:
does it mean all distinct functions are not supported now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #36773: [SPARK-39385][SQL] Translate linear regression aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #36773:
URL: https://github.com/apache/spark/pull/36773#discussion_r916719849
##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCV2Suite.scala:
##########
@@ -1685,6 +1709,42 @@ class JDBCV2Suite extends QueryTest with SharedSparkSession with ExplainSuiteHel
checkAnswer(df2, Seq(Row(1d), Row(1d), Row(null)))
}
+ test("scan with aggregate push-down: linear regression functions with filter and group by") {
Review Comment:
Looks like this is a problem from `Covariance` (from `RegrIntercept.covarPop`) which uses `/` that throws an exception when ANSI mode is on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #36773: [SPARK-39385][SQL] Translate linear regression aggregate functions for pushdown
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #36773:
URL: https://github.com/apache/spark/pull/36773#discussion_r916571989
##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCV2Suite.scala:
##########
@@ -1685,6 +1709,42 @@ class JDBCV2Suite extends QueryTest with SharedSparkSession with ExplainSuiteHel
checkAnswer(df2, Seq(Row(1d), Row(1d), Row(null)))
}
+ test("scan with aggregate push-down: linear regression functions with filter and group by") {
Review Comment:
it's due to a logical conflict between 2 PR merged at the same time. It has been fixed now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org