You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/08 07:56:00 UTC

[GitHub] [spark] linhongliu-db opened a new pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

linhongliu-db opened a new pull request #29673:
URL: https://github.com/apache/spark/pull/29673


   ### What changes were proposed in this pull request?
   This PR fixes a conflict between `RewriteDistinctAggregates` and `DecimalAggregates`.
   In some cases, `DecimalAggregates` will wrap the decimal column to `UnscaledValue` using
   different rules for different aggregates.
   
   This means, same distinct column with different aggregates will change to different distinct columns
   after `DecimalAggregates`. For example:
   `avg(distinct decimal_col), sum(distinct decimal_col)` may change to
   `avg(distinct UnscaledValue(decimal_col)), sum(distinct decimal_col)`
   
   We assume after `RewriteDistinctAggregates`, there will be at most one distinct column in aggregates,
   but `DecimalAggregates` breaks this assumption. To fix this, we have to switch the order of these two
   rules.
   
   
   ### Why are the changes needed?
   bug fix
   
   
   ### Does this PR introduce _any_ user-facing change?
   no
   
   
   ### How was this patch tested?
   added test cases
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] maropu commented on a change in pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29673:
URL: https://github.com/apache/spark/pull/29673#discussion_r485977538



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
##########
@@ -2555,6 +2555,15 @@ class DataFrameSuite extends QueryTest
     val df = Seq(0.0 -> -0.0).toDF("pos", "neg")
     checkAnswer(df.select($"pos" > $"neg"), Row(false))
   }
+
+  test("SPARK-32816: aggregating multiple distinct DECIMAL columns") {
+    spark.range(0, 100, 1, 1)
+      .selectExpr("id", "cast(id as decimal(9, 0)) as decimal_col")
+      .createOrReplaceTempView("test_table")

Review comment:
       nit: Also, could you move this test into `SQLQueryTestSuite`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-689945828






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-690423487






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-688695354


   **[Test build #128394 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128394/testReport)** for PR 29673 at commit [`b1ce4b5`](https://github.com/apache/spark/commit/b1ce4b5fe12d32a867b281d48556f952932afd55).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] TJX2014 commented on a change in pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
TJX2014 commented on a change in pull request #29673:
URL: https://github.com/apache/spark/pull/29673#discussion_r484740633



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
##########
@@ -2555,6 +2555,19 @@ class DataFrameSuite extends QueryTest
     val df = Seq(0.0 -> -0.0).toDF("pos", "neg")
     checkAnswer(df.select($"pos" > $"neg"), Row(false))
   }
+
+  test("SPARK-32816: aggregating multiple distinct DECIMAL columns") {
+    withTempPath { path =>
+      spark.range(0, 100, 1, 1)
+        .selectExpr("id", "cast(id as decimal(9, 0)) as decimal_col")
+        .write.mode("overwrite")
+        .parquet(path.getAbsolutePath)
+      spark.read.parquet(path.getAbsolutePath).createOrReplaceTempView("test_table")

Review comment:
       val df = spark.range(0, 50000, 1, 1).selectExpr("id", "cast(id as decimal(9, 0)) as ss_ext_list_price")
   df.createOrReplaceTempView("test_table")
   sql("select avg(distinct ss_ext_list_price), sum(distinct ss_ext_list_price) from test_table").explain 
   **seems enough to reproduce.**




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-690894654






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-689945828






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-690893672






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-690894654






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-690894654






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29673:
URL: https://github.com/apache/spark/pull/29673#discussion_r485670399



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
##########
@@ -2555,6 +2555,15 @@ class DataFrameSuite extends QueryTest
     val df = Seq(0.0 -> -0.0).toDF("pos", "neg")
     checkAnswer(df.select($"pos" > $"neg"), Row(false))
   }
+
+  test("SPARK-32816: aggregating multiple distinct DECIMAL columns") {
+    spark.range(0, 100, 1, 1)
+      .selectExpr("id", "cast(id as decimal(9, 0)) as decimal_col")
+      .createOrReplaceTempView("test_table")

Review comment:
       nit: wrap the test with `withTempView`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-690894654






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-690804618


   **[Test build #128540 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128540/testReport)** for PR 29673 at commit [`8510ff9`](https://github.com/apache/spark/commit/8510ff92177554d9e87bd2e30e6fa9e43f89b8e3).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-693532554


   thanks, merging to master!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] linhongliu-db commented on a change in pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
linhongliu-db commented on a change in pull request #29673:
URL: https://github.com/apache/spark/pull/29673#discussion_r486707046



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
##########
@@ -2555,6 +2555,16 @@ class DataFrameSuite extends QueryTest
     val df = Seq(0.0 -> -0.0).toDF("pos", "neg")
     checkAnswer(df.select($"pos" > $"neg"), Row(false))
   }
+
+  test("SPARK-32816: aggregating multiple distinct DECIMAL columns") {
+    checkAnswer(
+      sql(
+        s"""
+           |SELECT AVG(DISTINCT decimal_col), SUM(DISTINCT decimal_col)
+           |  FROM VALUES (CAST(1 AS DECIMAL(9, 0))) t(decimal_col)
+        """.stripMargin),
+      Row(1, 1))

Review comment:
       sorry, I forgot to remove this.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] linhongliu-db commented on a change in pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
linhongliu-db commented on a change in pull request #29673:
URL: https://github.com/apache/spark/pull/29673#discussion_r486034818



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
##########
@@ -2555,6 +2555,17 @@ class DataFrameSuite extends QueryTest
     val df = Seq(0.0 -> -0.0).toDF("pos", "neg")
     checkAnswer(df.select($"pos" > $"neg"), Row(false))
   }
+
+  test("SPARK-32816: aggregating multiple distinct DECIMAL columns") {
+    withTempView("test_table") {
+      spark.range(0, 100, 1, 1)
+        .selectExpr("id", "cast(id as decimal(9, 0)) as decimal_col")
+        .createOrReplaceTempView("test_table")

Review comment:
       sure




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-690110156






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-690036852


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128480/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-688695354


   **[Test build #128394 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128394/testReport)** for PR 29673 at commit [`b1ce4b5`](https://github.com/apache/spark/commit/b1ce4b5fe12d32a867b281d48556f952932afd55).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-690112934


   **[Test build #128504 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128504/testReport)** for PR 29673 at commit [`f2111df`](https://github.com/apache/spark/commit/f2111dfc4e22b11de82e6202e7f53992d492eef2).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] TJX2014 commented on a change in pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
TJX2014 commented on a change in pull request #29673:
URL: https://github.com/apache/spark/pull/29673#discussion_r484740633



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
##########
@@ -2555,6 +2555,19 @@ class DataFrameSuite extends QueryTest
     val df = Seq(0.0 -> -0.0).toDF("pos", "neg")
     checkAnswer(df.select($"pos" > $"neg"), Row(false))
   }
+
+  test("SPARK-32816: aggregating multiple distinct DECIMAL columns") {
+    withTempPath { path =>
+      spark.range(0, 100, 1, 1)
+        .selectExpr("id", "cast(id as decimal(9, 0)) as decimal_col")
+        .write.mode("overwrite")
+        .parquet(path.getAbsolutePath)
+      spark.read.parquet(path.getAbsolutePath).createOrReplaceTempView("test_table")

Review comment:
       `val df = spark.range(0, 50000, 1, 1).selectExpr("id", "cast(id as decimal(9, 0)) as ss_ext_list_price")
   df.createOrReplaceTempView("test_table")
   sql("select avg(distinct ss_ext_list_price), sum(distinct ss_ext_list_price) from test_table").explain` seems enough to reproduce.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-690423487






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-690804916






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-690804916






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] maropu commented on a change in pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29673:
URL: https://github.com/apache/spark/pull/29673#discussion_r485977416



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
##########
@@ -2555,6 +2555,15 @@ class DataFrameSuite extends QueryTest
     val df = Seq(0.0 -> -0.0).toDF("pos", "neg")
     checkAnswer(df.select($"pos" > $"neg"), Row(false))
   }
+
+  test("SPARK-32816: aggregating multiple distinct DECIMAL columns") {
+    spark.range(0, 100, 1, 1)
+      .selectExpr("id", "cast(id as decimal(9, 0)) as decimal_col")
+      .createOrReplaceTempView("test_table")

Review comment:
       How about writing it like this w/o a temp view;
   ```
     test("SPARK-32816: aggregating multiple distinct DECIMAL columns") {
       checkAnswer(
         sql(
           s"""
              |SELECT AVG(DISTINCT decimal_col), SUM(DISTINCT decimal_col)
              |  FROM VALUES (CAST(1 AS DECIMAL(9, 0))) t(decimal_col)
            """.stripMargin),
         Row(XXX, XXX))
     }
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-688948058






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-689264278






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-690035161


   **[Test build #128480 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128480/testReport)** for PR 29673 at commit [`737f996`](https://github.com/apache/spark/commit/737f9965ca6e87e7c43a87e56da7f47a4e08e43e).
    * This patch **fails due to an unknown error code, -9**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-690112934


   **[Test build #128504 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128504/testReport)** for PR 29673 at commit [`f2111df`](https://github.com/apache/spark/commit/f2111dfc4e22b11de82e6202e7f53992d492eef2).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] linhongliu-db commented on a change in pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
linhongliu-db commented on a change in pull request #29673:
URL: https://github.com/apache/spark/pull/29673#discussion_r486034720



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
##########
@@ -2555,6 +2555,15 @@ class DataFrameSuite extends QueryTest
     val df = Seq(0.0 -> -0.0).toDF("pos", "neg")
     checkAnswer(df.select($"pos" > $"neg"), Row(false))
   }
+
+  test("SPARK-32816: aggregating multiple distinct DECIMAL columns") {
+    spark.range(0, 100, 1, 1)
+      .selectExpr("id", "cast(id as decimal(9, 0)) as decimal_col")
+      .createOrReplaceTempView("test_table")

Review comment:
       let me try this, I'm not sure if literal will behave differently




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-688691697






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-690894654






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-689354327






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-689263764


   **[Test build #128428 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128428/testReport)** for PR 29673 at commit [`4df4f7c`](https://github.com/apache/spark/commit/4df4f7c8ebc80bcf854fb26764338789ea13b319).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-688691697






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-689945509


   **[Test build #128480 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128480/testReport)** for PR 29673 at commit [`737f996`](https://github.com/apache/spark/commit/737f9965ca6e87e7c43a87e56da7f47a4e08e43e).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29673:
URL: https://github.com/apache/spark/pull/29673#discussion_r486034361



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
##########
@@ -2555,6 +2555,17 @@ class DataFrameSuite extends QueryTest
     val df = Seq(0.0 -> -0.0).toDF("pos", "neg")
     checkAnswer(df.select($"pos" > $"neg"), Row(false))
   }
+
+  test("SPARK-32816: aggregating multiple distinct DECIMAL columns") {
+    withTempView("test_table") {
+      spark.range(0, 100, 1, 1)
+        .selectExpr("id", "cast(id as decimal(9, 0)) as decimal_col")
+        .createOrReplaceTempView("test_table")

Review comment:
       can you follow https://github.com/apache/spark/pull/29673#discussion_r485977416 ? That's a better idea.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-689945509


   **[Test build #128480 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128480/testReport)** for PR 29673 at commit [`737f996`](https://github.com/apache/spark/commit/737f9965ca6e87e7c43a87e56da7f47a4e08e43e).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-688946105


   **[Test build #128394 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128394/testReport)** for PR 29673 at commit [`b1ce4b5`](https://github.com/apache/spark/commit/b1ce4b5fe12d32a867b281d48556f952932afd55).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan closed pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #29673:
URL: https://github.com/apache/spark/pull/29673


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-690894654






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-689354343


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128428/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-690036848






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] maropu commented on a change in pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29673:
URL: https://github.com/apache/spark/pull/29673#discussion_r486232651



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
##########
@@ -2555,6 +2555,16 @@ class DataFrameSuite extends QueryTest
     val df = Seq(0.0 -> -0.0).toDF("pos", "neg")
     checkAnswer(df.select($"pos" > $"neg"), Row(false))
   }
+
+  test("SPARK-32816: aggregating multiple distinct DECIMAL columns") {
+    checkAnswer(
+      sql(
+        s"""
+           |SELECT AVG(DISTINCT decimal_col), SUM(DISTINCT decimal_col)
+           |  FROM VALUES (CAST(1 AS DECIMAL(9, 0))) t(decimal_col)
+        """.stripMargin),
+      Row(1, 1))

Review comment:
       nit: the test `group-by.sql` looks enough for this issue, so could you remove this?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29673:
URL: https://github.com/apache/spark/pull/29673#discussion_r484856511



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -196,6 +195,8 @@ abstract class Optimizer(catalogManager: CatalogManager)
       EliminateSorts) :+
     Batch("Decimal Optimizations", fixedPoint,
       DecimalAggregates) :+
+    Batch("Distinct Aggregate Rewrite", Once,

Review comment:
       can we add a comment to say: this batch must be run after "Decimal Optimizations", as "Decimal Optimizations" may change the aggregate distinct column?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-690804618


   **[Test build #128540 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128540/testReport)** for PR 29673 at commit [`8510ff9`](https://github.com/apache/spark/commit/8510ff92177554d9e87bd2e30e6fa9e43f89b8e3).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-689353366


   **[Test build #128428 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128428/testReport)** for PR 29673 at commit [`4df4f7c`](https://github.com/apache/spark/commit/4df4f7c8ebc80bcf854fb26764338789ea13b319).
    * This patch **fails due to an unknown error code, -9**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-690804618






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] linhongliu-db commented on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
linhongliu-db commented on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-688801580


   @cloud-fan 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] TJX2014 commented on a change in pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
TJX2014 commented on a change in pull request #29673:
URL: https://github.com/apache/spark/pull/29673#discussion_r484739788



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
##########
@@ -2555,6 +2555,19 @@ class DataFrameSuite extends QueryTest
     val df = Seq(0.0 -> -0.0).toDF("pos", "neg")
     checkAnswer(df.select($"pos" > $"neg"), Row(false))
   }
+
+  test("SPARK-32816: aggregating multiple distinct DECIMAL columns") {
+    withTempPath { path =>
+      spark.range(0, 100, 1, 1)
+        .selectExpr("id", "cast(id as decimal(9, 0)) as decimal_col")
+        .write.mode("overwrite")
+        .parquet(path.getAbsolutePath)
+      spark.read.parquet(path.getAbsolutePath).createOrReplaceTempView("test_table")

Review comment:
       Seems we need not to write parquet.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-690893672


   **[Test build #128540 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128540/testReport)** for PR 29673 at commit [`8510ff9`](https://github.com/apache/spark/commit/8510ff92177554d9e87bd2e30e6fa9e43f89b8e3).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-690036848


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-690894654






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-690893672






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-690110156






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-690804618


   **[Test build #128540 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128540/testReport)** for PR 29673 at commit [`8510ff9`](https://github.com/apache/spark/commit/8510ff92177554d9e87bd2e30e6fa9e43f89b8e3).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-689263764


   **[Test build #128428 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128428/testReport)** for PR 29673 at commit [`4df4f7c`](https://github.com/apache/spark/commit/4df4f7c8ebc80bcf854fb26764338789ea13b319).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-689354327


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-688948058






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-690804618


   **[Test build #128540 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128540/testReport)** for PR 29673 at commit [`8510ff9`](https://github.com/apache/spark/commit/8510ff92177554d9e87bd2e30e6fa9e43f89b8e3).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-690415141


   **[Test build #128504 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128504/testReport)** for PR 29673 at commit [`f2111df`](https://github.com/apache/spark/commit/f2111dfc4e22b11de82e6202e7f53992d492eef2).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29673:
URL: https://github.com/apache/spark/pull/29673#issuecomment-689264278






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gatorsmile commented on a change in pull request #29673: [SPARK-32816][SQL] Fix analyzer bug when aggregating multiple distinct DECIMAL columns

Posted by GitBox <gi...@apache.org>.
gatorsmile commented on a change in pull request #29673:
URL: https://github.com/apache/spark/pull/29673#discussion_r485010238



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -196,6 +195,8 @@ abstract class Optimizer(catalogManager: CatalogManager)
       EliminateSorts) :+
     Batch("Decimal Optimizations", fixedPoint,
       DecimalAggregates) :+
+    Batch("Distinct Aggregate Rewrite", Once,

Review comment:
       In the comment, we can also point out RewriteDistinctAggregates should be executed after the rule DecimalAggregates




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org