You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by jiangxb1987 <gi...@git.apache.org> on 2016/10/28 11:14:19 UTC

[GitHub] spark pull request #15672: [SPARK-18148][SQL] Misleading Error Message for A...

GitHub user jiangxb1987 opened a pull request:

    https://github.com/apache/spark/pull/15672

    [SPARK-18148][SQL] Misleading Error Message for Aggregation Without Window/GroupBy

    ## What changes were proposed in this pull request?
    
    Aggregation Without Window/GroupBy expressions will fail in `checkAnalysis`, the error message is a bit misleading, we should generate a more specific error message for this case.
    
    For example,
    ```
    spark.read.load("/some-data")
      .withColumn("date_dt", to_date($"date"))
      .withColumn("year", year($"date_dt"))
      .withColumn("week", weekofyear($"date_dt"))
      .withColumn("user_count", count($"userId"))
      .withColumn("daily_max_in_week", max($"user_count").over(weeklyWindow))
    )
    ```
    creates the following output:
    ```
    org.apache.spark.sql.AnalysisException: expression '`randomColumn`' is neither present in the group by, nor is it an aggregate function. Add to group by or wrap in first() (or first_value) if you don't care which value you get.;
    ```
    In the error message above, `randomColumn` doesn't appear in the query(acturally it's added by function `withColumn`), so the message is not enough for the user to address the problem.
    
    ## How was this patch tested?
    
    Manually test
    
    Before:
    ```
    scala> spark.sql("select col, count(col) from tbl")
    org.apache.spark.sql.AnalysisException: expression 'tbl.`col`' is neither present in the group by, nor is it an aggregate function. Add to group by or wrap in first() (or first_value) if you don't care which value you get.;;
    ```
    
    After:
    ```
    scala> spark.sql("select col, count(col) from tbl")
    org.apache.spark.sql.AnalysisException: grouping expressions sequence is empty, and 'tbl.`col`' is not an aggregate function. Wrap '(count(col#231L) AS count(col)#239L)' in windowing function(s) or wrap 'tbl.`col`' in first() (or first_value) if you don't care which value you get.;;
    ```

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/jiangxb1987/spark groupBy-empty

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/15672.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #15672
    
----
commit 494ef0931b6c8e0a4c0804f88a21b7dce7870750
Author: jiangxingbo <ji...@gmail.com>
Date:   2016-10-28T10:48:21Z

    improve error message when checkAnalysis fail on Aggregate operator.

commit 350b7a335038adbe8a5a766d7a77fa17db0dbdeb
Author: jiangxingbo <ji...@gmail.com>
Date:   2016-10-28T11:12:04Z

    better format.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #15672: [SPARK-18148][SQL] Misleading Error Message for Aggregat...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/15672
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #15672: [SPARK-18148][SQL] Misleading Error Message for Aggregat...

Posted by jiangxb1987 <gi...@git.apache.org>.

Github user jiangxb1987 commented on the issue:

    https://github.com/apache/spark/pull/15672
  
    retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #15672: [SPARK-18148][SQL] Misleading Error Message for Aggregat...

Posted by rxin <gi...@git.apache.org>.

Github user rxin commented on the issue:

    https://github.com/apache/spark/pull/15672
  
    Thanks - merging in master.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #15672: [SPARK-18148][SQL] Misleading Error Message for Aggregat...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/15672
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #15672: [SPARK-18148][SQL] Misleading Error Message for Aggregat...

Posted by rxin <gi...@git.apache.org>.

Github user rxin commented on the issue:

    https://github.com/apache/spark/pull/15672
  
    Is this confusing msg reproducible using SQL queries (not dataframe APIs)? If yes, perhaps we can add a test in SQLQueryTestSuite.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #15672: [SPARK-18148][SQL] Misleading Error Message for Aggregat...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/15672
  
    **[Test build #67889 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67889/consoleFull)** for PR 15672 at commit [`43f8fa6`](https://github.com/apache/spark/commit/43f8fa6d642fb5eae3de13a5f88d592d6dcd4483).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #15672: [SPARK-18148][SQL] Misleading Error Message for Aggregat...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/15672
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67889/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #15672: [SPARK-18148][SQL] Misleading Error Message for Aggregat...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/15672
  
    **[Test build #67803 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67803/consoleFull)** for PR 15672 at commit [`033f43f`](https://github.com/apache/spark/commit/033f43fdf7835c24423c456dd7f26fe0c12a766a).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #15672: [SPARK-18148][SQL] Misleading Error Message for Aggregat...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/15672
  
    **[Test build #67889 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67889/consoleFull)** for PR 15672 at commit [`43f8fa6`](https://github.com/apache/spark/commit/43f8fa6d642fb5eae3de13a5f88d592d6dcd4483).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #15672: [SPARK-18148][SQL] Misleading Error Message for Aggregat...

Posted by jiangxb1987 <gi...@git.apache.org>.

Github user jiangxb1987 commented on the issue:

    https://github.com/apache/spark/pull/15672
  
    @rxin I've updated test sqls to use one dataset. Thank you!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #15672: [SPARK-18148][SQL] Misleading Error Message for Aggregat...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/15672
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org