You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by yh...@apache.org on 2015/11/02 22:51:58 UTC

spark git commit: [SPARK-11371] Make "mean" an alias for "avg" operator

Repository: spark
Updated Branches:
  refs/heads/master 33ae7a35d -> db11ee5e5


[SPARK-11371] Make "mean" an alias for "avg" operator

>From Reynold in the thread 'Exception when using some aggregate operators' (http://search-hadoop.com/m/q3RTt0xFr22nXB4/):

I don't think these are bugs. The SQL standard for average is "avg", not "mean". Similarly, a distinct count is supposed to be written as "count(distinct col)", not "countDistinct(col)".
We can, however, make "mean" an alias for "avg" to improve compatibility between DataFrame and SQL.

Author: tedyu <yu...@gmail.com>

Closes #9332 from ted-yu/master.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/db11ee5e
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/db11ee5e
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/db11ee5e

Branch: refs/heads/master
Commit: db11ee5e56e5fac59895c772a9a87c5ac86888ef
Parents: 33ae7a3
Author: tedyu <yu...@gmail.com>
Authored: Mon Nov 2 13:51:53 2015 -0800
Committer: Yin Huai <yh...@databricks.com>
Committed: Mon Nov 2 13:51:53 2015 -0800

----------------------------------------------------------------------
 .../spark/sql/catalyst/analysis/FunctionRegistry.scala      | 1 +
 .../spark/sql/hive/execution/AggregationQuerySuite.scala    | 9 +++++++++
 2 files changed, 10 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/db11ee5e/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
----------------------------------------------------------------------
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
index 5f3ec74..24c1a7b 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
@@ -185,6 +185,7 @@ object FunctionRegistry {
     expression[Last]("last"),
     expression[Last]("last_value"),
     expression[Max]("max"),
+    expression[Average]("mean"),
     expression[Min]("min"),
     expression[Stddev]("stddev"),
     expression[StddevPop]("stddev_pop"),

http://git-wip-us.apache.org/repos/asf/spark/blob/db11ee5e/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/AggregationQuerySuite.scala
----------------------------------------------------------------------
diff --git a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/AggregationQuerySuite.scala b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/AggregationQuerySuite.scala
index 0cf0e0a..74061db 100644
--- a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/AggregationQuerySuite.scala
+++ b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/AggregationQuerySuite.scala
@@ -301,6 +301,15 @@ abstract class AggregationQuerySuite extends QueryTest with SQLTestUtils with Te
     checkAnswer(
       sqlContext.sql(
         """
+          |SELECT key, mean(value)
+          |FROM agg1
+          |GROUP BY key
+        """.stripMargin),
+      Row(1, 20.0) :: Row(2, -0.5) :: Row(3, null) :: Row(null, 10.0) :: Nil)
+
+    checkAnswer(
+      sqlContext.sql(
+        """
           |SELECT avg(value), key
           |FROM agg1
           |GROUP BY key


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org