You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by dreamquster <gi...@git.apache.org> on 2015/04/04 10:19:24 UTC
[GitHub] spark pull request: [SQL] SPARK-6548: Adding stddev to DataFrame f...
GitHub user dreamquster opened a pull request:
https://github.com/apache/spark/pull/5357
[SQL] SPARK-6548: Adding stddev to DataFrame functions
remerge SPARK-6548
https://github.com/apache/spark/pull/5228
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dreamquster/spark spark-stddev
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5357.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5357
----
commit 58dec463b8280b425b5534bdeb28b013ae02eec4
Author: dreamquster <dr...@163.com>
Date: 2015-04-04T08:13:56Z
[SQL] SPARK-6548: Adding stddev to DataFrame functions
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-6548: Adding stddev to DataFrame f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5357#issuecomment-94379206
[Test build #30577 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30577/consoleFull) for PR 5357 at commit [`c61d535`](https://github.com/apache/spark/commit/c61d53500f37ee4b73c41e547532ff25917bbda6).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class StdDeviation(child: Expression)`
* `case class StdDeviationFunction(expr: Expression, base: AggregateExpression)`
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-6548: Adding stddev to DataFrame f...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/5357#issuecomment-93868139
/cc @rxin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-6548: Adding stddev to DataFrame f...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5357#discussion_r28609709
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -344,6 +344,20 @@ object functions {
*/
def abs(e: Column): Column = Abs(e.expr)
+ /**
+ * Aggregate func:compute the stddev of the specific column.
+ *
+ * @group agg_funcs
+ */
+ def stddev(columnName: String): Column = stddev(Column(columnName))
--- End diff --
ah never mind -- was lookin at the wrong one
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-6548: Adding stddev to DataFrame f...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5357#discussion_r28609646
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -344,6 +344,20 @@ object functions {
*/
def abs(e: Column): Column = Abs(e.expr)
+ /**
+ * Aggregate func:compute the stddev of the specific column.
+ *
+ * @group agg_funcs
+ */
+ def stddev(columnName: String): Column = stddev(Column(columnName))
--- End diff --
to be consistent with other functiosn, can we just use "e"?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-6548: Adding stddev to DataFrame f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5357#issuecomment-93826615
[Test build #30432 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30432/consoleFull) for PR 5357 at commit [`a012fc3`](https://github.com/apache/spark/commit/a012fc370fb1884ca95d9055c52fb4bd2374e610).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-6548: Adding stddev to DataFrame f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5357#issuecomment-94383918
[Test build #30578 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30578/consoleFull) for PR 5357 at commit [`1052218`](https://github.com/apache/spark/commit/10522181d114744581cc6c3f794c41bfe740c642).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-6548: Adding stddev to DataFrame f...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/5357#issuecomment-94024412
@dreamquster thanks for doing this -- I think we also need to add the Python version of it here as well.
@yhuai can you review the stddev expression code itself?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-6548: Adding stddev to DataFrame f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5357#issuecomment-93845448
[Test build #30432 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30432/consoleFull) for PR 5357 at commit [`a012fc3`](https://github.com/apache/spark/commit/a012fc370fb1884ca95d9055c52fb4bd2374e610).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class StdDeviation(child: Expression)`
* `case class StdDeviationFunction(expr: Expression, base: AggregateExpression)`
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-6548: Adding stddev to DataFrame f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5357#issuecomment-93655610
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-6548: Adding stddev to DataFrame f...
Posted by dreamquster <gi...@git.apache.org>.
GitHub user dreamquster reopened a pull request:
https://github.com/apache/spark/pull/5357
[SQL] SPARK-6548: Adding stddev to DataFrame functions
remerge SPARK-6548
https://github.com/apache/spark/pull/5228
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dreamquster/spark spark-stddev
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5357.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5357
----
commit 58dec463b8280b425b5534bdeb28b013ae02eec4
Author: dreamquster <dr...@163.com>
Date: 2015-04-04T08:13:56Z
[SQL] SPARK-6548: Adding stddev to DataFrame functions
commit a012fc370fb1884ca95d9055c52fb4bd2374e610
Author: dreamquster <dr...@163.com>
Date: 2015-04-07T05:47:58Z
Update aggregates.scala
apply to scalastyle line length
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-6548: Adding stddev to DataFrame f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5357#issuecomment-94413601
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30578/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-6548: Adding stddev to DataFrame f...
Posted by dreamquster <gi...@git.apache.org>.
Github user dreamquster commented on the pull request:
https://github.com/apache/spark/pull/5357#issuecomment-89980788
What's wrong?Who can verify it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-6548: Adding stddev to DataFrame f...
Posted by dreamquster <gi...@git.apache.org>.
Github user dreamquster closed the pull request at:
https://github.com/apache/spark/pull/5357
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-6548: Adding stddev to DataFrame f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5357#issuecomment-93845459
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30432/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-6548: Adding stddev to DataFrame f...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/5357#issuecomment-93825495
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-6548: Adding stddev to DataFrame f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5357#issuecomment-89527728
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-6548: Adding stddev to DataFrame f...
Posted by dreamquster <gi...@git.apache.org>.
Github user dreamquster commented on the pull request:
https://github.com/apache/spark/pull/5357#issuecomment-95611837
@yhuai . Is this comment OK?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-6548: Adding stddev to DataFrame f...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/5357#issuecomment-94240966
@dreamquster Can you add comments in your code to explain:
1. Is it sample standard deviation or population standard deviation?
2. What is the algorithm used to calculate the standard deviation (how to calculate the partial result and how to calculate the final results)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-6548: Adding stddev to DataFrame f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5357#issuecomment-91108787
[Test build #29915 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29915/consoleFull) for PR 5357 at commit [`a012fc3`](https://github.com/apache/spark/commit/a012fc370fb1884ca95d9055c52fb4bd2374e610).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class StdDeviation(child: Expression)`
* `case class StdDeviationFunction(expr: Expression, base: AggregateExpression)`
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-6548: Adding stddev to DataFrame f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5357#issuecomment-94379026
[Test build #30577 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30577/consoleFull) for PR 5357 at commit [`c61d535`](https://github.com/apache/spark/commit/c61d53500f37ee4b73c41e547532ff25917bbda6).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-6548: Adding stddev to DataFrame f...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5357#discussion_r28609608
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala ---
@@ -334,6 +334,24 @@ class DataFrameSuite extends QueryTest {
Row(new java.math.BigDecimal(2.0), new java.math.BigDecimal(6)) :: Nil)
}
+ test("stddev") {
+ val testData2ADev = math.sqrt(4/6.0)
+ checkAnswer(
+ testData2.agg(stddev('a)),
+ Row(testData2ADev))
+
+ checkAnswer(
+ testData2.agg(stddev(testData2("a"))),
+ Row(testData2ADev))
+
+ checkAnswer(
+ decimalData.agg(stddev('a), sumDistinct('a)), // non-partial
+ Row(testData2ADev, new java.math.BigDecimal(6)) :: Nil)
+
--- End diff --
can you remove the excessive white space here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-6548: Adding stddev to DataFrame f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5357#issuecomment-94413583
[Test build #30578 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30578/consoleFull) for PR 5357 at commit [`1052218`](https://github.com/apache/spark/commit/10522181d114744581cc6c3f794c41bfe740c642).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class StdDeviation(child: Expression)`
* `case class StdDeviationFunction(expr: Expression, base: AggregateExpression)`
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-6548: Adding stddev to DataFrame f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5357#issuecomment-94379211
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30577/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-6548: Adding stddev to DataFrame f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5357#issuecomment-91108793
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29915/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-6548: Adding stddev to DataFrame f...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5357#discussion_r28609928
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregates.scala ---
@@ -616,6 +617,115 @@ case class SumFunction(expr: Expression, base: AggregateExpression) extends Aggr
}
}
+case class StdDeviation(child: Expression)
+ extends PartialAggregate with trees.UnaryNode[Expression]{
+ override def nullable: Boolean = true
+
+ override def dataType: DataType = child.dataType match {
+ case DecimalType.Fixed(precision, scale) =>
+ DecimalType(precision + 4, scale + 4) // Add 4 digits after decimal point, like Hive
+ case DecimalType.Unlimited =>
+ DecimalType.Unlimited
+ case _ =>
+ DoubleType
+ }
+
+ override def asPartial: SplitEvaluation = {
+ val (seqPartialData, castStddev) = child.dataType match {
+ case DecimalType.Fixed(_, _) =>
+ // Turn the child to unlimited decimals for calculation, before going back to fixed
+ val (seqPartialData, stddev) = calcPartialStddev(DecimalType.Unlimited)
+ (seqPartialData, Cast(stddev, dataType))
+ case _ =>
+ calcPartialStddev(dataType)
+ }
+
+ SplitEvaluation(castStddev, seqPartialData)
+ }
+
+ /**
+ *
+ * @param calcType data type during calcuation
+ * @return seqPartialData is data for partialEvaluations
+ */
+ protected def calcPartialStddev(calcType: DataType): (List[Alias], Expression) = {
+ val castedChild = Cast(child, calcType)
+ val partialSquredSum = Alias(Sum(Multiply(castedChild, castedChild)), "PartialSquredSum")()
+ val partialSum = Alias(Sum(castedChild), "PartialSum")()
+ val partialCount = Alias(Count(child), "PartialCount")()
+ val seqPartialData = partialCount :: partialSum :: partialSquredSum :: Nil
+
+ val castedSquredSum = Cast(Sum(partialSquredSum.toAttribute), calcType)
+ val castedSum = Cast(Sum(partialSum.toAttribute), calcType)
+ val castedCount = Cast(Sum(partialCount.toAttribute), calcType)
+ val castedAvg = Divide(castedSum, castedCount)
+ val stddev = Sqrt(Divide(
+ Subtract(castedSquredSum, Multiply(castedSum, castedAvg)),
+ castedCount));
+ (seqPartialData, stddev)
+ }
+
+ override def toString: String = s"STDDEV($child)"
+
+ override def newInstance(): StdDeviationFunction = new StdDeviationFunction(child, this)
+}
+
+case class StdDeviationFunction(expr: Expression, base: AggregateExpression)
+ extends AggregateFunction {
+ def this() = this(null, null) // /Required for serialization.
+
+ private val calcType =
+ expr.dataType match {
+ case DecimalType.Fixed(precision, scale) =>
+ DecimalType(precision + 4, scale + 4) // Add 4 digits after decimal point, like Hive
+ case DecimalType.Unlimited =>
+ DecimalType.Unlimited
+ case _ =>
+ DoubleType
+ }
+
+ private val zero = Cast(Literal(0), calcType)
+
+ private var count: Long = _
+
--- End diff --
remove the extra blank line here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-6548: Adding stddev to DataFrame f...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5357#discussion_r28609773
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -344,6 +344,20 @@ object functions {
*/
def abs(e: Column): Column = Abs(e.expr)
+ /**
+ * Aggregate func:compute the stddev of the specific column.
+ *
+ * @group agg_funcs
+ */
+ def stddev(columnName: String): Column = stddev(Column(columnName))
+
+ /**
+ * Aggregate func:compute the stddev of the specific column
+ *
+ * @group agg_funcs
+ */
+ def stddev(e: Column): Column = StdDeviation(e.expr)
--- End diff --
can you move these two functions to the aggregate function section, and change the comment to say
```
Aggregate function: Computes the standard deviation of the specific column.
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org