You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by viirya <gi...@git.apache.org> on 2015/09/03 18:50:57 UTC
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
GitHub user viirya opened a pull request:
https://github.com/apache/spark/pull/8587
[SPARK-9298][SQL] Add pearson correlation aggregation function
JIRA: https://issues.apache.org/jira/browse/SPARK-9298
This patch adds pearson correlation aggregation function based on `AggregateExpression2`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/viirya/spark-1 corr_aggregation
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/8587.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #8587
----
commit cb34a95e3dea152250b6409827fc869bd7fae407
Author: Liang-Chi Hsieh <vi...@appier.com>
Date: 2015-09-03T16:47:21Z
Add corr aggregate function.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-151462149
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44418/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-150070729
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44108/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-137509676
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152298716
**[Test build #44616 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44616/consoleFull)** for PR 8587 at commit [`2f7b864`](https://github.com/apache/spark/commit/2f7b8641ec0c7c7f56b0df2524064a2c47e13288).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `case class Corr(`\n * `case class Corr(left: Expression, right: Expression)`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152696993
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152689380
**[Test build #44726 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44726/consoleFull)** for PR 8587 at commit [`2de76b4`](https://github.com/apache/spark/commit/2de76b444456bc7e751fa9ccb85a6e8f0662ff76).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152680566
oh, that failure is going to fixed in #9387 now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-150060626
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152473997
Failure caused by: 0.6633880657639323 != 0.6633880657639322
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-150297842
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/8587#discussion_r43160606
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/functions.scala ---
@@ -524,6 +525,116 @@ case class Sum(child: Expression) extends DeclarativeAggregate {
override val evaluateExpression = Cast(currentSum, resultType)
}
+case class Corr(
+ left: Expression,
+ right: Expression,
+ mutableAggBufferOffset: Int = 0,
+ inputAggBufferOffset: Int = 0)
+ extends ImperativeAggregate {
+
+ def children: Seq[Expression] = Seq(left, right)
+
+ def nullable: Boolean = false
+
+ def dataType: DataType = DoubleType
+
+ def inputTypes: Seq[AbstractDataType] = Seq(DoubleType)
+
+ def aggBufferSchema: StructType = StructType.fromAttributes(aggBufferAttributes)
+
+ def inputAggBufferAttributes: Seq[AttributeReference] = aggBufferAttributes.map(_.newInstance())
+
+ val aggBufferAttributes: Seq[AttributeReference] = Seq(
+ AttributeReference("xAvg", DoubleType)(),
+ AttributeReference("yAvg", DoubleType)(),
+ AttributeReference("Ck", DoubleType)(),
+ AttributeReference("MkX", DoubleType)(),
+ AttributeReference("MkY", DoubleType)(),
+ AttributeReference("count", LongType)())
+
+ override def withNewMutableAggBufferOffset(newMutableAggBufferOffset: Int): ImperativeAggregate =
+ copy(mutableAggBufferOffset = newMutableAggBufferOffset)
+
+ override def withNewInputAggBufferOffset(newInputAggBufferOffset: Int): ImperativeAggregate =
+ copy(inputAggBufferOffset = newInputAggBufferOffset)
+
+ override def initialize(buffer: MutableRow): Unit = {
+ (0 until 5).map(idx => buffer.setDouble(mutableAggBufferOffset + idx, 0.0))
+ buffer.setLong(mutableAggBufferOffset + 5, 0L)
+ }
+
+ override def update(buffer: MutableRow, input: InternalRow): Unit = {
+ val x = left.eval(input).asInstanceOf[Double]
+ val y = right.eval(input).asInstanceOf[Double]
+
+ var xAvg = buffer.getDouble(mutableAggBufferOffset)
+ var yAvg = buffer.getDouble(mutableAggBufferOffset + 1)
--- End diff --
I mean cache `mutableAggBufferOffset + 2` to a `private[this] val` to avoid making addition for every record.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152261568
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152680213
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152298823
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44616/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/8587#discussion_r43090251
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/functions.scala ---
@@ -524,6 +525,116 @@ case class Sum(child: Expression) extends DeclarativeAggregate {
override val evaluateExpression = Cast(currentSum, resultType)
}
+case class Corr(
--- End diff --
* Please add ScalaDoc and document the behavior for `null` and `NaN` values.
* Provide a link to the wikipedia page that contains the update formula.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152473051
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44671/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-151422462
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-151422431
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152453498
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152681982
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152453486
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-150061984
**[Test build #44106 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44106/consoleFull)** for PR 8587 at commit [`d10afbe`](https://github.com/apache/spark/commit/d10afbefddacb8f779414948fc2217bdc6a9e791).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `case class First(child: Expression, ignoreNullsExpr: Expression) extends DeclarativeAggregate `\n * `case class Last(child: Expression, ignoreNullsExpr: Expression) extends DeclarativeAggregate `\n * `case class Corr(`\n * `case class First(`\n * `case class FirstFunction(`\n * `case class Last(`\n * `case class LastFunction(`\n * `case class Corr(`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-138124327
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/8587#discussion_r43451783
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/functions.scala ---
@@ -524,6 +525,133 @@ case class Sum(child: Expression) extends DeclarativeAggregate {
override val evaluateExpression = Cast(currentSum, resultType)
}
+/**
+ * Compute Pearson correlation between two expressions.
+ * When applied on empty data (i.e., count is zero), it returns NaN.
+ *
+ * Definition of Pearson correlation can be found at
+ * http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient
+ *
+ * @param left one of the expressions to compute correlation with.
+ * @param right another expression to compute correlation with.
+ */
+case class Corr(
+ left: Expression,
+ right: Expression,
+ mutableAggBufferOffset: Int = 0,
+ inputAggBufferOffset: Int = 0)
+ extends ImperativeAggregate {
+
+ def children: Seq[Expression] = Seq(left, right)
+
+ def nullable: Boolean = false
+
+ def dataType: DataType = DoubleType
+
+ def inputTypes: Seq[AbstractDataType] = Seq(DoubleType)
+
+ def aggBufferSchema: StructType = StructType.fromAttributes(aggBufferAttributes)
+
+ def inputAggBufferAttributes: Seq[AttributeReference] = aggBufferAttributes.map(_.newInstance())
+
+ val aggBufferAttributes: Seq[AttributeReference] = Seq(
+ AttributeReference("xAvg", DoubleType)(),
+ AttributeReference("yAvg", DoubleType)(),
+ AttributeReference("Ck", DoubleType)(),
+ AttributeReference("MkX", DoubleType)(),
+ AttributeReference("MkY", DoubleType)(),
+ AttributeReference("count", LongType)())
+
+ override def withNewMutableAggBufferOffset(newMutableAggBufferOffset: Int): ImperativeAggregate =
+ copy(mutableAggBufferOffset = newMutableAggBufferOffset)
+
+ override def withNewInputAggBufferOffset(newInputAggBufferOffset: Int): ImperativeAggregate =
+ copy(inputAggBufferOffset = newInputAggBufferOffset)
+
+ override def initialize(buffer: MutableRow): Unit = {
+ (0 until 5).map(idx => buffer.setDouble(mutableAggBufferOffset + idx, 0.0))
+ buffer.setLong(mutableAggBufferOffset + 5, 0L)
+ }
+
+ override def update(buffer: MutableRow, input: InternalRow): Unit = {
+ val x = left.eval(input).asInstanceOf[Double]
+ val y = right.eval(input).asInstanceOf[Double]
+
+ var xAvg = buffer.getDouble(mutableAggBufferOffset)
+ var yAvg = buffer.getDouble(mutableAggBufferOffset + 1)
+ var Ck = buffer.getDouble(mutableAggBufferOffset + 2)
+ var MkX = buffer.getDouble(mutableAggBufferOffset + 3)
+ var MkY = buffer.getDouble(mutableAggBufferOffset + 4)
+ var count = buffer.getLong(mutableAggBufferOffset + 5)
+
+ val deltaX = x - xAvg
+ val deltaY = y - yAvg
+ count += 1
+ xAvg += deltaX / count
+ yAvg += deltaY / count
+ Ck += deltaX * (y - yAvg)
+ MkX += deltaX * (x - xAvg)
+ MkY += deltaY * (y - yAvg)
+
+ buffer.setDouble(mutableAggBufferOffset, xAvg)
+ buffer.setDouble(mutableAggBufferOffset + 1, yAvg)
+ buffer.setDouble(mutableAggBufferOffset + 2, Ck)
+ buffer.setDouble(mutableAggBufferOffset + 3, MkX)
+ buffer.setDouble(mutableAggBufferOffset + 4, MkY)
+ buffer.setLong(mutableAggBufferOffset + 5, count)
+ }
+
+ // Merge counters from other partitions. Formula can be found at:
+ // http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance
+ override def merge(buffer1: MutableRow, buffer2: InternalRow): Unit = {
+ val count2 = buffer2.getLong(inputAggBufferOffset + 5)
+
+ if (count2 > 0) {
--- End diff --
No need to add comment for it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152372753
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152604997
**[Test build #44690 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44690/consoleFull)** for PR 8587 at commit [`2de76b4`](https://github.com/apache/spark/commit/2de76b444456bc7e751fa9ccb85a6e8f0662ff76).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `case class Corr(`\n * `case class Corr(left: Expression, right: Expression)`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/8587#discussion_r43090262
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/functions.scala ---
@@ -524,6 +525,116 @@ case class Sum(child: Expression) extends DeclarativeAggregate {
override val evaluateExpression = Cast(currentSum, resultType)
}
+case class Corr(
+ left: Expression,
+ right: Expression,
+ mutableAggBufferOffset: Int = 0,
+ inputAggBufferOffset: Int = 0)
+ extends ImperativeAggregate {
+
+ def children: Seq[Expression] = Seq(left, right)
+
+ def nullable: Boolean = false
+
+ def dataType: DataType = DoubleType
+
+ def inputTypes: Seq[AbstractDataType] = Seq(DoubleType)
+
+ def aggBufferSchema: StructType = StructType.fromAttributes(aggBufferAttributes)
+
+ def inputAggBufferAttributes: Seq[AttributeReference] = aggBufferAttributes.map(_.newInstance())
+
+ val aggBufferAttributes: Seq[AttributeReference] = Seq(
+ AttributeReference("xAvg", DoubleType)(),
+ AttributeReference("yAvg", DoubleType)(),
+ AttributeReference("Ck", DoubleType)(),
+ AttributeReference("MkX", DoubleType)(),
+ AttributeReference("MkY", DoubleType)(),
+ AttributeReference("count", LongType)())
+
+ override def withNewMutableAggBufferOffset(newMutableAggBufferOffset: Int): ImperativeAggregate =
+ copy(mutableAggBufferOffset = newMutableAggBufferOffset)
+
+ override def withNewInputAggBufferOffset(newInputAggBufferOffset: Int): ImperativeAggregate =
+ copy(inputAggBufferOffset = newInputAggBufferOffset)
+
+ override def initialize(buffer: MutableRow): Unit = {
+ (0 until 5).map(idx => buffer.setDouble(mutableAggBufferOffset + idx, 0.0))
+ buffer.setLong(mutableAggBufferOffset + 5, 0L)
+ }
+
+ override def update(buffer: MutableRow, input: InternalRow): Unit = {
+ val x = left.eval(input).asInstanceOf[Double]
+ val y = right.eval(input).asInstanceOf[Double]
+
+ var xAvg = buffer.getDouble(mutableAggBufferOffset)
+ var yAvg = buffer.getDouble(mutableAggBufferOffset + 1)
--- End diff --
It might be faster to cache the offsets as `private[this] var`s.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-137510618
[Test build #41976 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41976/consoleFull) for PR 8587 at commit [`cb34a95`](https://github.com/apache/spark/commit/cb34a95e3dea152250b6409827fc869bd7fae407).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152444982
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-150056525
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-150297843
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44154/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-138124328
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42076/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-150057635
**[Test build #44103 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44103/consoleFull)** for PR 8587 at commit [`1505cd2`](https://github.com/apache/spark/commit/1505cd23a8e0cbd1b26da499f53fd6416ac96ca8).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-151401142
LGTM except inline comments. Ping @yhuai @rxin for another pass on SQL side.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152596813
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/8587#discussion_r43094591
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/functions.scala ---
@@ -524,6 +525,116 @@ case class Sum(child: Expression) extends DeclarativeAggregate {
override val evaluateExpression = Cast(currentSum, resultType)
}
+case class Corr(
+ left: Expression,
+ right: Expression,
+ mutableAggBufferOffset: Int = 0,
+ inputAggBufferOffset: Int = 0)
+ extends ImperativeAggregate {
+
+ def children: Seq[Expression] = Seq(left, right)
+
+ def nullable: Boolean = false
+
+ def dataType: DataType = DoubleType
+
+ def inputTypes: Seq[AbstractDataType] = Seq(DoubleType)
+
+ def aggBufferSchema: StructType = StructType.fromAttributes(aggBufferAttributes)
+
+ def inputAggBufferAttributes: Seq[AttributeReference] = aggBufferAttributes.map(_.newInstance())
+
+ val aggBufferAttributes: Seq[AttributeReference] = Seq(
+ AttributeReference("xAvg", DoubleType)(),
+ AttributeReference("yAvg", DoubleType)(),
+ AttributeReference("Ck", DoubleType)(),
+ AttributeReference("MkX", DoubleType)(),
+ AttributeReference("MkY", DoubleType)(),
+ AttributeReference("count", LongType)())
+
+ override def withNewMutableAggBufferOffset(newMutableAggBufferOffset: Int): ImperativeAggregate =
+ copy(mutableAggBufferOffset = newMutableAggBufferOffset)
+
+ override def withNewInputAggBufferOffset(newInputAggBufferOffset: Int): ImperativeAggregate =
+ copy(inputAggBufferOffset = newInputAggBufferOffset)
+
+ override def initialize(buffer: MutableRow): Unit = {
+ (0 until 5).map(idx => buffer.setDouble(mutableAggBufferOffset + idx, 0.0))
+ buffer.setLong(mutableAggBufferOffset + 5, 0L)
+ }
+
+ override def update(buffer: MutableRow, input: InternalRow): Unit = {
+ val x = left.eval(input).asInstanceOf[Double]
+ val y = right.eval(input).asInstanceOf[Double]
+
+ var xAvg = buffer.getDouble(mutableAggBufferOffset)
+ var yAvg = buffer.getDouble(mutableAggBufferOffset + 1)
+ var Ck = buffer.getDouble(mutableAggBufferOffset + 2)
+ var MkX = buffer.getDouble(mutableAggBufferOffset + 3)
+ var MkY = buffer.getDouble(mutableAggBufferOffset + 4)
+ var count = buffer.getLong(mutableAggBufferOffset + 5)
+
+ val deltaX = x - xAvg
+ val deltaY = y - yAvg
+ count += 1
+ xAvg += deltaX / count
+ yAvg += deltaY / count
+ Ck += deltaX * (y - yAvg)
+ MkX += deltaX * (x - xAvg)
+ MkY += deltaY * (y - yAvg)
+
+ buffer.setDouble(mutableAggBufferOffset, xAvg)
+ buffer.setDouble(mutableAggBufferOffset + 1, yAvg)
+ buffer.setDouble(mutableAggBufferOffset + 2, Ck)
+ buffer.setDouble(mutableAggBufferOffset + 3, MkX)
+ buffer.setDouble(mutableAggBufferOffset + 4, MkY)
+ buffer.setLong(mutableAggBufferOffset + 5, count)
+ }
+
+ override def merge(buffer1: MutableRow, buffer2: InternalRow): Unit = {
+ val count2 = buffer2.getLong(inputAggBufferOffset + 5)
+
+ if (count2 > 0) {
+ var xAvg = buffer1.getDouble(mutableAggBufferOffset)
+ var yAvg = buffer1.getDouble(mutableAggBufferOffset + 1)
+ var Ck = buffer1.getDouble(mutableAggBufferOffset + 2)
+ var MkX = buffer1.getDouble(mutableAggBufferOffset + 3)
+ var MkY = buffer1.getDouble(mutableAggBufferOffset + 4)
+ var count = buffer1.getLong(mutableAggBufferOffset + 5)
+
+ val xAvg2 = buffer2.getDouble(inputAggBufferOffset)
+ val yAvg2 = buffer2.getDouble(inputAggBufferOffset + 1)
+ val Ck2 = buffer2.getDouble(inputAggBufferOffset + 2)
+ val MkX2 = buffer2.getDouble(inputAggBufferOffset + 3)
+ val MkY2 = buffer2.getDouble(inputAggBufferOffset + 4)
+
+ val totalCount = count + count2
+ val deltaX = xAvg - xAvg2
+ val deltaY = yAvg - yAvg2
+ Ck += Ck2 + deltaX * deltaY * count / totalCount * count2
+ xAvg = (xAvg * count + xAvg2 * count2) / totalCount
+ yAvg = (yAvg * count + yAvg2 * count2) / totalCount
+ MkX += MkX2 + deltaX * deltaX * count / totalCount * count2
+ MkY += MkY2 + deltaY * deltaY * count / totalCount * count2
+ count = totalCount
+
+ buffer1.setDouble(mutableAggBufferOffset, xAvg)
+ buffer1.setDouble(mutableAggBufferOffset + 1, yAvg)
+ buffer1.setDouble(mutableAggBufferOffset + 2, Ck)
+ buffer1.setDouble(mutableAggBufferOffset + 3, MkX)
+ buffer1.setDouble(mutableAggBufferOffset + 4, MkY)
+ buffer1.setLong(mutableAggBufferOffset + 5, count)
+ }
+ }
+
+ override def eval(buffer: InternalRow): Any = {
+ val Ck = buffer.getDouble(mutableAggBufferOffset + 2)
+ val MkX = buffer.getDouble(mutableAggBufferOffset + 3)
+ val MkY = buffer.getDouble(mutableAggBufferOffset + 4)
+ Ck / math.sqrt(MkX * MkY)
--- End diff --
Reasonable. I will let it return `NaN`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-137935670
Is it possible to do this with AlgebraicAggregate so it can be codegened?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/8587
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152444984
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44659/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-150063136
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152478815
@yhuai How do you think? Should we modify `HiveComparisonTest` to allow this kind of error?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152681983
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44719/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/8587#discussion_r43453305
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/functions.scala ---
@@ -524,6 +525,116 @@ case class Sum(child: Expression) extends DeclarativeAggregate {
override val evaluateExpression = Cast(currentSum, resultType)
}
+case class Corr(
+ left: Expression,
+ right: Expression,
+ mutableAggBufferOffset: Int = 0,
+ inputAggBufferOffset: Int = 0)
+ extends ImperativeAggregate {
+
+ def children: Seq[Expression] = Seq(left, right)
+
+ def nullable: Boolean = false
+
+ def dataType: DataType = DoubleType
+
+ def inputTypes: Seq[AbstractDataType] = Seq(DoubleType)
+
+ def aggBufferSchema: StructType = StructType.fromAttributes(aggBufferAttributes)
+
+ def inputAggBufferAttributes: Seq[AttributeReference] = aggBufferAttributes.map(_.newInstance())
+
+ val aggBufferAttributes: Seq[AttributeReference] = Seq(
+ AttributeReference("xAvg", DoubleType)(),
+ AttributeReference("yAvg", DoubleType)(),
+ AttributeReference("Ck", DoubleType)(),
+ AttributeReference("MkX", DoubleType)(),
+ AttributeReference("MkY", DoubleType)(),
+ AttributeReference("count", LongType)())
+
+ override def withNewMutableAggBufferOffset(newMutableAggBufferOffset: Int): ImperativeAggregate =
+ copy(mutableAggBufferOffset = newMutableAggBufferOffset)
+
+ override def withNewInputAggBufferOffset(newInputAggBufferOffset: Int): ImperativeAggregate =
+ copy(inputAggBufferOffset = newInputAggBufferOffset)
+
+ override def initialize(buffer: MutableRow): Unit = {
+ (0 until 5).map(idx => buffer.setDouble(mutableAggBufferOffset + idx, 0.0))
+ buffer.setLong(mutableAggBufferOffset + 5, 0L)
+ }
+
+ override def update(buffer: MutableRow, input: InternalRow): Unit = {
+ val x = left.eval(input).asInstanceOf[Double]
+ val y = right.eval(input).asInstanceOf[Double]
+
+ var xAvg = buffer.getDouble(mutableAggBufferOffset)
+ var yAvg = buffer.getDouble(mutableAggBufferOffset + 1)
+ var Ck = buffer.getDouble(mutableAggBufferOffset + 2)
+ var MkX = buffer.getDouble(mutableAggBufferOffset + 3)
+ var MkY = buffer.getDouble(mutableAggBufferOffset + 4)
+ var count = buffer.getLong(mutableAggBufferOffset + 5)
+
+ val deltaX = x - xAvg
+ val deltaY = y - yAvg
+ count += 1
+ xAvg += deltaX / count
+ yAvg += deltaY / count
+ Ck += deltaX * (y - yAvg)
+ MkX += deltaX * (x - xAvg)
+ MkY += deltaY * (y - yAvg)
+
+ buffer.setDouble(mutableAggBufferOffset, xAvg)
+ buffer.setDouble(mutableAggBufferOffset + 1, yAvg)
+ buffer.setDouble(mutableAggBufferOffset + 2, Ck)
+ buffer.setDouble(mutableAggBufferOffset + 3, MkX)
+ buffer.setDouble(mutableAggBufferOffset + 4, MkY)
+ buffer.setLong(mutableAggBufferOffset + 5, count)
+ }
+
+ override def merge(buffer1: MutableRow, buffer2: InternalRow): Unit = {
+ val count2 = buffer2.getLong(inputAggBufferOffset + 5)
+
+ if (count2 > 0) {
+ var xAvg = buffer1.getDouble(mutableAggBufferOffset)
+ var yAvg = buffer1.getDouble(mutableAggBufferOffset + 1)
+ var Ck = buffer1.getDouble(mutableAggBufferOffset + 2)
+ var MkX = buffer1.getDouble(mutableAggBufferOffset + 3)
+ var MkY = buffer1.getDouble(mutableAggBufferOffset + 4)
+ var count = buffer1.getLong(mutableAggBufferOffset + 5)
+
+ val xAvg2 = buffer2.getDouble(inputAggBufferOffset)
+ val yAvg2 = buffer2.getDouble(inputAggBufferOffset + 1)
+ val Ck2 = buffer2.getDouble(inputAggBufferOffset + 2)
+ val MkX2 = buffer2.getDouble(inputAggBufferOffset + 3)
+ val MkY2 = buffer2.getDouble(inputAggBufferOffset + 4)
+
+ val totalCount = count + count2
+ val deltaX = xAvg - xAvg2
+ val deltaY = yAvg - yAvg2
+ Ck += Ck2 + deltaX * deltaY * count / totalCount * count2
+ xAvg = (xAvg * count + xAvg2 * count2) / totalCount
+ yAvg = (yAvg * count + yAvg2 * count2) / totalCount
+ MkX += MkX2 + deltaX * deltaX * count / totalCount * count2
+ MkY += MkY2 + deltaY * deltaY * count / totalCount * count2
+ count = totalCount
+
+ buffer1.setDouble(mutableAggBufferOffset, xAvg)
+ buffer1.setDouble(mutableAggBufferOffset + 1, yAvg)
+ buffer1.setDouble(mutableAggBufferOffset + 2, Ck)
+ buffer1.setDouble(mutableAggBufferOffset + 3, MkX)
+ buffer1.setDouble(mutableAggBufferOffset + 4, MkY)
+ buffer1.setLong(mutableAggBufferOffset + 5, count)
+ }
+ }
+
+ override def eval(buffer: InternalRow): Any = {
+ val Ck = buffer.getDouble(mutableAggBufferOffset + 2)
+ val MkX = buffer.getDouble(mutableAggBufferOffset + 3)
+ val MkY = buffer.getDouble(mutableAggBufferOffset + 4)
+ Ck / math.sqrt(MkX * MkY)
--- End diff --
From the failed test HiveCompatibilitySuite, looks like Hive will return NULL for this case. I think we should follow it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-150297638
**[Test build #44154 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44154/consoleFull)** for PR 8587 at commit [`cc1657b`](https://github.com/apache/spark/commit/cc1657b9cf5da4f07804958420bb94a58c86593f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `case class ClassEncoder[T](`\n * `case class First(child: Expression, ignoreNullsExpr: Expression) extends DeclarativeAggregate `\n * `case class Last(child: Expression, ignoreNullsExpr: Expression) extends DeclarativeAggregate `\n * `case class Corr(`\n * `case class First(`\n * `case class FirstFunction(`\n * `case class Last(`\n * `case class LastFunction(`\n * `case class Corr(`\n * `case class CreateRow(children: Seq[Expression]) extends Expression `\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152681971
**[Test build #44719 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44719/consoleFull)** for PR 8587 at commit [`2de76b4`](https://github.com/apache/spark/commit/2de76b444456bc7e751fa9ccb85a6e8f0662ff76).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `case class Corr(`\n * `case class Corr(left: Expression, right: Expression)`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152453875
**[Test build #44671 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44671/consoleFull)** for PR 8587 at commit [`7dcf689`](https://github.com/apache/spark/commit/7dcf689cca8ae4bd89a1f7c0b28373c8cfb6b8f0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152893732
ping @yhuai any more comments?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152599151
**[Test build #44690 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44690/consoleFull)** for PR 8587 at commit [`2de76b4`](https://github.com/apache/spark/commit/2de76b444456bc7e751fa9ccb85a6e8f0662ff76).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152895083
Thanks! Merging to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-138124296
[Test build #42076 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42076/console) for PR 8587 at commit [`0dd6320`](https://github.com/apache/spark/commit/0dd63203e8bb62ce02727da9ec6e066576d23b3a).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class Corr(left: Expression, right: Expression) extends AlgebraicAggregate `
* `case class Corr(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-137544734
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-150070728
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/8587#discussion_r43090314
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/functions.scala ---
@@ -524,6 +525,116 @@ case class Sum(child: Expression) extends DeclarativeAggregate {
override val evaluateExpression = Cast(currentSum, resultType)
}
+case class Corr(
+ left: Expression,
+ right: Expression,
+ mutableAggBufferOffset: Int = 0,
+ inputAggBufferOffset: Int = 0)
+ extends ImperativeAggregate {
+
+ def children: Seq[Expression] = Seq(left, right)
+
+ def nullable: Boolean = false
+
+ def dataType: DataType = DoubleType
+
+ def inputTypes: Seq[AbstractDataType] = Seq(DoubleType)
+
+ def aggBufferSchema: StructType = StructType.fromAttributes(aggBufferAttributes)
+
+ def inputAggBufferAttributes: Seq[AttributeReference] = aggBufferAttributes.map(_.newInstance())
+
+ val aggBufferAttributes: Seq[AttributeReference] = Seq(
+ AttributeReference("xAvg", DoubleType)(),
+ AttributeReference("yAvg", DoubleType)(),
+ AttributeReference("Ck", DoubleType)(),
+ AttributeReference("MkX", DoubleType)(),
+ AttributeReference("MkY", DoubleType)(),
+ AttributeReference("count", LongType)())
+
+ override def withNewMutableAggBufferOffset(newMutableAggBufferOffset: Int): ImperativeAggregate =
+ copy(mutableAggBufferOffset = newMutableAggBufferOffset)
+
+ override def withNewInputAggBufferOffset(newInputAggBufferOffset: Int): ImperativeAggregate =
+ copy(inputAggBufferOffset = newInputAggBufferOffset)
+
+ override def initialize(buffer: MutableRow): Unit = {
+ (0 until 5).map(idx => buffer.setDouble(mutableAggBufferOffset + idx, 0.0))
+ buffer.setLong(mutableAggBufferOffset + 5, 0L)
+ }
+
+ override def update(buffer: MutableRow, input: InternalRow): Unit = {
+ val x = left.eval(input).asInstanceOf[Double]
+ val y = right.eval(input).asInstanceOf[Double]
+
+ var xAvg = buffer.getDouble(mutableAggBufferOffset)
+ var yAvg = buffer.getDouble(mutableAggBufferOffset + 1)
+ var Ck = buffer.getDouble(mutableAggBufferOffset + 2)
+ var MkX = buffer.getDouble(mutableAggBufferOffset + 3)
+ var MkY = buffer.getDouble(mutableAggBufferOffset + 4)
+ var count = buffer.getLong(mutableAggBufferOffset + 5)
+
+ val deltaX = x - xAvg
+ val deltaY = y - yAvg
+ count += 1
+ xAvg += deltaX / count
+ yAvg += deltaY / count
+ Ck += deltaX * (y - yAvg)
+ MkX += deltaX * (x - xAvg)
+ MkY += deltaY * (y - yAvg)
+
+ buffer.setDouble(mutableAggBufferOffset, xAvg)
+ buffer.setDouble(mutableAggBufferOffset + 1, yAvg)
+ buffer.setDouble(mutableAggBufferOffset + 2, Ck)
+ buffer.setDouble(mutableAggBufferOffset + 3, MkX)
+ buffer.setDouble(mutableAggBufferOffset + 4, MkY)
+ buffer.setLong(mutableAggBufferOffset + 5, count)
+ }
+
+ override def merge(buffer1: MutableRow, buffer2: InternalRow): Unit = {
+ val count2 = buffer2.getLong(inputAggBufferOffset + 5)
+
+ if (count2 > 0) {
+ var xAvg = buffer1.getDouble(mutableAggBufferOffset)
+ var yAvg = buffer1.getDouble(mutableAggBufferOffset + 1)
+ var Ck = buffer1.getDouble(mutableAggBufferOffset + 2)
+ var MkX = buffer1.getDouble(mutableAggBufferOffset + 3)
+ var MkY = buffer1.getDouble(mutableAggBufferOffset + 4)
+ var count = buffer1.getLong(mutableAggBufferOffset + 5)
+
+ val xAvg2 = buffer2.getDouble(inputAggBufferOffset)
+ val yAvg2 = buffer2.getDouble(inputAggBufferOffset + 1)
+ val Ck2 = buffer2.getDouble(inputAggBufferOffset + 2)
+ val MkX2 = buffer2.getDouble(inputAggBufferOffset + 3)
+ val MkY2 = buffer2.getDouble(inputAggBufferOffset + 4)
+
+ val totalCount = count + count2
+ val deltaX = xAvg - xAvg2
+ val deltaY = yAvg - yAvg2
+ Ck += Ck2 + deltaX * deltaY * count / totalCount * count2
+ xAvg = (xAvg * count + xAvg2 * count2) / totalCount
+ yAvg = (yAvg * count + yAvg2 * count2) / totalCount
+ MkX += MkX2 + deltaX * deltaX * count / totalCount * count2
+ MkY += MkY2 + deltaY * deltaY * count / totalCount * count2
+ count = totalCount
+
+ buffer1.setDouble(mutableAggBufferOffset, xAvg)
+ buffer1.setDouble(mutableAggBufferOffset + 1, yAvg)
+ buffer1.setDouble(mutableAggBufferOffset + 2, Ck)
+ buffer1.setDouble(mutableAggBufferOffset + 3, MkX)
+ buffer1.setDouble(mutableAggBufferOffset + 4, MkY)
+ buffer1.setLong(mutableAggBufferOffset + 5, count)
+ }
+ }
+
+ override def eval(buffer: InternalRow): Any = {
+ val Ck = buffer.getDouble(mutableAggBufferOffset + 2)
+ val MkX = buffer.getDouble(mutableAggBufferOffset + 3)
+ val MkY = buffer.getDouble(mutableAggBufferOffset + 4)
+ Ck / math.sqrt(MkX * MkY)
--- End diff --
What if `count` is zero? Shall we return `NaN`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152696972
**[Test build #44726 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44726/consoleFull)** for PR 8587 at commit [`2de76b4`](https://github.com/apache/spark/commit/2de76b444456bc7e751fa9ccb85a6e8f0662ff76).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `case class Corr(`\n * `case class Corr(left: Expression, right: Expression)`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152416560
**[Test build #44659 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44659/consoleFull)** for PR 8587 at commit [`4f8c381`](https://github.com/apache/spark/commit/4f8c381c48244e46ac6437b83e70eccbf42ac907).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152354238
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152689136
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/8587#discussion_r43352853
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/functions.scala ---
@@ -524,6 +525,116 @@ case class Sum(child: Expression) extends DeclarativeAggregate {
override val evaluateExpression = Cast(currentSum, resultType)
}
+case class Corr(
+ left: Expression,
+ right: Expression,
+ mutableAggBufferOffset: Int = 0,
+ inputAggBufferOffset: Int = 0)
+ extends ImperativeAggregate {
+
+ def children: Seq[Expression] = Seq(left, right)
+
+ def nullable: Boolean = false
+
+ def dataType: DataType = DoubleType
+
+ def inputTypes: Seq[AbstractDataType] = Seq(DoubleType)
+
+ def aggBufferSchema: StructType = StructType.fromAttributes(aggBufferAttributes)
+
+ def inputAggBufferAttributes: Seq[AttributeReference] = aggBufferAttributes.map(_.newInstance())
+
+ val aggBufferAttributes: Seq[AttributeReference] = Seq(
+ AttributeReference("xAvg", DoubleType)(),
+ AttributeReference("yAvg", DoubleType)(),
+ AttributeReference("Ck", DoubleType)(),
+ AttributeReference("MkX", DoubleType)(),
+ AttributeReference("MkY", DoubleType)(),
+ AttributeReference("count", LongType)())
+
+ override def withNewMutableAggBufferOffset(newMutableAggBufferOffset: Int): ImperativeAggregate =
+ copy(mutableAggBufferOffset = newMutableAggBufferOffset)
+
+ override def withNewInputAggBufferOffset(newInputAggBufferOffset: Int): ImperativeAggregate =
+ copy(inputAggBufferOffset = newInputAggBufferOffset)
+
+ override def initialize(buffer: MutableRow): Unit = {
+ (0 until 5).map(idx => buffer.setDouble(mutableAggBufferOffset + idx, 0.0))
+ buffer.setLong(mutableAggBufferOffset + 5, 0L)
+ }
+
+ override def update(buffer: MutableRow, input: InternalRow): Unit = {
+ val x = left.eval(input).asInstanceOf[Double]
+ val y = right.eval(input).asInstanceOf[Double]
+
+ var xAvg = buffer.getDouble(mutableAggBufferOffset)
+ var yAvg = buffer.getDouble(mutableAggBufferOffset + 1)
--- End diff --
ok. reasonable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-138108134
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/8587#discussion_r43395879
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/functions.scala ---
@@ -524,6 +525,133 @@ case class Sum(child: Expression) extends DeclarativeAggregate {
override val evaluateExpression = Cast(currentSum, resultType)
}
+/**
+ * Compute Pearson correlation between two expressions.
+ * When applied on empty data (i.e., count is zero), it returns NaN.
+ *
+ * Definition of Pearson correlation can be found at
+ * http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient
+ *
+ * @param left one of the expressions to compute correlation with.
+ * @param right another expression to compute correlation with.
+ */
+case class Corr(
+ left: Expression,
+ right: Expression,
+ mutableAggBufferOffset: Int = 0,
+ inputAggBufferOffset: Int = 0)
+ extends ImperativeAggregate {
+
+ def children: Seq[Expression] = Seq(left, right)
+
+ def nullable: Boolean = false
+
+ def dataType: DataType = DoubleType
+
+ def inputTypes: Seq[AbstractDataType] = Seq(DoubleType)
+
+ def aggBufferSchema: StructType = StructType.fromAttributes(aggBufferAttributes)
+
+ def inputAggBufferAttributes: Seq[AttributeReference] = aggBufferAttributes.map(_.newInstance())
+
+ val aggBufferAttributes: Seq[AttributeReference] = Seq(
+ AttributeReference("xAvg", DoubleType)(),
+ AttributeReference("yAvg", DoubleType)(),
+ AttributeReference("Ck", DoubleType)(),
+ AttributeReference("MkX", DoubleType)(),
+ AttributeReference("MkY", DoubleType)(),
+ AttributeReference("count", LongType)())
+
+ override def withNewMutableAggBufferOffset(newMutableAggBufferOffset: Int): ImperativeAggregate =
+ copy(mutableAggBufferOffset = newMutableAggBufferOffset)
+
+ override def withNewInputAggBufferOffset(newInputAggBufferOffset: Int): ImperativeAggregate =
+ copy(inputAggBufferOffset = newInputAggBufferOffset)
+
+ override def initialize(buffer: MutableRow): Unit = {
+ (0 until 5).map(idx => buffer.setDouble(mutableAggBufferOffset + idx, 0.0))
+ buffer.setLong(mutableAggBufferOffset + 5, 0L)
+ }
+
+ override def update(buffer: MutableRow, input: InternalRow): Unit = {
+ val x = left.eval(input).asInstanceOf[Double]
+ val y = right.eval(input).asInstanceOf[Double]
+
+ var xAvg = buffer.getDouble(mutableAggBufferOffset)
+ var yAvg = buffer.getDouble(mutableAggBufferOffset + 1)
+ var Ck = buffer.getDouble(mutableAggBufferOffset + 2)
+ var MkX = buffer.getDouble(mutableAggBufferOffset + 3)
+ var MkY = buffer.getDouble(mutableAggBufferOffset + 4)
+ var count = buffer.getLong(mutableAggBufferOffset + 5)
+
+ val deltaX = x - xAvg
+ val deltaY = y - yAvg
+ count += 1
+ xAvg += deltaX / count
+ yAvg += deltaY / count
+ Ck += deltaX * (y - yAvg)
+ MkX += deltaX * (x - xAvg)
+ MkY += deltaY * (y - yAvg)
+
+ buffer.setDouble(mutableAggBufferOffset, xAvg)
+ buffer.setDouble(mutableAggBufferOffset + 1, yAvg)
+ buffer.setDouble(mutableAggBufferOffset + 2, Ck)
+ buffer.setDouble(mutableAggBufferOffset + 3, MkX)
+ buffer.setDouble(mutableAggBufferOffset + 4, MkY)
+ buffer.setLong(mutableAggBufferOffset + 5, count)
+ }
+
+ // Merge counters from other partitions. Formula can be found at:
+ // http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance
+ override def merge(buffer1: MutableRow, buffer2: InternalRow): Unit = {
+ val count2 = buffer2.getLong(inputAggBufferOffset + 5)
+
+ if (count2 > 0) {
--- End diff --
We only need to consider count in buffer2. I will add document for it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-138109329
[Test build #42076 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42076/consoleFull) for PR 8587 at commit [`0dd6320`](https://github.com/apache/spark/commit/0dd63203e8bb62ce02727da9ec6e066576d23b3a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-150061679
**[Test build #44106 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44106/consoleFull)** for PR 8587 at commit [`d10afbe`](https://github.com/apache/spark/commit/d10afbefddacb8f779414948fc2217bdc6a9e791).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/8587#discussion_r43162488
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/AggregationQuerySuite.scala ---
@@ -556,6 +556,33 @@ abstract class AggregationQuerySuite extends QueryTest with SQLTestUtils with Te
Row(0, null, 1, 1, null, 0) :: Nil)
}
+ test("pearson correlation") {
+ val df = Seq.tabulate(10)(i => (1.0 * i, 2.0 * i, i * -1.0)).toDF("a", "b", "c")
+ val corr1 = df.repartition(2).groupBy().agg(corr("a", "b")).collect()(0).getDouble(0)
+ assert(math.abs(corr1 - 1.0) < 1e-12)
+ val corr2 = df.groupBy().agg(corr("a", "c")).collect()(0).getDouble(0)
+ assert(math.abs(corr2 + 1.0) < 1e-12)
+ // non-trivial example. To reproduce in python, use:
+ // >>> from scipy.stats import pearsonr
+ // >>> import numpy as np
+ // >>> a = np.array(range(20))
+ // >>> b = np.array([x * x - 2 * x + 3.5 for x in range(20)])
+ // >>> pearsonr(a, b)
+ // (0.95723391394758572, 3.8902121417802199e-11)
+ // In R, use:
+ // > a <- 0:19
+ // > b <- mapply(function(x) x * x - 2 * x + 3.5, a)
+ // > cor(a, b)
+ // [1] 0.957233913947585835
+ val df2 = Seq.tabulate(20)(x => (1.0 * x, x * x - 2 * x + 3.5)).toDF("a", "b")
+ val corr3 = df2.groupBy().agg(corr("a", "b")).collect()(0).getDouble(0)
+ assert(math.abs(corr3 - 0.95723391394758572) < 1e-12)
+
+ val df3 = Seq.tabulate(0)(i => (1.0 * i, 2.0 * i)).toDF("a", "b")
+ val corr4 = df3.groupBy().agg(corr("a", "b")).collect()(0).getDouble(0)
+ assert(corr4.isNaN)
+ }
--- End diff --
What will happen if the data type of input parameters are not double?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/8587#discussion_r43162271
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/functions.scala ---
@@ -524,6 +525,133 @@ case class Sum(child: Expression) extends DeclarativeAggregate {
override val evaluateExpression = Cast(currentSum, resultType)
}
+/**
+ * Compute Pearson correlation between two expressions.
+ * When applied on empty data (i.e., count is zero), it returns NaN.
+ *
+ * Definition of Pearson correlation can be found at
+ * http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient
+ *
+ * @param left one of the expressions to compute correlation with.
+ * @param right another expression to compute correlation with.
+ */
+case class Corr(
+ left: Expression,
+ right: Expression,
+ mutableAggBufferOffset: Int = 0,
+ inputAggBufferOffset: Int = 0)
+ extends ImperativeAggregate {
+
+ def children: Seq[Expression] = Seq(left, right)
+
+ def nullable: Boolean = false
+
+ def dataType: DataType = DoubleType
+
+ def inputTypes: Seq[AbstractDataType] = Seq(DoubleType)
+
+ def aggBufferSchema: StructType = StructType.fromAttributes(aggBufferAttributes)
+
+ def inputAggBufferAttributes: Seq[AttributeReference] = aggBufferAttributes.map(_.newInstance())
+
+ val aggBufferAttributes: Seq[AttributeReference] = Seq(
+ AttributeReference("xAvg", DoubleType)(),
+ AttributeReference("yAvg", DoubleType)(),
+ AttributeReference("Ck", DoubleType)(),
+ AttributeReference("MkX", DoubleType)(),
+ AttributeReference("MkY", DoubleType)(),
+ AttributeReference("count", LongType)())
+
+ override def withNewMutableAggBufferOffset(newMutableAggBufferOffset: Int): ImperativeAggregate =
+ copy(mutableAggBufferOffset = newMutableAggBufferOffset)
+
+ override def withNewInputAggBufferOffset(newInputAggBufferOffset: Int): ImperativeAggregate =
+ copy(inputAggBufferOffset = newInputAggBufferOffset)
+
+ override def initialize(buffer: MutableRow): Unit = {
+ (0 until 5).map(idx => buffer.setDouble(mutableAggBufferOffset + idx, 0.0))
+ buffer.setLong(mutableAggBufferOffset + 5, 0L)
+ }
+
+ override def update(buffer: MutableRow, input: InternalRow): Unit = {
+ val x = left.eval(input).asInstanceOf[Double]
+ val y = right.eval(input).asInstanceOf[Double]
+
+ var xAvg = buffer.getDouble(mutableAggBufferOffset)
+ var yAvg = buffer.getDouble(mutableAggBufferOffset + 1)
+ var Ck = buffer.getDouble(mutableAggBufferOffset + 2)
+ var MkX = buffer.getDouble(mutableAggBufferOffset + 3)
+ var MkY = buffer.getDouble(mutableAggBufferOffset + 4)
+ var count = buffer.getLong(mutableAggBufferOffset + 5)
+
+ val deltaX = x - xAvg
+ val deltaY = y - yAvg
+ count += 1
+ xAvg += deltaX / count
+ yAvg += deltaY / count
+ Ck += deltaX * (y - yAvg)
+ MkX += deltaX * (x - xAvg)
+ MkY += deltaY * (y - yAvg)
+
+ buffer.setDouble(mutableAggBufferOffset, xAvg)
+ buffer.setDouble(mutableAggBufferOffset + 1, yAvg)
+ buffer.setDouble(mutableAggBufferOffset + 2, Ck)
+ buffer.setDouble(mutableAggBufferOffset + 3, MkX)
+ buffer.setDouble(mutableAggBufferOffset + 4, MkY)
+ buffer.setLong(mutableAggBufferOffset + 5, count)
+ }
+
+ // Merge counters from other partitions. Formula can be found at:
+ // http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance
+ override def merge(buffer1: MutableRow, buffer2: InternalRow): Unit = {
+ val count2 = buffer2.getLong(inputAggBufferOffset + 5)
+
+ if (count2 > 0) {
--- End diff --
Is it safe to assume that the ```count2``` in ```buffer1``` is non zero? There is - currently - no documentation on this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-151590941
Can you add this function to FunctionRegistry? Otherwise, we cannot use it in SQL.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/8587#discussion_r43439837
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/functions.scala ---
@@ -524,6 +525,133 @@ case class Sum(child: Expression) extends DeclarativeAggregate {
override val evaluateExpression = Cast(currentSum, resultType)
}
+/**
+ * Compute Pearson correlation between two expressions.
+ * When applied on empty data (i.e., count is zero), it returns NaN.
+ *
+ * Definition of Pearson correlation can be found at
+ * http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient
+ *
+ * @param left one of the expressions to compute correlation with.
+ * @param right another expression to compute correlation with.
+ */
+case class Corr(
+ left: Expression,
+ right: Expression,
+ mutableAggBufferOffset: Int = 0,
+ inputAggBufferOffset: Int = 0)
+ extends ImperativeAggregate {
+
+ def children: Seq[Expression] = Seq(left, right)
+
+ def nullable: Boolean = false
+
+ def dataType: DataType = DoubleType
+
+ def inputTypes: Seq[AbstractDataType] = Seq(DoubleType)
+
+ def aggBufferSchema: StructType = StructType.fromAttributes(aggBufferAttributes)
+
+ def inputAggBufferAttributes: Seq[AttributeReference] = aggBufferAttributes.map(_.newInstance())
+
+ val aggBufferAttributes: Seq[AttributeReference] = Seq(
+ AttributeReference("xAvg", DoubleType)(),
+ AttributeReference("yAvg", DoubleType)(),
+ AttributeReference("Ck", DoubleType)(),
+ AttributeReference("MkX", DoubleType)(),
+ AttributeReference("MkY", DoubleType)(),
+ AttributeReference("count", LongType)())
+
+ override def withNewMutableAggBufferOffset(newMutableAggBufferOffset: Int): ImperativeAggregate =
+ copy(mutableAggBufferOffset = newMutableAggBufferOffset)
+
+ override def withNewInputAggBufferOffset(newInputAggBufferOffset: Int): ImperativeAggregate =
+ copy(inputAggBufferOffset = newInputAggBufferOffset)
+
+ override def initialize(buffer: MutableRow): Unit = {
+ (0 until 5).map(idx => buffer.setDouble(mutableAggBufferOffset + idx, 0.0))
+ buffer.setLong(mutableAggBufferOffset + 5, 0L)
+ }
+
+ override def update(buffer: MutableRow, input: InternalRow): Unit = {
+ val x = left.eval(input).asInstanceOf[Double]
+ val y = right.eval(input).asInstanceOf[Double]
+
+ var xAvg = buffer.getDouble(mutableAggBufferOffset)
+ var yAvg = buffer.getDouble(mutableAggBufferOffset + 1)
+ var Ck = buffer.getDouble(mutableAggBufferOffset + 2)
+ var MkX = buffer.getDouble(mutableAggBufferOffset + 3)
+ var MkY = buffer.getDouble(mutableAggBufferOffset + 4)
+ var count = buffer.getLong(mutableAggBufferOffset + 5)
+
+ val deltaX = x - xAvg
+ val deltaY = y - yAvg
+ count += 1
+ xAvg += deltaX / count
+ yAvg += deltaY / count
+ Ck += deltaX * (y - yAvg)
+ MkX += deltaX * (x - xAvg)
+ MkY += deltaY * (y - yAvg)
+
+ buffer.setDouble(mutableAggBufferOffset, xAvg)
+ buffer.setDouble(mutableAggBufferOffset + 1, yAvg)
+ buffer.setDouble(mutableAggBufferOffset + 2, Ck)
+ buffer.setDouble(mutableAggBufferOffset + 3, MkX)
+ buffer.setDouble(mutableAggBufferOffset + 4, MkY)
+ buffer.setLong(mutableAggBufferOffset + 5, count)
+ }
+
+ // Merge counters from other partitions. Formula can be found at:
+ // http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance
+ override def merge(buffer1: MutableRow, buffer2: InternalRow): Unit = {
+ val count2 = buffer2.getLong(inputAggBufferOffset + 5)
+
+ if (count2 > 0) {
--- End diff --
Thanks for the comment. Now it is obvious, I wasn't thinking...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152372702
**[Test build #44645 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44645/consoleFull)** for PR 8587 at commit [`3b731e2`](https://github.com/apache/spark/commit/3b731e2c9b08dbade38da73bcff94cf1b2cd7636).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `case class Corr(`\n * `case class Corr(left: Expression, right: Expression)`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-150070696
**[Test build #44108 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44108/consoleFull)** for PR 8587 at commit [`cc1657b`](https://github.com/apache/spark/commit/cc1657b9cf5da4f07804958420bb94a58c86593f).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `case class Corr(`\n * `case class Corr(`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152680203
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-137951293
Seems it is feasible. I will update this later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-137544737
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41976/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-150265212
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-150061987
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152897683
@yhuai Thank you!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/8587#discussion_r43094562
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/functions.scala ---
@@ -524,6 +525,116 @@ case class Sum(child: Expression) extends DeclarativeAggregate {
override val evaluateExpression = Cast(currentSum, resultType)
}
+case class Corr(
+ left: Expression,
+ right: Expression,
+ mutableAggBufferOffset: Int = 0,
+ inputAggBufferOffset: Int = 0)
+ extends ImperativeAggregate {
+
+ def children: Seq[Expression] = Seq(left, right)
+
+ def nullable: Boolean = false
+
+ def dataType: DataType = DoubleType
+
+ def inputTypes: Seq[AbstractDataType] = Seq(DoubleType)
+
+ def aggBufferSchema: StructType = StructType.fromAttributes(aggBufferAttributes)
+
+ def inputAggBufferAttributes: Seq[AttributeReference] = aggBufferAttributes.map(_.newInstance())
+
+ val aggBufferAttributes: Seq[AttributeReference] = Seq(
+ AttributeReference("xAvg", DoubleType)(),
+ AttributeReference("yAvg", DoubleType)(),
+ AttributeReference("Ck", DoubleType)(),
+ AttributeReference("MkX", DoubleType)(),
+ AttributeReference("MkY", DoubleType)(),
+ AttributeReference("count", LongType)())
+
+ override def withNewMutableAggBufferOffset(newMutableAggBufferOffset: Int): ImperativeAggregate =
+ copy(mutableAggBufferOffset = newMutableAggBufferOffset)
+
+ override def withNewInputAggBufferOffset(newInputAggBufferOffset: Int): ImperativeAggregate =
+ copy(inputAggBufferOffset = newInputAggBufferOffset)
+
+ override def initialize(buffer: MutableRow): Unit = {
+ (0 until 5).map(idx => buffer.setDouble(mutableAggBufferOffset + idx, 0.0))
+ buffer.setLong(mutableAggBufferOffset + 5, 0L)
+ }
+
+ override def update(buffer: MutableRow, input: InternalRow): Unit = {
+ val x = left.eval(input).asInstanceOf[Double]
+ val y = right.eval(input).asInstanceOf[Double]
+
+ var xAvg = buffer.getDouble(mutableAggBufferOffset)
+ var yAvg = buffer.getDouble(mutableAggBufferOffset + 1)
--- End diff --
I think we don't want to modify mutableAggBufferOffset but only want to use mutableAggBufferOffset + 1, mutableAggBufferOffset + 2...etc.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/8587#discussion_r43162805
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/functions.scala ---
@@ -524,6 +525,133 @@ case class Sum(child: Expression) extends DeclarativeAggregate {
override val evaluateExpression = Cast(currentSum, resultType)
}
+/**
+ * Compute Pearson correlation between two expressions.
+ * When applied on empty data (i.e., count is zero), it returns NaN.
+ *
+ * Definition of Pearson correlation can be found at
+ * http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient
+ *
+ * @param left one of the expressions to compute correlation with.
+ * @param right another expression to compute correlation with.
+ */
+case class Corr(
+ left: Expression,
+ right: Expression,
+ mutableAggBufferOffset: Int = 0,
+ inputAggBufferOffset: Int = 0)
+ extends ImperativeAggregate {
+
+ def children: Seq[Expression] = Seq(left, right)
+
+ def nullable: Boolean = false
+
+ def dataType: DataType = DoubleType
+
+ def inputTypes: Seq[AbstractDataType] = Seq(DoubleType)
--- End diff --
Probably we need to do something like what we have done in `AVERAGE` (https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/functions.scala#L40-L52).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-150063426
**[Test build #44108 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44108/consoleFull)** for PR 8587 at commit [`cc1657b`](https://github.com/apache/spark/commit/cc1657b9cf5da4f07804958420bb94a58c86593f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152264277
**[Test build #44616 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44616/consoleFull)** for PR 8587 at commit [`2f7b864`](https://github.com/apache/spark/commit/2f7b8641ec0c7c7f56b0df2524064a2c47e13288).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152596117
@yhuai ok. I've put the test in the blacklist, and also add corresponding test to AggregationQuerySuite.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-150063146
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-151462008
**[Test build #44418 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44418/consoleFull)** for PR 8587 at commit [`02562f3`](https://github.com/apache/spark/commit/02562f3a9ab9cda941b260a834d505f7cacd46f2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `case class Corr(`\n * `case class Corr(`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-151462145
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-150264033
retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152261607
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152689133
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-138091594
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152444874
**[Test build #44659 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44659/consoleFull)** for PR 8587 at commit [`4f8c381`](https://github.com/apache/spark/commit/4f8c381c48244e46ac6437b83e70eccbf42ac907).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `case class Corr(`\n * `case class Corr(left: Expression, right: Expression)`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-138091393
@rxin I've updated this patch to use `AlgebraicAggregate` as you suggested. Please take a look when you are available. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152415818
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152605050
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152472938
**[Test build #44671 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44671/consoleFull)** for PR 8587 at commit [`7dcf689`](https://github.com/apache/spark/commit/7dcf689cca8ae4bd89a1f7c0b28373c8cfb6b8f0).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `case class Corr(`\n * `case class Corr(left: Expression, right: Expression)`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152696994
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44726/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152680422
**[Test build #44719 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44719/consoleFull)** for PR 8587 at commit [`2de76b4`](https://github.com/apache/spark/commit/2de76b444456bc7e751fa9ccb85a6e8f0662ff76).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-150308124
ping @mengxr
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-150057960
**[Test build #44103 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44103/consoleFull)** for PR 8587 at commit [`1505cd2`](https://github.com/apache/spark/commit/1505cd23a8e0cbd1b26da499f53fd6416ac96ca8).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `case class Corr(`\n * `case class Corr(`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152596761
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/8587#discussion_r43403727
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/functions.scala ---
@@ -524,6 +525,133 @@ case class Sum(child: Expression) extends DeclarativeAggregate {
override val evaluateExpression = Cast(currentSum, resultType)
}
+/**
+ * Compute Pearson correlation between two expressions.
+ * When applied on empty data (i.e., count is zero), it returns NaN.
+ *
+ * Definition of Pearson correlation can be found at
+ * http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient
+ *
+ * @param left one of the expressions to compute correlation with.
+ * @param right another expression to compute correlation with.
+ */
+case class Corr(
+ left: Expression,
+ right: Expression,
+ mutableAggBufferOffset: Int = 0,
+ inputAggBufferOffset: Int = 0)
+ extends ImperativeAggregate {
+
+ def children: Seq[Expression] = Seq(left, right)
+
+ def nullable: Boolean = false
+
+ def dataType: DataType = DoubleType
+
+ def inputTypes: Seq[AbstractDataType] = Seq(DoubleType)
--- End diff --
+1. The return type should always be double.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/8587#discussion_r43452255
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregates.scala ---
@@ -747,6 +747,19 @@ case class LastFunction(
}
}
+/**
+ * Calculate Pearson Correlation Coefficient for the given columns.
+ * Only support AggregateExpression2.
+ *
+ */
+case class Corr(left: Expression, right: Expression)
+ extends BinaryExpression with AggregateExpression with ImplicitCastInputTypes {
--- End diff --
But how do we check spark.sql.useAggregate2=false at this expression?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/8587#discussion_r43162228
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregates.scala ---
@@ -747,6 +747,19 @@ case class LastFunction(
}
}
+/**
+ * Calculate Pearson Correlation Coefficient for the given columns.
+ * Only support AggregateExpression2.
+ *
+ */
+case class Corr(
+ left: Expression,
+ right: Expression) extends BinaryExpression with AggregateExpression {
+ override def nullable: Boolean = false
+ override def dataType: DoubleType.type = DoubleType
+ override def toString: String = s"CORRELATION($left, $right)"
+}
--- End diff --
What will be the error message if we call this function when `spark.sql.useAggregate2=false`? It will be good to provide a meaning error message.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-149753551
@mengxr I have the original code. I wii revert to it soon as I come back in these two days.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-137544576
[Test build #41976 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41976/console) for PR 8587 at commit [`cb34a95`](https://github.com/apache/spark/commit/cb34a95e3dea152250b6409827fc869bd7fae407).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class Corr(left: Expression, right: Expression) extends AggregateFunction2 `
* `case class Corr(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-151425036
**[Test build #44418 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44418/consoleFull)** for PR 8587 at commit [`02562f3`](https://github.com/apache/spark/commit/02562f3a9ab9cda941b260a834d505f7cacd46f2).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152354220
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-149650308
@viirya We compared the performance of declarative approach vs imperative in https://issues.apache.org/jira/browse/SPARK-10953. The declarative approach is much slower because of extra code and lack of common sub-expression elimination. If you still keep the original code, could you revert this PR to your first commit? Also cc @rxin for this performance issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152572288
If that is the answer generated by Hive, we should not change that (at least for now). The thing we can to is that if we believe our answer is valid (please double check it), we can put that test in the blacklist (with comments on why we do not run it with HiveComparisonTest). Then, we add a query test using queries from that file.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-150061988
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44106/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-150057969
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44103/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/8587#discussion_r43395797
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/AggregationQuerySuite.scala ---
@@ -556,6 +556,33 @@ abstract class AggregationQuerySuite extends QueryTest with SQLTestUtils with Te
Row(0, null, 1, 1, null, 0) :: Nil)
}
+ test("pearson correlation") {
+ val df = Seq.tabulate(10)(i => (1.0 * i, 2.0 * i, i * -1.0)).toDF("a", "b", "c")
+ val corr1 = df.repartition(2).groupBy().agg(corr("a", "b")).collect()(0).getDouble(0)
+ assert(math.abs(corr1 - 1.0) < 1e-12)
+ val corr2 = df.groupBy().agg(corr("a", "c")).collect()(0).getDouble(0)
+ assert(math.abs(corr2 + 1.0) < 1e-12)
+ // non-trivial example. To reproduce in python, use:
+ // >>> from scipy.stats import pearsonr
+ // >>> import numpy as np
+ // >>> a = np.array(range(20))
+ // >>> b = np.array([x * x - 2 * x + 3.5 for x in range(20)])
+ // >>> pearsonr(a, b)
+ // (0.95723391394758572, 3.8902121417802199e-11)
+ // In R, use:
+ // > a <- 0:19
+ // > b <- mapply(function(x) x * x - 2 * x + 3.5, a)
+ // > cor(a, b)
+ // [1] 0.957233913947585835
+ val df2 = Seq.tabulate(20)(x => (1.0 * x, x * x - 2 * x + 3.5)).toDF("a", "b")
+ val corr3 = df2.groupBy().agg(corr("a", "b")).collect()(0).getDouble(0)
+ assert(math.abs(corr3 - 0.95723391394758572) < 1e-12)
+
+ val df3 = Seq.tabulate(0)(i => (1.0 * i, 2.0 * i)).toDF("a", "b")
+ val corr4 = df3.groupBy().agg(corr("a", "b")).collect()(0).getDouble(0)
+ assert(corr4.isNaN)
+ }
--- End diff --
I will add ImplicitCastInputTypes to case class Corr. So the other NumericType can be automatically casting to double.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152605053
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44690/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152372756
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44645/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-150265191
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152679902
retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-150056545
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-137509722
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152415834
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-150265511
**[Test build #44154 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44154/consoleFull)** for PR 8587 at commit [`cc1657b`](https://github.com/apache/spark/commit/cc1657b9cf5da4f07804958420bb94a58c86593f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-150060667
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-151133195
ping @mengxr any other comments?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152688976
retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/8587#discussion_r43404086
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/functions.scala ---
@@ -524,6 +525,133 @@ case class Sum(child: Expression) extends DeclarativeAggregate {
override val evaluateExpression = Cast(currentSum, resultType)
}
+/**
+ * Compute Pearson correlation between two expressions.
+ * When applied on empty data (i.e., count is zero), it returns NaN.
+ *
+ * Definition of Pearson correlation can be found at
+ * http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient
+ *
+ * @param left one of the expressions to compute correlation with.
+ * @param right another expression to compute correlation with.
+ */
+case class Corr(
+ left: Expression,
+ right: Expression,
+ mutableAggBufferOffset: Int = 0,
+ inputAggBufferOffset: Int = 0)
+ extends ImperativeAggregate {
+
+ def children: Seq[Expression] = Seq(left, right)
+
+ def nullable: Boolean = false
+
+ def dataType: DataType = DoubleType
+
+ def inputTypes: Seq[AbstractDataType] = Seq(DoubleType)
--- End diff --
Actually, I was not questioning the return type. My question is the input type.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152473046
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/8587#discussion_r43395619
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/functions.scala ---
@@ -524,6 +525,133 @@ case class Sum(child: Expression) extends DeclarativeAggregate {
override val evaluateExpression = Cast(currentSum, resultType)
}
+/**
+ * Compute Pearson correlation between two expressions.
+ * When applied on empty data (i.e., count is zero), it returns NaN.
+ *
+ * Definition of Pearson correlation can be found at
+ * http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient
+ *
+ * @param left one of the expressions to compute correlation with.
+ * @param right another expression to compute correlation with.
+ */
+case class Corr(
+ left: Expression,
+ right: Expression,
+ mutableAggBufferOffset: Int = 0,
+ inputAggBufferOffset: Int = 0)
+ extends ImperativeAggregate {
+
+ def children: Seq[Expression] = Seq(left, right)
+
+ def nullable: Boolean = false
+
+ def dataType: DataType = DoubleType
+
+ def inputTypes: Seq[AbstractDataType] = Seq(DoubleType)
--- End diff --
For pearson correlation, I think the return data type should be fixed to DoubleType.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/8587#discussion_r43435127
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregates.scala ---
@@ -747,6 +747,19 @@ case class LastFunction(
}
}
+/**
+ * Calculate Pearson Correlation Coefficient for the given columns.
+ * Only support AggregateExpression2.
+ *
+ */
+case class Corr(left: Expression, right: Expression)
+ extends BinaryExpression with AggregateExpression with ImplicitCastInputTypes {
--- End diff --
This is just a place holder, right? Can we change it to `with AggregateExpression1` then we throw an exception (UnsupportedOperatorException) in the `newInstance` method?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152298821
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-150057965
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9298][SQL] Add pearson correlation aggr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8587#issuecomment-152355676
**[Test build #44645 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44645/consoleFull)** for PR 8587 at commit [`3b731e2`](https://github.com/apache/spark/commit/3b731e2c9b08dbade38da73bcff94cf1b2cd7636).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org