You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by attilapiros <gi...@git.apache.org> on 2017/12/21 16:49:05 UTC
[GitHub] spark pull request #20046: Adding unit tests for missing aggregate functions
GitHub user attilapiros opened a pull request:
https://github.com/apache/spark/pull/20046
Adding unit tests for missing aggregate functions
## What changes were proposed in this pull request?
Improving the test coverage of window functions focusing on missing test for window aggregate functions. No new UDAF test is added as it has been tested already.
## How was this patch tested?
Only new tests were added, automated tests were executed.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/attilapiros/spark SPARK-22362
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20046.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20046
----
commit 7b73f572538ad571c85b7c5c6aedf8887550114f
Author: “attilapiros” <pi...@...>
Date: 2017-12-21T16:34:38Z
Adding unit tests for missing aggregate functions
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20046
**[Test build #85310 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85310/testReport)** for PR 20046 at commit [`00f50c0`](https://github.com/apache/spark/commit/00f50c0d99d8b11fe88df5ee9af41becb5ff8d28).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20046
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20046
**[Test build #86474 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86474/testReport)** for PR 20046 at commit [`458a0cc`](https://github.com/apache/spark/commit/458a0ccd7530afeededc52c72bfc38bb83f0bbd1).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20046
**[Test build #86476 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86476/testReport)** for PR 20046 at commit [`a0e14cc`](https://github.com/apache/spark/commit/a0e14cc5ec320df430993f3c2f67c08ce9474163).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by attilapiros <gi...@git.apache.org>.
Github user attilapiros commented on the issue:
https://github.com/apache/spark/pull/20046
gentle ping @gatorsmile @hvanhovell
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20046
**[Test build #89066 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89066/testReport)** for PR 20046 at commit [`93aa930`](https://github.com/apache/spark/commit/93aa930760a3c1a9c38dc4a57928374de63bf1a7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20046
**[Test build #89100 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89100/testReport)** for PR 20046 at commit [`9baef9c`](https://github.com/apache/spark/commit/9baef9cfa34448229cfe3a74029339488d31d21f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20046
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20046
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20046: Adding unit tests for missing aggregate functions
Posted by smurakozi <gi...@git.apache.org>.
Github user smurakozi commented on a diff in the pull request:
https://github.com/apache/spark/pull/20046#discussion_r158326073
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameWindowFunctionsSuite.scala ---
@@ -154,6 +154,217 @@ class DataFrameWindowFunctionsSuite extends QueryTest with SharedSQLContext {
Row(2.0d), Row(2.0d)))
}
+ test("corr, covar_pop, stddev_pop functions in specific window") {
+ val df = Seq(
+ ("a", "p1", 10.0, 20.0),
+ ("b", "p1", 20.0, 10.0),
+ ("c", "p2", 20.0, 20.0),
+ ("d", "p2", 20.0, 20.0),
+ ("e", "p3", 0.0, 0.0),
+ ("f", "p3", 6.0, 12.0),
+ ("g", "p3", 6.0, 12.0),
+ ("h", "p3", 8.0, 16.0),
+ ("i", "p4", 5.0, 5.0)).toDF("key", "partitionId", "value1", "value2")
+ checkAnswer(
+ df.select(
+ $"key",
+ corr("value1", "value2").over(Window.partitionBy("partitionId")
+ .orderBy("key").rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing)),
+ covar_pop("value1", "value2")
+ .over(Window.partitionBy("partitionId")
+ .orderBy("key").rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing)),
+ var_pop("value1")
+ .over(Window.partitionBy("partitionId")
+ .orderBy("key").rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing)),
+ stddev_pop("value1")
+ .over(Window.partitionBy("partitionId")
+ .orderBy("key").rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing)),
+ var_pop("value2")
+ .over(Window.partitionBy("partitionId")
+ .orderBy("key").rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing)),
+ stddev_pop("value2")
+ .over(Window.partitionBy("partitionId")
+ .orderBy("key").rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing))),
+
+ // As stddev_pop(expr) = sqrt(var_pop(expr))
+ // the "stddev_pop" column can be calculated from the "var_pop" column.
+ //
+ // As corr(expr1, expr2) = covar_pop(expr1, expr2) / (stddev_pop(expr1) * stddev_pop(expr2))
+ // the "corr" column can be calculated from the "covar_pop" and the two "stddev_pop" columns.
+ Seq(
+ Row("a", -1.0, -25.0, 25.0, 5.0, 25.0, 5.0),
+ Row("b", -1.0, -25.0, 25.0, 5.0, 25.0, 5.0),
+ Row("c", null, 0.0, 0.0, 0.0, 0.0, 0.0),
+ Row("d", null, 0.0, 0.0, 0.0, 0.0, 0.0),
+ Row("e", 1.0, 18.0, 9.0, 3.0, 36.0, 6.0),
+ Row("f", 1.0, 18.0, 9.0, 3.0, 36.0, 6.0),
+ Row("g", 1.0, 18.0, 9.0, 3.0, 36.0, 6.0),
+ Row("h", 1.0, 18.0, 9.0, 3.0, 36.0, 6.0),
+ Row("i", Double.NaN, 0.0, 0.0, 0.0, 0.0, 0.0)))
+ }
+
+ test("covar_samp, var_samp (variance), stddev_samp (stddev) functions in specific window") {
+ val df = Seq(
+ ("a", "p1", 10.0, 20.0),
+ ("b", "p1", 20.0, 10.0),
+ ("c", "p2", 20.0, 20.0),
+ ("d", "p2", 20.0, 20.0),
+ ("e", "p3", 0.0, 0.0),
+ ("f", "p3", 6.0, 12.0),
+ ("g", "p3", 6.0, 12.0),
+ ("h", "p3", 8.0, 16.0),
+ ("i", "p4", 5.0, 5.0)).toDF("key", "partitionId", "value1", "value2")
+ checkAnswer(
+ df.select(
+ $"key",
+ covar_samp("value1", "value2").over(Window.partitionBy("partitionId")
+ .orderBy("key").rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing)),
+ var_samp("value1").over(Window.partitionBy("partitionId")
+ .orderBy("key").rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing)),
+ variance("value1").over(Window.partitionBy("partitionId")
+ .orderBy("key").rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing)),
+ stddev_samp("value1").over(Window.partitionBy("partitionId")
+ .orderBy("key").rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing)),
+ stddev("value1").over(Window.partitionBy("partitionId")
+ .orderBy("key").rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing))
+ ),
+ Seq(
+ Row("a", -50.0, 50.0, 50.0, 7.0710678118654755, 7.0710678118654755),
+ Row("b", -50.0, 50.0, 50.0, 7.0710678118654755, 7.0710678118654755),
+ Row("c", 0.0, 0.0, 0.0, 0.0, 0.0 ),
+ Row("d", 0.0, 0.0, 0.0, 0.0, 0.0 ),
+ Row("e", 24.0, 12.0, 12.0, 3.4641016151377544, 3.4641016151377544 ),
+ Row("f", 24.0, 12.0, 12.0, 3.4641016151377544, 3.4641016151377544 ),
+ Row("g", 24.0, 12.0, 12.0, 3.4641016151377544, 3.4641016151377544 ),
+ Row("h", 24.0, 12.0, 12.0, 3.4641016151377544, 3.4641016151377544 ),
+ Row("i", Double.NaN, Double.NaN, Double.NaN, Double.NaN, Double.NaN)))
+ }
+
+ test("collect_list in ascending ordered window") {
+ val df = Seq(
+ ("a", "p1", "1"),
+ ("b", "p1", "2"),
+ ("c", "p1", "2"),
+ ("d", "p1", null),
+ ("e", "p1", "3"),
+ ("f", "p2", "10"),
+ ("g", "p2", "11"),
+ ("h", "p3", "20"),
+ ("i", "p4", null)).toDF("key", "partition", "value")
+ checkAnswer(
+ df.select(
+ $"key",
+ collect_list("value").over(Window.partitionBy($"partition").orderBy($"value")
+ .rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing))),
+ Seq(
+ Row("a", Array("1", "2", "2", "3")),
+ Row("b", Array("1", "2", "2", "3")),
+ Row("c", Array("1", "2", "2", "3")),
+ Row("d", Array("1", "2", "2", "3")),
+ Row("e", Array("1", "2", "2", "3")),
+ Row("f", Array("10", "11")),
+ Row("g", Array("10", "11")),
+ Row("h", Array("20")),
+ Row("i", Array())))
+ }
+
+ test("collect_list in descending ordered window") {
+ val df = Seq(
+ ("a", "p1", "1"),
+ ("b", "p1", "2"),
+ ("c", "p1", "2"),
+ ("d", "p1", null),
+ ("e", "p1", "3"),
+ ("f", "p2", "10"),
+ ("g", "p2", "11"),
+ ("h", "p3", "20"),
+ ("i", "p4", null)).toDF("key", "partition", "value")
+ checkAnswer(
+ df.select(
+ $"key",
+ collect_list("value").over(Window.partitionBy($"partition").orderBy($"value".desc)
+ .rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing))),
+ Seq(
+ Row("a", Array("3", "2", "2", "1")),
+ Row("b", Array("3", "2", "2", "1")),
+ Row("c", Array("3", "2", "2", "1")),
+ Row("d", Array("3", "2", "2", "1")),
+ Row("e", Array("3", "2", "2", "1")),
+ Row("f", Array("11", "10")),
+ Row("g", Array("11", "10")),
+ Row("h", Array("20")),
+ Row("i", Array())))
+ }
+
+ test("collect_set in window") {
+ val df = Seq(
+ ("a", "p1", 1),
+ ("b", "p1", 2),
+ ("c", "p1", 2),
+ ("d", "p1", 3),
+ ("e", "p1", 3),
+ ("f", "p2", 10),
+ ("g", "p2", 11),
+ ("h", "p3", 20)).toDF("key", "partition", "value")
+ checkAnswer(
+ df.select(
+ $"key",
+ collect_set("value").over(Window.partitionBy($"partition").orderBy($"value")
+ .rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing))),
+ Seq(
+ Row("a", Array(1, 2, 3)),
+ Row("b", Array(1, 2, 3)),
+ Row("c", Array(1, 2, 3)),
+ Row("d", Array(1, 2, 3)),
+ Row("e", Array(1, 2, 3)),
+ Row("f", Array(10, 11)),
+ Row("g", Array(10, 11)),
+ Row("h", Array(20))))
+ }
+
+ test("skewness and kurtosis functions in window") {
+ val df = Seq(
+ ("a", "p1", 1.0),
+ ("b", "p1", 1.0),
+ ("c", "p1", 2.0),
+ ("d", "p1", 2.0),
+ ("e", "p1", 3.0),
+ ("f", "p1", 3.0),
+ ("g", "p1", 3.0),
+ ("h", "p2", 1.0),
+ ("i", "p2", 2.0),
+ ("j", "p2", 5.0)).toDF("key", "partition", "value")
+ checkAnswer(
+ df.select(
+ $"key",
+ skewness("value").over(Window.partitionBy("partition").orderBy($"key")
+ .rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing)),
+ kurtosis("value").over(Window.partitionBy("partition").orderBy($"key")
+ .rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing))),
+ // results are checked by scipy.stats.skew() and scipy.stats.kurtosis()
+ Seq(
+ Row("a", -0.27238010581457267, -1.506920415224914),
+ Row("b", -0.27238010581457267, -1.506920415224914),
+ Row("c", -0.27238010581457267, -1.506920415224914),
+ Row("d", -0.27238010581457267, -1.506920415224914),
+ Row("e", -0.27238010581457267, -1.506920415224914),
+ Row("f", -0.27238010581457267, -1.506920415224914),
+ Row("g", -0.27238010581457267, -1.506920415224914),
+ Row("h", 0.5280049792181881, -1.5000000000000013),
+ Row("i", 0.5280049792181881, -1.5000000000000013),
+ Row("j", 0.5280049792181881, -1.5000000000000013)))
+ }
+
+ test("aggregation function on invalid column") {
+ val df = Seq((1, "1")).toDF("key", "value")
+ val e = intercept[AnalysisException](
+ df.select(
+ $"key",
+ count("invalid").over(
--- End diff --
`over()` would be enough, as partition and orderBy is ignored anyway
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20046: [SPARK-22362][SQL] Add unit test for Window Aggre...
Posted by attilapiros <gi...@git.apache.org>.
Github user attilapiros commented on a diff in the pull request:
https://github.com/apache/spark/pull/20046#discussion_r158456767
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameWindowFunctionsSuite.scala ---
@@ -154,6 +154,217 @@ class DataFrameWindowFunctionsSuite extends QueryTest with SharedSQLContext {
Row(2.0d), Row(2.0d)))
}
+ test("corr, covar_pop, stddev_pop functions in specific window") {
+ val df = Seq(
+ ("a", "p1", 10.0, 20.0),
+ ("b", "p1", 20.0, 10.0),
+ ("c", "p2", 20.0, 20.0),
+ ("d", "p2", 20.0, 20.0),
+ ("e", "p3", 0.0, 0.0),
+ ("f", "p3", 6.0, 12.0),
+ ("g", "p3", 6.0, 12.0),
+ ("h", "p3", 8.0, 16.0),
+ ("i", "p4", 5.0, 5.0)).toDF("key", "partitionId", "value1", "value2")
+ checkAnswer(
+ df.select(
+ $"key",
+ corr("value1", "value2").over(Window.partitionBy("partitionId")
+ .orderBy("key").rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing)),
+ covar_pop("value1", "value2")
+ .over(Window.partitionBy("partitionId")
+ .orderBy("key").rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing)),
+ var_pop("value1")
+ .over(Window.partitionBy("partitionId")
+ .orderBy("key").rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing)),
+ stddev_pop("value1")
+ .over(Window.partitionBy("partitionId")
+ .orderBy("key").rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing)),
+ var_pop("value2")
+ .over(Window.partitionBy("partitionId")
+ .orderBy("key").rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing)),
+ stddev_pop("value2")
+ .over(Window.partitionBy("partitionId")
+ .orderBy("key").rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing))),
+
+ // As stddev_pop(expr) = sqrt(var_pop(expr))
+ // the "stddev_pop" column can be calculated from the "var_pop" column.
+ //
+ // As corr(expr1, expr2) = covar_pop(expr1, expr2) / (stddev_pop(expr1) * stddev_pop(expr2))
+ // the "corr" column can be calculated from the "covar_pop" and the two "stddev_pop" columns.
+ Seq(
+ Row("a", -1.0, -25.0, 25.0, 5.0, 25.0, 5.0),
+ Row("b", -1.0, -25.0, 25.0, 5.0, 25.0, 5.0),
+ Row("c", null, 0.0, 0.0, 0.0, 0.0, 0.0),
+ Row("d", null, 0.0, 0.0, 0.0, 0.0, 0.0),
+ Row("e", 1.0, 18.0, 9.0, 3.0, 36.0, 6.0),
+ Row("f", 1.0, 18.0, 9.0, 3.0, 36.0, 6.0),
+ Row("g", 1.0, 18.0, 9.0, 3.0, 36.0, 6.0),
+ Row("h", 1.0, 18.0, 9.0, 3.0, 36.0, 6.0),
+ Row("i", Double.NaN, 0.0, 0.0, 0.0, 0.0, 0.0)))
+ }
+
+ test("covar_samp, var_samp (variance), stddev_samp (stddev) functions in specific window") {
+ val df = Seq(
+ ("a", "p1", 10.0, 20.0),
+ ("b", "p1", 20.0, 10.0),
+ ("c", "p2", 20.0, 20.0),
+ ("d", "p2", 20.0, 20.0),
+ ("e", "p3", 0.0, 0.0),
+ ("f", "p3", 6.0, 12.0),
+ ("g", "p3", 6.0, 12.0),
+ ("h", "p3", 8.0, 16.0),
+ ("i", "p4", 5.0, 5.0)).toDF("key", "partitionId", "value1", "value2")
+ checkAnswer(
+ df.select(
+ $"key",
+ covar_samp("value1", "value2").over(Window.partitionBy("partitionId")
+ .orderBy("key").rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing)),
+ var_samp("value1").over(Window.partitionBy("partitionId")
+ .orderBy("key").rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing)),
+ variance("value1").over(Window.partitionBy("partitionId")
+ .orderBy("key").rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing)),
+ stddev_samp("value1").over(Window.partitionBy("partitionId")
+ .orderBy("key").rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing)),
+ stddev("value1").over(Window.partitionBy("partitionId")
+ .orderBy("key").rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing))
+ ),
+ Seq(
+ Row("a", -50.0, 50.0, 50.0, 7.0710678118654755, 7.0710678118654755),
+ Row("b", -50.0, 50.0, 50.0, 7.0710678118654755, 7.0710678118654755),
+ Row("c", 0.0, 0.0, 0.0, 0.0, 0.0 ),
+ Row("d", 0.0, 0.0, 0.0, 0.0, 0.0 ),
+ Row("e", 24.0, 12.0, 12.0, 3.4641016151377544, 3.4641016151377544 ),
+ Row("f", 24.0, 12.0, 12.0, 3.4641016151377544, 3.4641016151377544 ),
+ Row("g", 24.0, 12.0, 12.0, 3.4641016151377544, 3.4641016151377544 ),
+ Row("h", 24.0, 12.0, 12.0, 3.4641016151377544, 3.4641016151377544 ),
+ Row("i", Double.NaN, Double.NaN, Double.NaN, Double.NaN, Double.NaN)))
+ }
+
+ test("collect_list in ascending ordered window") {
+ val df = Seq(
+ ("a", "p1", "1"),
+ ("b", "p1", "2"),
+ ("c", "p1", "2"),
+ ("d", "p1", null),
+ ("e", "p1", "3"),
+ ("f", "p2", "10"),
+ ("g", "p2", "11"),
+ ("h", "p3", "20"),
+ ("i", "p4", null)).toDF("key", "partition", "value")
+ checkAnswer(
+ df.select(
+ $"key",
+ collect_list("value").over(Window.partitionBy($"partition").orderBy($"value")
+ .rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing))),
+ Seq(
+ Row("a", Array("1", "2", "2", "3")),
+ Row("b", Array("1", "2", "2", "3")),
+ Row("c", Array("1", "2", "2", "3")),
+ Row("d", Array("1", "2", "2", "3")),
+ Row("e", Array("1", "2", "2", "3")),
+ Row("f", Array("10", "11")),
+ Row("g", Array("10", "11")),
+ Row("h", Array("20")),
+ Row("i", Array())))
+ }
+
+ test("collect_list in descending ordered window") {
+ val df = Seq(
+ ("a", "p1", "1"),
+ ("b", "p1", "2"),
+ ("c", "p1", "2"),
+ ("d", "p1", null),
+ ("e", "p1", "3"),
+ ("f", "p2", "10"),
+ ("g", "p2", "11"),
+ ("h", "p3", "20"),
+ ("i", "p4", null)).toDF("key", "partition", "value")
+ checkAnswer(
+ df.select(
+ $"key",
+ collect_list("value").over(Window.partitionBy($"partition").orderBy($"value".desc)
+ .rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing))),
+ Seq(
+ Row("a", Array("3", "2", "2", "1")),
+ Row("b", Array("3", "2", "2", "1")),
+ Row("c", Array("3", "2", "2", "1")),
+ Row("d", Array("3", "2", "2", "1")),
+ Row("e", Array("3", "2", "2", "1")),
+ Row("f", Array("11", "10")),
+ Row("g", Array("11", "10")),
+ Row("h", Array("20")),
+ Row("i", Array())))
+ }
+
+ test("collect_set in window") {
+ val df = Seq(
+ ("a", "p1", 1),
+ ("b", "p1", 2),
+ ("c", "p1", 2),
+ ("d", "p1", 3),
+ ("e", "p1", 3),
+ ("f", "p2", 10),
+ ("g", "p2", 11),
+ ("h", "p3", 20)).toDF("key", "partition", "value")
+ checkAnswer(
+ df.select(
+ $"key",
+ collect_set("value").over(Window.partitionBy($"partition").orderBy($"value")
+ .rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing))),
+ Seq(
+ Row("a", Array(1, 2, 3)),
+ Row("b", Array(1, 2, 3)),
+ Row("c", Array(1, 2, 3)),
+ Row("d", Array(1, 2, 3)),
+ Row("e", Array(1, 2, 3)),
+ Row("f", Array(10, 11)),
+ Row("g", Array(10, 11)),
+ Row("h", Array(20))))
+ }
+
+ test("skewness and kurtosis functions in window") {
+ val df = Seq(
+ ("a", "p1", 1.0),
+ ("b", "p1", 1.0),
+ ("c", "p1", 2.0),
+ ("d", "p1", 2.0),
+ ("e", "p1", 3.0),
+ ("f", "p1", 3.0),
+ ("g", "p1", 3.0),
+ ("h", "p2", 1.0),
+ ("i", "p2", 2.0),
+ ("j", "p2", 5.0)).toDF("key", "partition", "value")
+ checkAnswer(
+ df.select(
+ $"key",
+ skewness("value").over(Window.partitionBy("partition").orderBy($"key")
+ .rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing)),
+ kurtosis("value").over(Window.partitionBy("partition").orderBy($"key")
+ .rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing))),
+ // results are checked by scipy.stats.skew() and scipy.stats.kurtosis()
+ Seq(
+ Row("a", -0.27238010581457267, -1.506920415224914),
+ Row("b", -0.27238010581457267, -1.506920415224914),
+ Row("c", -0.27238010581457267, -1.506920415224914),
+ Row("d", -0.27238010581457267, -1.506920415224914),
+ Row("e", -0.27238010581457267, -1.506920415224914),
+ Row("f", -0.27238010581457267, -1.506920415224914),
+ Row("g", -0.27238010581457267, -1.506920415224914),
+ Row("h", 0.5280049792181881, -1.5000000000000013),
+ Row("i", 0.5280049792181881, -1.5000000000000013),
+ Row("j", 0.5280049792181881, -1.5000000000000013)))
+ }
+
+ test("aggregation function on invalid column") {
+ val df = Seq((1, "1")).toDF("key", "value")
+ val e = intercept[AnalysisException](
+ df.select(
+ $"key",
+ count("invalid").over(
--- End diff --
Thanks, I will remove the unnecessary parts.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20046
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85610/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20046
**[Test build #85599 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85599/testReport)** for PR 20046 at commit [`4dd08dc`](https://github.com/apache/spark/commit/4dd08dc3c6d37c322f2ce481c9898d829ca827c6).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: Adding unit tests for missing aggregate functions
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20046
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20046
**[Test build #86362 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86362/testReport)** for PR 20046 at commit [`adab4da`](https://github.com/apache/spark/commit/adab4da3adfb02446d5ee8ccb635d7af0ba42810).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20046
**[Test build #86352 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86352/testReport)** for PR 20046 at commit [`adab4da`](https://github.com/apache/spark/commit/adab4da3adfb02446d5ee8ccb635d7af0ba42810).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20046: [SPARK-22362][SQL] Add unit test for Window Aggre...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/20046#discussion_r171190314
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameWindowFunctionsSuite.scala ---
@@ -33,6 +35,11 @@ import org.apache.spark.unsafe.types.CalendarInterval
class DataFrameWindowFunctionsSuite extends QueryTest with SharedSQLContext {
import testImplicits._
+ private def sortWrappedArrayInRow(d: DataFrame) = d.map {
+ case Row(key: String, unsorted: mutable.WrappedArray[String]) =>
--- End diff --
Let's not pattern match against `mutable.WrappedArray` and use `Seq` instead. `mutable.WrappedArray` is pretty much an implementation detail, and pattern matching against it is brittle.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20046
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86476/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by attilapiros <gi...@git.apache.org>.
Github user attilapiros commented on the issue:
https://github.com/apache/spark/pull/20046
I have already extended sql/core/src/test/resources/sql-tests/inputs/window.sql with the missing window aggregate functions but if you would like I can move it to a different PR too.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20046: [SPARK-22362][SQL] Add unit test for Window Aggre...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/20046
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20046
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85276/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by attilapiros <gi...@git.apache.org>.
Github user attilapiros commented on the issue:
https://github.com/apache/spark/pull/20046
PySpark failure must be unrelated as only unit tests are added.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20046
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89066/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20046
**[Test build #86472 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86472/testReport)** for PR 20046 at commit [`5c941c7`](https://github.com/apache/spark/commit/5c941c7e47e0f8782c97da6765465d85a66345e5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20046
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86352/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20046
**[Test build #85311 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85311/testReport)** for PR 20046 at commit [`7290ec4`](https://github.com/apache/spark/commit/7290ec42de93cc497c795acb4314c1721a504209).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20046
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20046
**[Test build #86352 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86352/testReport)** for PR 20046 at commit [`adab4da`](https://github.com/apache/spark/commit/adab4da3adfb02446d5ee8ccb635d7af0ba42810).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20046
**[Test build #85311 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85311/testReport)** for PR 20046 at commit [`7290ec4`](https://github.com/apache/spark/commit/7290ec42de93cc497c795acb4314c1721a504209).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20046
**[Test build #86476 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86476/testReport)** for PR 20046 at commit [`a0e14cc`](https://github.com/apache/spark/commit/a0e14cc5ec320df430993f3c2f67c08ce9474163).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20046
**[Test build #89066 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89066/testReport)** for PR 20046 at commit [`93aa930`](https://github.com/apache/spark/commit/93aa930760a3c1a9c38dc4a57928374de63bf1a7).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by attilapiros <gi...@git.apache.org>.
Github user attilapiros commented on the issue:
https://github.com/apache/spark/pull/20046
cc @gatorsmile @hvanhovell
@jiangxb1987 your review comments are applied, is there something else I should work on regarding this PR?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20046
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85310/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20046
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86474/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20046
**[Test build #85601 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85601/testReport)** for PR 20046 at commit [`03d6159`](https://github.com/apache/spark/commit/03d61592855827be29cfa0e6127fe5cd55a29213).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20046
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20046
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20046
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20046
**[Test build #85276 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85276/testReport)** for PR 20046 at commit [`7b73f57`](https://github.com/apache/spark/commit/7b73f572538ad571c85b7c5c6aedf8887550114f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20046
**[Test build #85610 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85610/testReport)** for PR 20046 at commit [`caafba7`](https://github.com/apache/spark/commit/caafba7c31329a3a85eeae11cdebf9bff9053e3a).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by smurakozi <gi...@git.apache.org>.
Github user smurakozi commented on the issue:
https://github.com/apache/spark/pull/20046
@jiangxb1987 how does your request to cover the sql interface relates to SPARK-23160?
I assume it is to be covered in that issue, is that correct?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20046
**[Test build #85276 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85276/testReport)** for PR 20046 at commit [`7b73f57`](https://github.com/apache/spark/commit/7b73f572538ad571c85b7c5c6aedf8887550114f).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20046
**[Test build #86474 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86474/testReport)** for PR 20046 at commit [`458a0cc`](https://github.com/apache/spark/commit/458a0ccd7530afeededc52c72bfc38bb83f0bbd1).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/20046
LGTM - merging to master. Thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20046
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89100/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by attilapiros <gi...@git.apache.org>.
Github user attilapiros commented on the issue:
https://github.com/apache/spark/pull/20046
cc @gatorsmile @hvanhovell @jiangxb1987
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20046
**[Test build #85601 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85601/testReport)** for PR 20046 at commit [`03d6159`](https://github.com/apache/spark/commit/03d61592855827be29cfa0e6127fe5cd55a29213).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20046
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86472/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20046
**[Test build #85310 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85310/testReport)** for PR 20046 at commit [`00f50c0`](https://github.com/apache/spark/commit/00f50c0d99d8b11fe88df5ee9af41becb5ff8d28).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20046: [SPARK-22362][SQL] Add unit test for Window Aggre...
Posted by attilapiros <gi...@git.apache.org>.
Github user attilapiros commented on a diff in the pull request:
https://github.com/apache/spark/pull/20046#discussion_r162894686
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameWindowFunctionsSuite.scala ---
@@ -86,6 +93,429 @@ class DataFrameWindowFunctionsSuite extends QueryTest with SharedSQLContext {
assert(e.message.contains("requires window to be ordered"))
}
+ test("aggregation and rows between") {
+ val df = Seq((1, "1"), (2, "1"), (2, "2"), (1, "1"), (2, "2")).toDF("key", "value")
--- End diff --
This tests was removed and re-added as result of merge conflict. Now I cleaned up.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20046
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20046
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20046
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86362/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20046
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20046: [SPARK-22362][SQL] Add unit test for Window Aggre...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20046#discussion_r162759216
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameWindowFunctionsSuite.scala ---
@@ -86,6 +93,429 @@ class DataFrameWindowFunctionsSuite extends QueryTest with SharedSQLContext {
assert(e.message.contains("requires window to be ordered"))
}
+ test("aggregation and rows between") {
+ val df = Seq((1, "1"), (2, "1"), (2, "2"), (1, "1"), (2, "2")).toDF("key", "value")
--- End diff --
We shall also include null data.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by attilapiros <gi...@git.apache.org>.
Github user attilapiros commented on the issue:
https://github.com/apache/spark/pull/20046
@hvanhovell I would like to ask you to take another quick glance to these change
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20046
**[Test build #86362 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86362/testReport)** for PR 20046 at commit [`adab4da`](https://github.com/apache/spark/commit/adab4da3adfb02446d5ee8ccb635d7af0ba42810).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by attilapiros <gi...@git.apache.org>.
Github user attilapiros commented on the issue:
https://github.com/apache/spark/pull/20046
gentle reminder @hvanhovell
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20046
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the issue:
https://github.com/apache/spark/pull/20046
Jenkins, ok to test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20046
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20046
**[Test build #85610 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85610/testReport)** for PR 20046 at commit [`caafba7`](https://github.com/apache/spark/commit/caafba7c31329a3a85eeae11cdebf9bff9053e3a).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20046
**[Test build #89100 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89100/testReport)** for PR 20046 at commit [`9baef9c`](https://github.com/apache/spark/commit/9baef9cfa34448229cfe3a74029339488d31d21f).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20046
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85311/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20046
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20046
**[Test build #86472 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86472/testReport)** for PR 20046 at commit [`5c941c7`](https://github.com/apache/spark/commit/5c941c7e47e0f8782c97da6765465d85a66345e5).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by attilapiros <gi...@git.apache.org>.
Github user attilapiros commented on the issue:
https://github.com/apache/spark/pull/20046
cc @gatorsmile @hvanhovell @jiangxb1987
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20046
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85599/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/20046
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20046: [SPARK-22362][SQL] Add unit test for Window Aggre...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/20046#discussion_r171190560
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameWindowFunctionsSuite.scala ---
@@ -33,6 +35,11 @@ import org.apache.spark.unsafe.types.CalendarInterval
class DataFrameWindowFunctionsSuite extends QueryTest with SharedSQLContext {
import testImplicits._
+ private def sortWrappedArrayInRow(d: DataFrame) = d.map {
--- End diff --
You can also just use the `array_sort` function. That is probably a lot cheaper.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by attilapiros <gi...@git.apache.org>.
Github user attilapiros commented on the issue:
https://github.com/apache/spark/pull/20046
ping @hvanhovell @cloud-fan
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20046
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85601/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/20046
We shall also cover the sql interface, you can find some example in `sql/core/src/test/resources/sql-tests/inputs/udaf.sql`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20046
**[Test build #85599 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85599/testReport)** for PR 20046 at commit [`4dd08dc`](https://github.com/apache/spark/commit/4dd08dc3c6d37c322f2ce481c9898d829ca827c6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20046
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20046
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org