You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by cloud-fan <gi...@git.apache.org> on 2016/12/26 10:44:12 UTC
[GitHub] spark pull request #16404: [SPARK-18969][SQL] Support grouping by nondetermi...
GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/16404
[SPARK-18969][SQL] Support grouping by nondeterministic expressions
## What changes were proposed in this pull request?
Currently nondeterministic expressions are allowed in `Aggregate`(see the [comment](https://github.com/apache/spark/blob/v2.0.2/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala#L249-L251)), but the `PullOutNondeterministic` analyzer rule failed to handle `Aggregate`, this PR fixes it.
## How was this patch tested?
a new test suite
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark groupby
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16404.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16404
----
commit 67f032b6ab7b23aecf432b13a6f455582754fc98
Author: Wenchen Fan <we...@databricks.com>
Date: 2016-12-26T10:24:07Z
Support grouping by nondeterministic expressions
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16404
**[Test build #70597 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70597/testReport)** for PR 16404 at commit [`67f032b`](https://github.com/apache/spark/commit/67f032b6ab7b23aecf432b13a6f455582754fc98).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16404: [SPARK-18969][SQL] Support grouping by nondetermi...
Posted by cloud-fan <gi...@git.apache.org>.
GitHub user cloud-fan reopened a pull request:
https://github.com/apache/spark/pull/16404
[SPARK-18969][SQL] Support grouping by nondeterministic expressions
## What changes were proposed in this pull request?
Currently nondeterministic expressions are allowed in `Aggregate`(see the [comment](https://github.com/apache/spark/blob/v2.0.2/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala#L249-L251)), but the `PullOutNondeterministic` analyzer rule failed to handle `Aggregate`, this PR fixes it.
close https://github.com/apache/spark/pull/16379
## How was this patch tested?
a new test suite
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark groupby
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16404.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16404
----
commit f1451883df9077ecbf31f3a86d2427b60262f863
Author: Wenchen Fan <we...@databricks.com>
Date: 2016-12-26T10:24:07Z
Support grouping by nondeterministic expressions
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16404
**[Test build #70754 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70754/testReport)** for PR 16404 at commit [`f145188`](https://github.com/apache/spark/commit/f1451883df9077ecbf31f3a86d2427b60262f863).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16404: [SPARK-18969][SQL] Support grouping by nondetermi...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16404#discussion_r93998144
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -1918,28 +1918,37 @@ class Analyzer(
case p: Project => p
case f: Filter => f
+ case a: Aggregate if a.groupingExpressions.exists(!_.deterministic) =>
+ val nondeterToAttr = getNondeterToAttr(a.groupingExpressions)
+ val newChild = Project(a.child.output ++ nondeterToAttr.values, a.child)
+ a.transformExpressions { case e =>
+ nondeterToAttr.get(e).map(_.toAttribute).getOrElse(e)
+ }.copy(child = newChild)
+
// todo: It's hard to write a general rule to pull out nondeterministic expressions
// from LogicalPlan, currently we only do it for UnaryNode which has same output
// schema with its child.
case p: UnaryNode if p.output == p.child.output && p.expressions.exists(!_.deterministic) =>
- val nondeterministicExprs = p.expressions.filterNot(_.deterministic).flatMap { expr =>
- val leafNondeterministic = expr.collect {
- case n: Nondeterministic => n
- }
- leafNondeterministic.map { e =>
- val ne = e match {
- case n: NamedExpression => n
- case _ => Alias(e, "_nondeterministic")(isGenerated = true)
- }
- new TreeNodeRef(e) -> ne
- }
- }.toMap
+ val nondeterToAttr = getNondeterToAttr(p.expressions)
val newPlan = p.transformExpressions { case e =>
- nondeterministicExprs.get(new TreeNodeRef(e)).map(_.toAttribute).getOrElse(e)
+ nondeterToAttr.get(e).map(_.toAttribute).getOrElse(e)
}
- val newChild = Project(p.child.output ++ nondeterministicExprs.values, p.child)
+ val newChild = Project(p.child.output ++ nondeterToAttr.values, p.child)
Project(p.output, newPlan.withNewChildren(newChild :: Nil))
}
+
+ private def getNondeterToAttr(exprs: Seq[Expression]): Map[Expression, NamedExpression] = {
+ exprs.filterNot(_.deterministic).flatMap { expr =>
+ val leafNondeterministic = expr.collect { case n: Nondeterministic => n }
--- End diff --
`statefulUDF()` is a stateful/non-deterministic UDF which does not exend `Nondeterministic`, but its `deterministic` is equal to `false`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16404
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70753/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16404
**[Test build #70597 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70597/testReport)** for PR 16404 at commit [`67f032b`](https://github.com/apache/spark/commit/67f032b6ab7b23aecf432b13a6f455582754fc98).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait Nondeterministic extends LeafExpression `
* `abstract class RDG extends LeafExpression with ExpectsInputTypes with Nondeterministic `
* `case class Rand(seedExpr: Expression) extends RDG `
* `case class Randn(seedExpr: Expression) extends RDG `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16404: [SPARK-18969][SQL] Support grouping by nondetermi...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16404#discussion_r94229396
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -1918,28 +1918,37 @@ class Analyzer(
case p: Project => p
case f: Filter => f
+ case a: Aggregate if a.groupingExpressions.exists(!_.deterministic) =>
+ val nondeterToAttr = getNondeterToAttr(a.groupingExpressions)
+ val newChild = Project(a.child.output ++ nondeterToAttr.values, a.child)
+ a.transformExpressions { case e =>
+ nondeterToAttr.get(e).map(_.toAttribute).getOrElse(e)
+ }.copy(child = newChild)
+
// todo: It's hard to write a general rule to pull out nondeterministic expressions
// from LogicalPlan, currently we only do it for UnaryNode which has same output
// schema with its child.
case p: UnaryNode if p.output == p.child.output && p.expressions.exists(!_.deterministic) =>
- val nondeterministicExprs = p.expressions.filterNot(_.deterministic).flatMap { expr =>
- val leafNondeterministic = expr.collect {
- case n: Nondeterministic => n
- }
- leafNondeterministic.map { e =>
- val ne = e match {
- case n: NamedExpression => n
- case _ => Alias(e, "_nondeterministic")(isGenerated = true)
- }
- new TreeNodeRef(e) -> ne
- }
- }.toMap
+ val nondeterToAttr = getNondeterToAttr(p.expressions)
val newPlan = p.transformExpressions { case e =>
- nondeterministicExprs.get(new TreeNodeRef(e)).map(_.toAttribute).getOrElse(e)
+ nondeterToAttr.get(e).map(_.toAttribute).getOrElse(e)
}
- val newChild = Project(p.child.output ++ nondeterministicExprs.values, p.child)
+ val newChild = Project(p.child.output ++ nondeterToAttr.values, p.child)
Project(p.output, newPlan.withNewChildren(newChild :: Nil))
}
+
+ private def getNondeterToAttr(exprs: Seq[Expression]): Map[Expression, NamedExpression] = {
+ exprs.filterNot(_.deterministic).flatMap { expr =>
+ val leafNondeterministic = expr.collect { case n: Nondeterministic => n }
--- End diff --
this problem was already there, let's send a new PR to fix it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16404
LGTM cc @rxin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16404
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16404
**[Test build #71244 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71244/testReport)** for PR 16404 at commit [`f145188`](https://github.com/apache/spark/commit/f1451883df9077ecbf31f3a86d2427b60262f863).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16404
updated. Merging to master!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16404: [SPARK-18969][SQL] Support grouping by nondetermi...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16404#discussion_r93921258
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -1918,28 +1918,37 @@ class Analyzer(
case p: Project => p
case f: Filter => f
+ case a: Aggregate if a.groupingExpressions.exists(!_.deterministic) =>
+ val nondeterToAttr = getNondeterToAttr(a.groupingExpressions)
+ val newChild = Project(a.child.output ++ nondeterToAttr.values, a.child)
+ a.transformExpressions { case e =>
+ nondeterToAttr.get(e).map(_.toAttribute).getOrElse(e)
+ }.copy(child = newChild)
+
// todo: It's hard to write a general rule to pull out nondeterministic expressions
// from LogicalPlan, currently we only do it for UnaryNode which has same output
// schema with its child.
case p: UnaryNode if p.output == p.child.output && p.expressions.exists(!_.deterministic) =>
--- End diff --
to narrow down the scope of the affected operators, but ideally we should use a white-list
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16404: [SPARK-18969][SQL] Support grouping by nondetermi...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16404#discussion_r93942716
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -1918,28 +1918,37 @@ class Analyzer(
case p: Project => p
case f: Filter => f
+ case a: Aggregate if a.groupingExpressions.exists(!_.deterministic) =>
+ val nondeterToAttr = getNondeterToAttr(a.groupingExpressions)
+ val newChild = Project(a.child.output ++ nondeterToAttr.values, a.child)
+ a.transformExpressions { case e =>
+ nondeterToAttr.get(e).map(_.toAttribute).getOrElse(e)
+ }.copy(child = newChild)
+
// todo: It's hard to write a general rule to pull out nondeterministic expressions
// from LogicalPlan, currently we only do it for UnaryNode which has same output
// schema with its child.
case p: UnaryNode if p.output == p.child.output && p.expressions.exists(!_.deterministic) =>
--- End diff --
Thanks! I see
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16404
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16404: [SPARK-18969][SQL] Support grouping by nondetermi...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16404#discussion_r93921170
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -1918,28 +1918,37 @@ class Analyzer(
case p: Project => p
case f: Filter => f
+ case a: Aggregate if a.groupingExpressions.exists(!_.deterministic) =>
+ val nondeterToAttr = getNondeterToAttr(a.groupingExpressions)
+ val newChild = Project(a.child.output ++ nondeterToAttr.values, a.child)
+ a.transformExpressions { case e =>
+ nondeterToAttr.get(e).map(_.toAttribute).getOrElse(e)
+ }.copy(child = newChild)
+
// todo: It's hard to write a general rule to pull out nondeterministic expressions
// from LogicalPlan, currently we only do it for UnaryNode which has same output
// schema with its child.
case p: UnaryNode if p.output == p.child.output && p.expressions.exists(!_.deterministic) =>
- val nondeterministicExprs = p.expressions.filterNot(_.deterministic).flatMap { expr =>
- val leafNondeterministic = expr.collect {
- case n: Nondeterministic => n
- }
- leafNondeterministic.map { e =>
- val ne = e match {
- case n: NamedExpression => n
- case _ => Alias(e, "_nondeterministic")(isGenerated = true)
- }
- new TreeNodeRef(e) -> ne
- }
- }.toMap
+ val nondeterToAttr = getNondeterToAttr(p.expressions)
val newPlan = p.transformExpressions { case e =>
- nondeterministicExprs.get(new TreeNodeRef(e)).map(_.toAttribute).getOrElse(e)
+ nondeterToAttr.get(e).map(_.toAttribute).getOrElse(e)
}
- val newChild = Project(p.child.output ++ nondeterministicExprs.values, p.child)
+ val newChild = Project(p.child.output ++ nondeterToAttr.values, p.child)
Project(p.output, newPlan.withNewChildren(newChild :: Nil))
}
+
+ private def getNondeterToAttr(exprs: Seq[Expression]): Map[Expression, NamedExpression] = {
+ exprs.filterNot(_.deterministic).flatMap { expr =>
+ val leafNondeterministic = expr.collect { case n: Nondeterministic => n }
--- End diff --
1. this is same with the previous behavior
2. according to the variable name: `leafNondeterministic`, it seems reasonable to collect `Nondeterministic` here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16404
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16404
```Scala
sql("select a + rand() from testData2 group by a, a + rand()").explain(true)
```
After we merging this PR, I am afraid we might hitting a common misunderstanding. Users might assume `a + rand()` in the aggregate and group by are referring to the same expression. However, the non-deterministic expressions `rand()` in aggregate and group by actually return different results. See the plan
```
(CAST(a AS DOUBLE) + rand(-8319334304660323190)): double
Aggregate [a#23, (cast(a#23 as double) + _nondeterministic#249)], [(cast(a#23 as double) + rand(-8319334304660323190)) AS (CAST(a AS DOUBLE) + rand(-8319334304660323190))#248]
+- Project [a#23, b#24, rand(-993354445377164024) AS _nondeterministic#249]
+- SubqueryAlias testdata2, `testData2`
+- SerializeFromObject [assertnotnull(assertnotnull(input[0, org.apache.spark.sql.test.SQLTestData$TestData2, true], top level Product input object), - root class: "org.apache.spark.sql.test.SQLTestData.TestData2").a AS a#23, assertnotnull(assertnotnull(input[0, org.apache.spark.sql.test.SQLTestData$TestData2, true], top level Product input object), - root class: "org.apache.spark.sql.test.SQLTestData.TestData2").b AS b#24]
+- ExternalRDD [obj#22]
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16404: [SPARK-18969][SQL] Support grouping by nondetermi...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan closed the pull request at:
https://github.com/apache/spark/pull/16404
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16404
cc @rxin @gatorsmile
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16404
**[Test build #70617 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70617/testReport)** for PR 16404 at commit [`f145188`](https://github.com/apache/spark/commit/f1451883df9077ecbf31f3a86d2427b60262f863).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16404
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16404
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70754/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16404
**[Test build #71244 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71244/testReport)** for PR 16404 at commit [`f145188`](https://github.com/apache/spark/commit/f1451883df9077ecbf31f3a86d2427b60262f863).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16404
postgres handles `rand` in aggregate specially
```
cloud=# select random(), random();
random | random
-------------------+-------------------
0.627067061141133 | 0.324854184873402
(1 row)
cloud=# select random(), random() from (select 1) t group by random();
random | random
-------------------+-------------------
0.459220345597714 | 0.459220345597714
(1 row)
```
I think we can follow postgres, to use a fixed seed for `rand` in `Aggregate`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16404
how about we fix this in follow-up PR? Looks like the fix is not trivial.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16404
**[Test build #70603 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70603/testReport)** for PR 16404 at commit [`2ba6469`](https://github.com/apache/spark/commit/2ba6469473ae44d669b0a66095f573e3df4a1948).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait Nondeterministic extends LeafExpression `
* `abstract class RDG extends LeafExpression with ExpectsInputTypes with Nondeterministic `
* `case class Rand(seedExpr: Expression) extends RDG `
* `case class Randn(seedExpr: Expression) extends RDG `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16404
**[Test build #70753 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70753/testReport)** for PR 16404 at commit [`f145188`](https://github.com/apache/spark/commit/f1451883df9077ecbf31f3a86d2427b60262f863).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16404
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on the issue:
https://github.com/apache/spark/pull/16404
Presto allows it and uses the same value of rand() computed during projection for doing aggregation. Internally, every aggregation column is hashed and hash values are appended to the intermediate row. These hashes are used for local aggregation and final aggregation after a shuffle.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16404: [SPARK-18969][SQL] Support grouping by nondetermi...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16404#discussion_r93997700
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -1918,28 +1918,37 @@ class Analyzer(
case p: Project => p
case f: Filter => f
+ case a: Aggregate if a.groupingExpressions.exists(!_.deterministic) =>
+ val nondeterToAttr = getNondeterToAttr(a.groupingExpressions)
+ val newChild = Project(a.child.output ++ nondeterToAttr.values, a.child)
+ a.transformExpressions { case e =>
+ nondeterToAttr.get(e).map(_.toAttribute).getOrElse(e)
+ }.copy(child = newChild)
+
// todo: It's hard to write a general rule to pull out nondeterministic expressions
// from LogicalPlan, currently we only do it for UnaryNode which has same output
// schema with its child.
case p: UnaryNode if p.output == p.child.output && p.expressions.exists(!_.deterministic) =>
- val nondeterministicExprs = p.expressions.filterNot(_.deterministic).flatMap { expr =>
- val leafNondeterministic = expr.collect {
- case n: Nondeterministic => n
- }
- leafNondeterministic.map { e =>
- val ne = e match {
- case n: NamedExpression => n
- case _ => Alias(e, "_nondeterministic")(isGenerated = true)
- }
- new TreeNodeRef(e) -> ne
- }
- }.toMap
+ val nondeterToAttr = getNondeterToAttr(p.expressions)
val newPlan = p.transformExpressions { case e =>
- nondeterministicExprs.get(new TreeNodeRef(e)).map(_.toAttribute).getOrElse(e)
+ nondeterToAttr.get(e).map(_.toAttribute).getOrElse(e)
}
- val newChild = Project(p.child.output ++ nondeterministicExprs.values, p.child)
+ val newChild = Project(p.child.output ++ nondeterToAttr.values, p.child)
Project(p.output, newPlan.withNewChildren(newChild :: Nil))
}
+
+ private def getNondeterToAttr(exprs: Seq[Expression]): Map[Expression, NamedExpression] = {
+ exprs.filterNot(_.deterministic).flatMap { expr =>
+ val leafNondeterministic = expr.collect { case n: Nondeterministic => n }
--- End diff --
is it same with the existing test? `select a, rand(0), sum(b) from data group by a, 2`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16404
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70597/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16404
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16404
Agree. Oracle behaves the same.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16404
It sounds like different RDBMS have different behaviors. Have we decided which way we should follow?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/16404
LGTM on the behavior
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16404
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70617/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16404
Oracle allows it. It sounds like they treat ` (username || dbms_random.string('a', 10))` in aggregate and group-by as the same expression.
```SQL
SQL> select (username || dbms_random.string('a', 10)) from all_users group by (username || dbms_random.string('a', 10));
(USERNAME||DBMS_RANDOM.STRING('A',10))
--------------------------------------------------------------------------------
APEX_040000cklbMYhekl
FLOWS_FILESVmTbIIeiUs
CTXSYSPmgqeRFPry
SYSTEMxQLrzXxHth
XDBRRTfatsLlU
SYSoLDWRKMvlZ
XS$NULLXAaOykZCDH
APEX_PUBLIC_USERvcLswvpbcw
ANONYMOUSgupWiktQKh
OUTLNjLdKOTZoFI
MDSYSxEOhwTwQqa
(USERNAME||DBMS_RANDOM.STRING('A',10))
--------------------------------------------------------------------------------
HRkovpxQztYU
12 rows selected.
```
If I change the order, I got the error:
```SQL
SQL> select (dbms_random.string('a', 10) || username) from all_users group by (username || dbms_random.string('a', 10))
2 ;
select (dbms_random.string('a', 10) || username) from all_users group by (username || dbms_random.string('a', 10))
*
ERROR at line 1:
ORA-00979: not a GROUP BY expression
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16404: [SPARK-18969][SQL] Support grouping by nondetermi...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/16404
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16404
**[Test build #70617 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70617/testReport)** for PR 16404 at commit [`f145188`](https://github.com/apache/spark/commit/f1451883df9077ecbf31f3a86d2427b60262f863).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16404: [SPARK-18969][SQL] Support grouping by nondetermi...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16404#discussion_r93903360
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -1918,28 +1918,37 @@ class Analyzer(
case p: Project => p
case f: Filter => f
+ case a: Aggregate if a.groupingExpressions.exists(!_.deterministic) =>
+ val nondeterToAttr = getNondeterToAttr(a.groupingExpressions)
+ val newChild = Project(a.child.output ++ nondeterToAttr.values, a.child)
+ a.transformExpressions { case e =>
+ nondeterToAttr.get(e).map(_.toAttribute).getOrElse(e)
+ }.copy(child = newChild)
+
// todo: It's hard to write a general rule to pull out nondeterministic expressions
// from LogicalPlan, currently we only do it for UnaryNode which has same output
// schema with its child.
case p: UnaryNode if p.output == p.child.output && p.expressions.exists(!_.deterministic) =>
--- End diff --
Just a question. What is the reason why we need to have such a condition `p.output == p.child.output`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16404
I am fine to merge this at first.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16404
**[Test build #70603 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70603/testReport)** for PR 16404 at commit [`2ba6469`](https://github.com/apache/spark/commit/2ba6469473ae44d669b0a66095f573e3df4a1948).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16404
DB2 has such a limit. See the error message `SQL -583`: http://www.ibm.com/support/knowledgecenter/SSEPGG_10.5.0/com.ibm.db2.luw.messages.sql.doc/doc/msql00583n.html
> The routine (function or method) or expression is defined as non-deterministic or as having external action. This is not supported in the context in which it is used. The contexts in which these are not valid are:
> in an expression of a GROUP BY clause
It documents the same workaround:
> Remove the non-deterministic or external action routine or expression from the GROUP BY clause. If grouping is desired on a column of the result that is based on a non-deterministic or external action routine or expression use a nested table expression or a common table expression to first provide a result table with the expression as a column of the result.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16404
How do other databases handle this case? Do they forbid using non-deterministic expressions in GROUP BY, or give a better error message?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16404
also cc @rxin to take a look
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16404: [SPARK-18969][SQL] Support grouping by nondetermi...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16404#discussion_r93901178
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -1918,28 +1918,37 @@ class Analyzer(
case p: Project => p
case f: Filter => f
+ case a: Aggregate if a.groupingExpressions.exists(!_.deterministic) =>
+ val nondeterToAttr = getNondeterToAttr(a.groupingExpressions)
+ val newChild = Project(a.child.output ++ nondeterToAttr.values, a.child)
+ a.transformExpressions { case e =>
+ nondeterToAttr.get(e).map(_.toAttribute).getOrElse(e)
+ }.copy(child = newChild)
+
// todo: It's hard to write a general rule to pull out nondeterministic expressions
// from LogicalPlan, currently we only do it for UnaryNode which has same output
// schema with its child.
case p: UnaryNode if p.output == p.child.output && p.expressions.exists(!_.deterministic) =>
- val nondeterministicExprs = p.expressions.filterNot(_.deterministic).flatMap { expr =>
- val leafNondeterministic = expr.collect {
- case n: Nondeterministic => n
- }
- leafNondeterministic.map { e =>
- val ne = e match {
- case n: NamedExpression => n
- case _ => Alias(e, "_nondeterministic")(isGenerated = true)
- }
- new TreeNodeRef(e) -> ne
- }
- }.toMap
+ val nondeterToAttr = getNondeterToAttr(p.expressions)
val newPlan = p.transformExpressions { case e =>
- nondeterministicExprs.get(new TreeNodeRef(e)).map(_.toAttribute).getOrElse(e)
+ nondeterToAttr.get(e).map(_.toAttribute).getOrElse(e)
}
- val newChild = Project(p.child.output ++ nondeterministicExprs.values, p.child)
+ val newChild = Project(p.child.output ++ nondeterToAttr.values, p.child)
Project(p.output, newPlan.withNewChildren(newChild :: Nil))
}
+
+ private def getNondeterToAttr(exprs: Seq[Expression]): Map[Expression, NamedExpression] = {
+ exprs.filterNot(_.deterministic).flatMap { expr =>
+ val leafNondeterministic = expr.collect { case n: Nondeterministic => n }
--- End diff --
Not all the non-deterministic expressions extend `Nondeterministic`. This might not cover all the cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16404
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16404
**[Test build #70753 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70753/testReport)** for PR 16404 at commit [`f145188`](https://github.com/apache/spark/commit/f1451883df9077ecbf31f3a86d2427b60262f863).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16404
Since Hive and postgres have same behavior on this, I'd like to follow them.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16404
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/16404
Make sure you update the pull request and jira ticket description before you merge.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16404
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70603/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16404: [SPARK-18969][SQL] Support grouping by nondetermi...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16404#discussion_r93997721
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -1918,28 +1918,37 @@ class Analyzer(
case p: Project => p
case f: Filter => f
+ case a: Aggregate if a.groupingExpressions.exists(!_.deterministic) =>
+ val nondeterToAttr = getNondeterToAttr(a.groupingExpressions)
+ val newChild = Project(a.child.output ++ nondeterToAttr.values, a.child)
+ a.transformExpressions { case e =>
+ nondeterToAttr.get(e).map(_.toAttribute).getOrElse(e)
+ }.copy(child = newChild)
+
// todo: It's hard to write a general rule to pull out nondeterministic expressions
// from LogicalPlan, currently we only do it for UnaryNode which has same output
// schema with its child.
case p: UnaryNode if p.output == p.child.output && p.expressions.exists(!_.deterministic) =>
- val nondeterministicExprs = p.expressions.filterNot(_.deterministic).flatMap { expr =>
- val leafNondeterministic = expr.collect {
- case n: Nondeterministic => n
- }
- leafNondeterministic.map { e =>
- val ne = e match {
- case n: NamedExpression => n
- case _ => Alias(e, "_nondeterministic")(isGenerated = true)
- }
- new TreeNodeRef(e) -> ne
- }
- }.toMap
+ val nondeterToAttr = getNondeterToAttr(p.expressions)
val newPlan = p.transformExpressions { case e =>
- nondeterministicExprs.get(new TreeNodeRef(e)).map(_.toAttribute).getOrElse(e)
+ nondeterToAttr.get(e).map(_.toAttribute).getOrElse(e)
}
- val newChild = Project(p.child.output ++ nondeterministicExprs.values, p.child)
+ val newChild = Project(p.child.output ++ nondeterToAttr.values, p.child)
Project(p.output, newPlan.withNewChildren(newChild :: Nil))
}
+
+ private def getNondeterToAttr(exprs: Seq[Expression]): Map[Expression, NamedExpression] = {
+ exprs.filterNot(_.deterministic).flatMap { expr =>
+ val leafNondeterministic = expr.collect { case n: Nondeterministic => n }
--- End diff --
https://github.com/apache/spark/pull/16404/files#diff-a0f2e45a5da747e9ec483f3557aa1b8bR138
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16404: [SPARK-18969][SQL] Support grouping by nondetermi...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16404#discussion_r94229355
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -1918,28 +1918,37 @@ class Analyzer(
case p: Project => p
case f: Filter => f
+ case a: Aggregate if a.groupingExpressions.exists(!_.deterministic) =>
+ val nondeterToAttr = getNondeterToAttr(a.groupingExpressions)
+ val newChild = Project(a.child.output ++ nondeterToAttr.values, a.child)
+ a.transformExpressions { case e =>
+ nondeterToAttr.get(e).map(_.toAttribute).getOrElse(e)
+ }.copy(child = newChild)
+
// todo: It's hard to write a general rule to pull out nondeterministic expressions
// from LogicalPlan, currently we only do it for UnaryNode which has same output
// schema with its child.
case p: UnaryNode if p.output == p.child.output && p.expressions.exists(!_.deterministic) =>
- val nondeterministicExprs = p.expressions.filterNot(_.deterministic).flatMap { expr =>
- val leafNondeterministic = expr.collect {
- case n: Nondeterministic => n
- }
- leafNondeterministic.map { e =>
- val ne = e match {
- case n: NamedExpression => n
- case _ => Alias(e, "_nondeterministic")(isGenerated = true)
- }
- new TreeNodeRef(e) -> ne
- }
- }.toMap
+ val nondeterToAttr = getNondeterToAttr(p.expressions)
val newPlan = p.transformExpressions { case e =>
- nondeterministicExprs.get(new TreeNodeRef(e)).map(_.toAttribute).getOrElse(e)
+ nondeterToAttr.get(e).map(_.toAttribute).getOrElse(e)
}
- val newChild = Project(p.child.output ++ nondeterministicExprs.values, p.child)
+ val newChild = Project(p.child.output ++ nondeterToAttr.values, p.child)
Project(p.output, newPlan.withNewChildren(newChild :: Nil))
}
+
+ private def getNondeterToAttr(exprs: Seq[Expression]): Map[Expression, NamedExpression] = {
+ exprs.filterNot(_.deterministic).flatMap { expr =>
+ val leafNondeterministic = expr.collect { case n: Nondeterministic => n }
--- End diff --
seems we need to add a new API, here we wanna get non-deterministic leaf nodes, and trait `Nondeterministic` is not suitable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16404
Found a bug filed in a JIRA https://issues.apache.org/jira/browse/SPARK-19035. This PR does not resolves it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16404
**[Test build #70754 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70754/testReport)** for PR 16404 at commit [`f145188`](https://github.com/apache/spark/commit/f1451883df9077ecbf31f3a86d2427b60262f863).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16404: [SPARK-18969][SQL] Support grouping by nondetermi...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16404#discussion_r93942649
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -1918,28 +1918,37 @@ class Analyzer(
case p: Project => p
case f: Filter => f
+ case a: Aggregate if a.groupingExpressions.exists(!_.deterministic) =>
+ val nondeterToAttr = getNondeterToAttr(a.groupingExpressions)
+ val newChild = Project(a.child.output ++ nondeterToAttr.values, a.child)
+ a.transformExpressions { case e =>
+ nondeterToAttr.get(e).map(_.toAttribute).getOrElse(e)
+ }.copy(child = newChild)
+
// todo: It's hard to write a general rule to pull out nondeterministic expressions
// from LogicalPlan, currently we only do it for UnaryNode which has same output
// schema with its child.
case p: UnaryNode if p.output == p.child.output && p.expressions.exists(!_.deterministic) =>
- val nondeterministicExprs = p.expressions.filterNot(_.deterministic).flatMap { expr =>
- val leafNondeterministic = expr.collect {
- case n: Nondeterministic => n
- }
- leafNondeterministic.map { e =>
- val ne = e match {
- case n: NamedExpression => n
- case _ => Alias(e, "_nondeterministic")(isGenerated = true)
- }
- new TreeNodeRef(e) -> ne
- }
- }.toMap
+ val nondeterToAttr = getNondeterToAttr(p.expressions)
val newPlan = p.transformExpressions { case e =>
- nondeterministicExprs.get(new TreeNodeRef(e)).map(_.toAttribute).getOrElse(e)
+ nondeterToAttr.get(e).map(_.toAttribute).getOrElse(e)
}
- val newChild = Project(p.child.output ++ nondeterministicExprs.values, p.child)
+ val newChild = Project(p.child.output ++ nondeterToAttr.values, p.child)
Project(p.output, newPlan.withNewChildren(newChild :: Nil))
}
+
+ private def getNondeterToAttr(exprs: Seq[Expression]): Map[Expression, NamedExpression] = {
+ exprs.filterNot(_.deterministic).flatMap { expr =>
+ val leafNondeterministic = expr.collect { case n: Nondeterministic => n }
--- End diff --
This rule is running once. Thus, it should be safe; otherwise, it might generate many useless `Project` when some expressions are not extending `Nondeterministic` but its deterministics is false.
Maybe added a nagative test case for [this check](https://github.com/apache/spark/blob/v2.0.2/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala#L248-L254)
```scala
sql(s"CREATE TEMPORARY FUNCTION statefulUDF AS '${classOf[StatefulUDF].getName}'")
val df = Seq((1, 1)).toDF("a", "b")
df.createOrReplaceTempView("data")
sql("select a, statefulUDF(), sum(b) from data group by a, 2").show()
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16404
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71244/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16404: [SPARK-18969][SQL] Support grouping by nondeterministic ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16404
MySQL treats them differently...
```SQL
mysql> select c1, concat(rand(), c1) from t1 group by c1;
+------+----------------------+
| c1 | concat(rand(), c1) |
+------+----------------------+
| 1 | 0.084388771172974981 |
| 3 | 0.116890648488784823 |
+------+----------------------+
2 rows in set (0.00 sec)
mysql> select c1, concat(rand(), c1) from t1 group by c1, concat(rand(), c1);
+------+----------------------+
| c1 | concat(rand(), c1) |
+------+----------------------+
| 1 | 0.16241911441313021 |
| 1 | 0.461423657332941551 |
| 3 | 0.81986097415896223 |
+------+----------------------+
3 rows in set (0.00 sec)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org