You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by nongli <gi...@git.apache.org> on 2015/11/05 02:22:32 UTC
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
GitHub user nongli opened a pull request:
https://github.com/apache/spark/pull/9480
[SPARK-10371] [SQL] Implement subexpr elimination for UnsafeProjections
This patch adds the building blocks for codegening subexpr elimination and implements
it end to end for UnsafeProjection. The building blocks can be used to do the same thing
for other operators.
It introduces some utilities to compute common sub expressions. Expressions can be added to
this data structure. The expr and its children will be recursively matched against existing
expressions (ones previously added) and grouped into common groups. This is built using
the existing `semanticEquals`. It does not understand things like commutative or associative
expressions. This can be done as future work.
After building this data structure, the codegen process takes advantage of it by:
1. Generating a helper function in the generated class that computes the common
subexpression. This is done for all common subexpressions that have at least
two occurrences and the expression tree is sufficiently complex.
2. When generating the apply() function, if the helper function exists, call that
instead of regenerating the expression tree. Repeated calls to the helper function
shortcircuit the evaluation logic.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nongli/spark spark-10371
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9480.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9480
----
commit 2feafbcc2472503048d9d81c6985c1fcdd1dab80
Author: Nong Li <no...@gmail.com>
Date: 2015-10-28T20:40:17Z
[SPARK-10371] [SQL] Implement subexpr elimination for UnsafeProjections
This patch adds the building blocks for codegening subexpr elimination and implements
it end to end for UnsafeProjection. The building blocks can be used to do the same thing
for other operators.
It introduces some utilities to compute common sub expressions. Expressions can be added to
this data structure. The expr and its children will be recursively matched against existing
expressions (ones previously added) and grouped into common groups. This is built using
the existing `semanticEquals`. It does not understand things like commutative or associative
expressions. This can be done as future work.
After building this data structure, the codegen process takes advantage of it by:
1. Generating a helper function in the generated class that computes the common
subexpression. This is done for all common subexpressions that have at least
two occurrences and the expression tree is sufficiently complex.
2. When generating the apply() function, if the helper function exists, call that
instead of regenerating the expression tree. Repeated calls to the helper function
shortcircuit the evaluation logic.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155250085
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by nongli <gi...@git.apache.org>.
Github user nongli commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44105110
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -151,6 +164,30 @@ abstract class Expression extends TreeNode[Expression] {
}
/**
+ * Returns the hash for this expression. Expressions that compute the same result, even if
+ * they differ cosmetically should return the same hash.
+ */
+ def semanticHash() : Int = {
--- End diff --
I think that's what I do. I put the exprs in a hash set using semantic hash/semantic equals.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44106783
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -151,6 +164,30 @@ abstract class Expression extends TreeNode[Expression] {
}
/**
+ * Returns the hash for this expression. Expressions that compute the same result, even if
+ * they differ cosmetically should return the same hash.
+ */
+ def semanticHash() : Int = {
--- End diff --
Sorry, I am still confusing, why we can't use the `hashCode` instead.
The motivation of `semanticEquals` is to ignore the `AttributeReference.name` in comparison for `AttributeReference`, and the `AttributeReference.hashCode` also does ignore the `AttributeReference.name`, I don't think the `cosmetic differences` really exists for `hashCode`. Isn't it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155537739
I think I can resolve the conflicts manually. Merging to master and 1.6.
Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155259218
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155282054
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44108481
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -151,6 +164,30 @@ abstract class Expression extends TreeNode[Expression] {
}
/**
+ * Returns the hash for this expression. Expressions that compute the same result, even if
+ * they differ cosmetically should return the same hash.
+ */
+ def semanticHash() : Int = {
--- End diff --
And there is more discussion on the `semanticEquals` can be found at #6587, even, I don't think we need the `semanticEquals` if we changed the implementation of `AttributeReference.equals`, as it does make lots of code complicated like this one, by using the `semanticEquals`, other than `equals` or `==`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44197514
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala ---
@@ -203,6 +203,10 @@ case class AttributeReference(
case _ => false
}
+ override def semanticHash(): Int = {
+ this.exprId.hashCode()
--- End diff --
I see, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44099965
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -151,6 +164,30 @@ abstract class Expression extends TreeNode[Expression] {
}
/**
+ * Returns the hash for this expression. Expressions that compute the same result, even if
+ * they differ cosmetically should return the same hash.
+ */
+ def semanticHash() : Int = {
--- End diff --
Identical hash values doesn't mean the identical values, I am suggesting to use the `hashCode` plus `semanticEquals` to identity the common expression, that's also the motivation of `semanticEquals`. See `AttributeReference` for details.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154567121
**[Test build #45259 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45259/consoleFull)** for PR 9480 at commit [`e65def9`](https://github.com/apache/spark/commit/e65def93d58de363661ec3722670f6e766106b6a).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `class EquivalentExpressions `\n * ` case class Expr(e: Expression) `\n * ` case class SubExprEliminationState(`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154288464
I took one pass. Overall pretty good! @davies Can you also take a look?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-153924518
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r43966655
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -151,6 +164,30 @@ abstract class Expression extends TreeNode[Expression] {
}
/**
+ * Returns the hash for this expression. Expressions that compute the same result, even if
+ * they differ cosmetically should return the same hash.
+ */
+ def semanticHash() : Int = {
--- End diff --
Since all of the expressions are the case class, probably we don't need to our own way to `computeHash`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154586930
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by nongli <gi...@git.apache.org>.
Github user nongli commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154518082
> In general, I think we'd better add more IR (Intermediate Representation, should be the Expression or Logical Plan node here) for common sub expression cases, and we then we can identify the common sub expressions and transform to the new IRs, which should be more clean for code gen part of work.
>As the codegen code is a runtime behavior, which is difficult to debug and maintain, more IR will be helpful for this case, and that's also how a modern compiler works.
>And I don't think the approach I described can be done right now, it's will be a big change, actually, we have lots of similar optimization like "Common Table Expression" (CTE), or self join can be optimized in the same way, it will be cool if we can figure a approach for the general common xxx elimination.
I'm still not sure how you are suggesting we execute the version you are proposing. I'd guess you do that by adding an additional project that creates a new intermediate row and substitute the common subexpression to read from that new intermediate row. I think this is bad for performance. One of the big benefits of codegen is to remove those "unnecessary" intermediate rows.
I agree that we should expand this logic in planning and analysis but I think the utility class to compute equivalence is reusable in other parts.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155236703
**[Test build #45443 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45443/consoleFull)** for PR 9480 at commit [`6cf0186`](https://github.com/apache/spark/commit/6cf0186579863f28f5c23b1936c9f727e90cf2de).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44177613
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/EquivalentExpressions.scala ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import scala.collection.mutable
+
+/**
+ * This class is used to compute equality of (sub)expression trees. Expressions can be added
+ * to this class and they subsequently query for expression equality. Expression trees are
+ * considered equal if for the same input(s), the same result is produced.
+ */
+class EquivalentExpressions {
+ /**
+ * Wrapper around an Expression that provides semantic equality.
+ */
+ case class Expr(e: Expression) {
+ val hash = e.semanticHash()
+ override def equals(o: Any): Boolean = o match {
+ case other: Expr => e.semanticEquals(other.e)
+ case _ => false
+ }
+ override def hashCode: Int = hash
+ }
+
+ // For each expression, the set of equivalent expressions.
+ private val equivalenceMap: mutable.HashMap[Expr, mutable.MutableList[Expression]] =
+ new mutable.HashMap[Expr, mutable.MutableList[Expression]]
+
+ /**
+ * Adds each expression to this data structure, grouping them with existing equivalent
+ * expressions. Non-recursive.
+ * Returns if there was already a matching expression.
+ */
+ def addExpr(expr: Expression): Boolean = {
+ if (expr.deterministic) {
+ val e: Expr = Expr(expr)
+ val f = equivalenceMap.get(e)
+ if (f.isDefined) {
+ f.get.+= (expr)
+ true
+ } else {
+ equivalenceMap.put(e, mutable.MutableList(expr))
+ false
+ }
+ } else {
+ false
+ }
+ }
+
+ /**
+ * Adds the expression to this datastructure recursively. Stops if a matching expression
+ * is found. That is, if `expr` has already been added, its children are not added.
+ * If ignoreLeaf is true, leaf nodes are ignored.
+ */
+ def addExprTree(root: Expression, ignoreLeaf: Boolean): Unit = {
--- End diff --
Since ignoreLeaf is only useful for testing, we can have a default `true` here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155281866
**[Test build #45468 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45468/consoleFull)** for PR 9480 at commit [`6cf0186`](https://github.com/apache/spark/commit/6cf0186579863f28f5c23b1936c9f727e90cf2de).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `class EquivalentExpressions `\n * ` case class Expr(e: Expression) `\n * ` case class SubExprEliminationState(`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by nongli <gi...@git.apache.org>.
Github user nongli commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44194583
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala ---
@@ -203,6 +203,10 @@ case class AttributeReference(
case _ => false
}
+ override def semanticHash(): Int = {
+ this.exprId.hashCode()
--- End diff --
It works fine currently because by the time this has run, the expression is a bound attribute. We should consider running the logic of finding subexpressions earlier, as chenghao suggests. I added a unit test that exposes the problem of not defining this. As a general rule, I think any class that defines semanticEquals should define semanticHash, even if it is the default implementation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154586827
**[Test build #45263 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45263/consoleFull)** for PR 9480 at commit [`3a10f7d`](https://github.com/apache/spark/commit/3a10f7d0fac41025ced67c42bd60b8203356379c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `class EquivalentExpressions `\n * ` case class Expr(e: Expression) `\n * ` case class SubExprEliminationState(`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155258413
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-153948155
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44177472
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala ---
@@ -203,6 +203,10 @@ case class AttributeReference(
case _ => false
}
+ override def semanticHash(): Int = {
+ this.exprId.hashCode()
--- End diff --
Should we have a test case for this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155259185
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155185293
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-153924543
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44178151
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala ---
@@ -341,7 +440,18 @@ abstract class CodeGenerator[InType <: AnyRef, OutType <: AnyRef] extends Loggin
protected def declareMutableStates(ctx: CodeGenContext): String = {
ctx.mutableStates.map { case (javaType, variableName, _) =>
s"private $javaType $variableName;"
- }.mkString("\n")
+ }.mkString("\n") + "\n" +
+ // Maintain the loaded value and isNull as member variables. This is necessary if the codegen
+ // function is split across multiple functions.
+ // TODO: maintaining this as a local variable probably allows the compiler to do better
+ // optimizations.
+ ctx.subExprEliminationStates.map { s => {
--- End diff --
Should we use a separate method for this part?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155250029
**[Test build #45443 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45443/consoleFull)** for PR 9480 at commit [`6cf0186`](https://github.com/apache/spark/commit/6cf0186579863f28f5c23b1936c9f727e90cf2de).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `class EquivalentExpressions `\n * ` case class Expr(e: Expression) `\n * ` case class SubExprEliminationState(`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155259441
**[Test build #45468 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45468/consoleFull)** for PR 9480 at commit [`6cf0186`](https://github.com/apache/spark/commit/6cf0186579863f28f5c23b1936c9f727e90cf2de).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by nongli <gi...@git.apache.org>.
Github user nongli commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44043807
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -151,6 +164,30 @@ abstract class Expression extends TreeNode[Expression] {
}
/**
+ * Returns the hash for this expression. Expressions that compute the same result, even if
+ * they differ cosmetically should return the same hash.
+ */
+ def semanticHash() : Int = {
--- End diff --
I don't think so. Looking at thecomments on semanticEquals, we want to ignore cosmetic differences.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154562314
**[Test build #45263 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45263/consoleFull)** for PR 9480 at commit [`3a10f7d`](https://github.com/apache/spark/commit/3a10f7d0fac41025ced67c42bd60b8203356379c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154561748
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155143767
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-153925305
**[Test build #45075 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45075/consoleFull)** for PR 9480 at commit [`2feafbc`](https://github.com/apache/spark/commit/2feafbcc2472503048d9d81c6985c1fcdd1dab80).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-153948080
**[Test build #45075 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45075/consoleFull)** for PR 9480 at commit [`2feafbc`](https://github.com/apache/spark/commit/2feafbcc2472503048d9d81c6985c1fcdd1dab80).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `class EquivalentExpressions `\n * ` case class Expr(e: Expression) `\n * ` case class SubExprEliminationState(`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155185089
**[Test build #45384 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45384/consoleFull)** for PR 9480 at commit [`5dbc047`](https://github.com/apache/spark/commit/5dbc0477939fa521f7da3a6d43c0de8221219368).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `class EquivalentExpressions `\n * ` case class Expr(e: Expression) `\n * ` case class SubExprEliminationState(`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44202414
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -135,6 +147,7 @@ abstract class Expression extends TreeNode[Expression] {
/**
* Returns true when two expressions will always compute the same result, even if they differ
* cosmetically (i.e. capitalization of names in attributes may be different).
+ * TODO: how should this deal with nonDeterministic
--- End diff --
Yeah, I guess we could just add `!nonDeteminstic && ...` since nonDeterministic expressions can't equal anything?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by nongli <gi...@git.apache.org>.
Github user nongli commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44190096
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala ---
@@ -341,7 +440,18 @@ abstract class CodeGenerator[InType <: AnyRef, OutType <: AnyRef] extends Loggin
protected def declareMutableStates(ctx: CodeGenContext): String = {
ctx.mutableStates.map { case (javaType, variableName, _) =>
s"private $javaType $variableName;"
- }.mkString("\n")
+ }.mkString("\n") + "\n" +
+ // Maintain the loaded value and isNull as member variables. This is necessary if the codegen
+ // function is split across multiple functions.
+ // TODO: maintaining this as a local variable probably allows the compiler to do better
+ // optimizations.
+ ctx.subExprEliminationStates.map { s => {
--- End diff --
I changed this code to just use mutable state.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154590257
**[Test build #1999 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1999/consoleFull)** for PR 9480 at commit [`3a10f7d`](https://github.com/apache/spark/commit/3a10f7d0fac41025ced67c42bd60b8203356379c).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `class EquivalentExpressions `\n * ` case class Expr(e: Expression) `\n * ` case class SubExprEliminationState(`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-153948156
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45075/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44205755
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -151,6 +164,30 @@ abstract class Expression extends TreeNode[Expression] {
}
/**
+ * Returns the hash for this expression. Expressions that compute the same result, even if
+ * they differ cosmetically should return the same hash.
+ */
+ def semanticHash() : Int = {
--- End diff --
When you look at the methods `equals`, `semanticEquals` and `hashCode` of the `AttributeReference`, you will see that they are not matched, as `equals` will take consideration of the `name`, but the other 2 are not, that's why I am thinking we can also use the `hashCode`, instead of adding the new method `semanticHash`.
Anyway, it's not an external API, we can change it back anytime, as 1.6 is almost code freeze, and this is critical for people now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155143818
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155236534
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154570265
**[Test build #1999 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1999/consoleFull)** for PR 9480 at commit [`3a10f7d`](https://github.com/apache/spark/commit/3a10f7d0fac41025ced67c42bd60b8203356379c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154551510
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44202061
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeProjection.scala ---
@@ -275,8 +275,9 @@ object GenerateUnsafeProjection extends CodeGenerator[Seq[Expression], UnsafePro
"""
}
- def createCode(ctx: CodeGenContext, expressions: Seq[Expression]): GeneratedExpressionCode = {
- val exprEvals = expressions.map(e => e.gen(ctx))
+ def createCode(ctx: CodeGenContext, expressions: Seq[Expression],
--- End diff --
Style nit: generally once wrapping we wrap all function args.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44201615
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -151,6 +164,30 @@ abstract class Expression extends TreeNode[Expression] {
}
/**
+ * Returns the hash for this expression. Expressions that compute the same result, even if
+ * they differ cosmetically should return the same hash.
+ */
+ def semanticHash() : Int = {
+ def computeHash(e: Seq[Any]): Int = {
--- End diff --
Oh sorry, I get it now...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/9480
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by nongli <gi...@git.apache.org>.
Github user nongli commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154137436
@chenghao-intel I'm not sure what you are suggesting. How do you suggest we execute the expressions? The transformation you have as an example makes sense but how does it get evaluated? What does Sequential do for the execution side? Does it add an additional materialization to evaluate the earlier stages? How does this work with a more complicated expression DAG?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155236517
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154599464
>
I'm still not sure how you are suggesting we execute the version you are proposing. I'd guess you do that by adding an additional project that creates a new intermediate row and substitute the common subexpression to read from that new intermediate row. I think this is bad for performance. One of the big benefits of codegen is to remove those "unnecessary" intermediate rows.
>
I don't think I will do that in the way you described, actually we are working on the common xxx elimination right now, probably more information will be shared later, will ping you then.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154316345
In general, I think we'd better add more IR (Intermediate Representation, should be the Expression or Logical Plan node here) for common sub expression cases, and we then we can identify the common sub expressions and transform to the new IRs, which should be more clean for code gen part of work.
As the codegen code is a runtime behavior, which is difficult to debug and maintain, more IR will be helpful for this case, and that's also how a modern compiler works.
And I don't think the approach I described can be done right now, it's will be a big change, actually, we have lots of similar optimization like "Common Table Expression" (CTE), or self join can be optimized in the same way, it will be cool if we can figure a approach for the general common xxx elimination.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by nongli <gi...@git.apache.org>.
Github user nongli commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44190675
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala ---
@@ -203,6 +203,10 @@ case class AttributeReference(
case _ => false
}
+ override def semanticHash(): Int = {
+ this.exprId.hashCode()
--- End diff --
Is there an easy way to test this? The end to end tests to exercise this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44191412
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala ---
@@ -203,6 +203,10 @@ case class AttributeReference(
case _ => false
}
+ override def semanticHash(): Int = {
+ this.exprId.hashCode()
--- End diff --
What happens if we do not have this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44201576
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -151,6 +164,30 @@ abstract class Expression extends TreeNode[Expression] {
}
/**
+ * Returns the hash for this expression. Expressions that compute the same result, even if
+ * they differ cosmetically should return the same hash.
+ */
+ def semanticHash() : Int = {
+ def computeHash(e: Seq[Any]): Int = {
--- End diff --
Whats wrong with `hashCode` on `Seq[Any]`? I think this its just going to use [this](https://github.com/scala/scala/blob/27da46343cd545534819300235bc64ab74958c92/src/library/scala/util/hashing/MurmurHash3.scala#L217) which seems pretty reasonable. Maybe I'm missing something though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-153926546
My 2 cents, Common Expression Elimination is a very interesting/useful optimization, and I did the same thing for Shark quite long time ago, for my understanding, we probably need to add additional expression/operator for Sequential/Conditional execution. instead of do it directly via code gen.
e.g.
```
(a+b) + (a+b) => Add(Add(a, b), Add(a, b)) => Sequential(Alias(a1, Add(a, b)), Add(a1,a1))
```
So we can eliminate the common operations for Intermediate Representation (IR), which will make the codegen part code more clean and simple.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44201792
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/EquivalentExpressions.scala ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import scala.collection.mutable
+
+/**
+ * This class is used to compute equality of (sub)expression trees. Expressions can be added
+ * to this class and they subsequently query for expression equality. Expression trees are
+ * considered equal if for the same input(s), the same result is produced.
+ */
+class EquivalentExpressions {
+ /**
+ * Wrapper around an Expression that provides semantic equality.
+ */
+ case class Expr(e: Expression) {
+ val hash = e.semanticHash()
+ override def equals(o: Any): Boolean = o match {
+ case other: Expr => e.semanticEquals(other.e)
+ case _ => false
+ }
+ override def hashCode: Int = hash
+ }
+
+ // For each expression, the set of equivalent expressions.
+ private val equivalenceMap: mutable.HashMap[Expr, mutable.MutableList[Expression]] =
+ new mutable.HashMap[Expr, mutable.MutableList[Expression]]
+
+ /**
+ * Adds each expression to this data structure, grouping them with existing equivalent
+ * expressions. Non-recursive.
+ * Returns if there was already a matching expression.
+ */
+ def addExpr(expr: Expression): Boolean = {
+ if (expr.deterministic) {
+ val e: Expr = Expr(expr)
+ val f = equivalenceMap.get(e)
+ if (f.isDefined) {
+ f.get.+= (expr)
+ true
+ } else {
+ equivalenceMap.put(e, mutable.MutableList(expr))
+ false
+ }
+ } else {
+ false
+ }
+ }
+
+ /**
+ * Adds the expression to this datastructure recursively. Stops if a matching expression
+ * is found. That is, if `expr` has already been added, its children are not added.
+ * If ignoreLeaf is true, leaf nodes are ignored.
+ */
+ def addExprTree(root: Expression, ignoreLeaf: Boolean = true): Unit = {
+ val skip = root.isInstanceOf[LeafExpression] && ignoreLeaf
+ if (!skip && root.deterministic && !addExpr(root)) {
+ root.children.foreach(addExprTree(_, ignoreLeaf))
+ }
+ }
+
+ /**
+ * Returns all fo the expression trees that are equivalent to `e`. Returns
+ * an empty collection if there are none.
+ */
+ def getEquivalentExprs(e: Expression): Seq[Expression] = {
+ equivalenceMap.get(Expr(e)).getOrElse(mutable.MutableList())
+ }
+
+ /**
+ * Returns all the equivalent sets of expressions.
+ */
+ def getAllEquivalentExprs: Seq[Seq[Expression]] = {
+ equivalenceMap.map { case(k, v) => {
--- End diff --
`equivalenceMap.values`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155282055
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45468/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154561767
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155145415
**[Test build #45384 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45384/consoleFull)** for PR 9480 at commit [`5dbc047`](https://github.com/apache/spark/commit/5dbc0477939fa521f7da3a6d43c0de8221219368).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154585715
This is awesome. LGTM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154567189
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154554772
Thanks for the numbers, that's cool.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by nongli <gi...@git.apache.org>.
Github user nongli commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44179462
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -151,6 +164,30 @@ abstract class Expression extends TreeNode[Expression] {
}
/**
+ * Returns the hash for this expression. Expressions that compute the same result, even if
+ * they differ cosmetically should return the same hash.
+ */
+ def semanticHash() : Int = {
--- End diff --
I took a quick look at 6587 and I agree with michael about semantic equals. I think when equivalence classes are added, semantic equals is even more different than equals.
That being said, this patch is not the motivation to do this. If we decide to remove semanticEquals, this patch can be updated trivially to use equals.
Regarding hashCode vs semanticHash code, I think it does no? It looks to me like the hash everything, including the cosmetic stuff but please correct me if I'm wrong. In general, i think it makes sense to implement hash if you implement equals.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154570059
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44200916
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala ---
@@ -203,6 +203,10 @@ case class AttributeReference(
case _ => false
}
+ override def semanticHash(): Int = {
+ this.exprId.hashCode()
--- End diff --
Nit: I think the indent is off here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154554001
**[Test build #45259 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45259/consoleFull)** for PR 9480 at commit [`e65def9`](https://github.com/apache/spark/commit/e65def93d58de363661ec3722670f6e766106b6a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154518695
This looks pretty good to me, just a few minor comments. Should we have a benchmark to see is there any regression for tiny common subexpression? If not much, then we don't need to have a cost based one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by nongli <gi...@git.apache.org>.
Github user nongli commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154553190
I ran this benchmark:
```
val data = sqlContext.range(20 * 1024 * 1024).toDF("v").registerTempTable("t")
for (i <- 0 until 4) {
val t1 = System.currentTimeMillis ()
val c1 = sql("select v FROM t").rdd.filter (_ => true).count ()
val t2 = System.currentTimeMillis ()
val c2 = sql("select (v + v), (v + v) from t").rdd.filter (_ => true).count
val t3 = System.currentTimeMillis ()
println(s"Iteration $i")
println(s" Q1($c1): ${t2 - t1} ms")
println(s" Q2($c2): ${t3 - t2} ms")
}
```
With subexpression elimination enabled:
```
Iteration 0
Q1(20971520): 2304 ms
Q2(20971520): 1595 ms
Iteration 1
Q1(20971520): 1298 ms
Q2(20971520): 1460 ms
Iteration 2
Q1(20971520): 1351 ms
Q2(20971520): 1435 ms
Iteration 3
Q1(20971520): 1259 ms
Q2(20971520): 1497 ms
```
```
With it disabled:
Iteration 0
Q1(20971520): 2091 ms
Q2(20971520): 1618 ms
Iteration 1
Q1(20971520): 1277 ms
Q2(20971520): 1505 ms
Iteration 2
Q1(20971520): 1222 ms
Q2(20971520): 1468 ms
Iteration 3
Q1(20971520): 1239 ms
Q2(20971520): 1489 ms
```
The difference is small but it appears to be faster even in this case when the exprs are simple.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154590883
I don't think the most recent failure is your fault.
@rxin, did you patch break the build?
```
File "/home/jenkins/workspace/NewSparkPullRequestBuilder/python/pyspark/sql/readwriter.py", line 205, in pyspark.sql.readwriter.DataFrameReader.text
Failed example:
df.collect()
Differences (ndiff with -expected +actual):
- [Row(text=u'hello'), Row(text=u'this')]
? ^ -- ^ --
+ [Row(value=u'hello'), Row(value=u'this')]
? ^^^^ ^^^^
**********************************************************************
1 of 2 in pyspark.sql.readwriter.DataFrameReader.text
***Test Failed*** 1 failures.
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10371] [SQL] Implement subexpr eliminat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154551545
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org