You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by flyson <gi...@git.apache.org> on 2015/12/03 03:45:10 UTC
[GitHub] spark pull request: [SPARK-12114][SQL]Bug fix for Column Pruning r...
GitHub user flyson opened a pull request:
https://github.com/apache/spark/pull/10117
[SPARK-12114][SQL]Bug fix for Column Pruning rule failure in case of "Project<-Filter<-Join"
This PR resolves the failure of Column Pruning rule when there is pattern "Project<-Filter<-Join" in the logical plan.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/flyson/spark bugfix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10117.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10117
----
commit d09d905180c0eb9d8b1ed9aaf5d207e2fac6547c
Author: Min Qiu <mi...@huawei.com>
Date: 2015-12-02T02:01:02Z
Extract the common equality conditions that can be used as a join condition
commit 08a76aefcc6036d600fa92aee037515cce22ae09
Author: Min Qiu <mi...@huawei.com>
Date: 2015-12-03T01:01:41Z
bug fix for ColumnPruning rule to deal with Project <- Filter <- Join case
commit b002b393124568f6d171e5619d40ff749b9b639d
Author: Min Qiu <mi...@huawei.com>
Date: 2015-12-03T02:04:54Z
bug fix for ColumnPruning rule to deal with Project <- Filter <- Join case
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12114][SQL]Bug fix for Column Pruning r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10117#issuecomment-161499887
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10117: [SPARK-12114][SQL]Bug fix for Column Pruning rule failur...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/10117
I believe this problem has been fixed in 2.0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #10117: [SPARK-12114][SQL]Bug fix for Column Pruning rule...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/10117
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12114][SQL]Bug fix for Column Pruning r...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/10117#discussion_r46510956
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala ---
@@ -246,6 +247,16 @@ object ColumnPruning extends Rule[LogicalPlan] {
Project(projectList, Join(pruneJoinChild(left), pruneJoinChild(right), joinType, condition))
+ // Eliminate unneeded attributes from either side of a Join.
+ case Project(projectList, Filter(predicates, Join(left, right, joinType, condition))) =>
--- End diff --
Is it a problem for all operator under `Filter`? like `Project<-Filter<-Join`, `Project<-Filter<-Aggregate`, `Project<-Filter<-Sort`, etc.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12114][SQL]Bug fix for Column Pruning r...
Posted by flyson <gi...@git.apache.org>.
Github user flyson commented on a diff in the pull request:
https://github.com/apache/spark/pull/10117#discussion_r46615751
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala ---
@@ -246,6 +247,16 @@ object ColumnPruning extends Rule[LogicalPlan] {
Project(projectList, Join(pruneJoinChild(left), pruneJoinChild(right), joinType, condition))
+ // Eliminate unneeded attributes from either side of a Join.
+ case Project(projectList, Filter(predicates, Join(left, right, joinType, condition))) =>
--- End diff --
I think this problem shouldn't occur in Aggregate since it's covered by the pattern match:
case a @ Aggregate(_,_,child) if (child.outputSet -- a.references).nonempty.
As for Sort, I still need to think it over.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org