You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/18 10:00:21 UTC
[GitHub] [spark] peter-toth opened a new pull request #29805: [SPARK-32635][SQL][2.4] Fix foldable propagation
peter-toth opened a new pull request #29805:
URL: https://github.com/apache/spark/pull/29805
### What changes were proposed in this pull request?
This PR rewrites `FoldablePropagation` rule to replace attribute references in a node with foldables coming only from the node's children.
Before this PR in the case of this example (with setting`spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation`):
```scala
val a = Seq("1").toDF("col1").withColumn("col2", lit("1"))
val b = Seq("2").toDF("col1").withColumn("col2", lit("2"))
val aub = a.union(b)
val c = aub.filter($"col1" === "2").cache()
val d = Seq("2").toDF( "col4")
val r = d.join(aub, $"col2" === $"col4").select("col4")
val l = c.select("col2")
val df = l.join(r, $"col2" === $"col4", "LeftOuter")
df.show()
```
foldable propagation happens incorrectly:
```
Join LeftOuter, (col2#6 = col4#34) Join LeftOuter, (col2#6 = col4#34)
!:- Project [col2#6] :- Project [1 AS col2#6]
: +- InMemoryRelation [col1#4, col2#6], StorageLevel(disk, memory, deserialized, 1 replicas) : +- InMemoryRelation [col1#4, col2#6], StorageLevel(disk, memory, deserialized, 1 replicas)
: +- Union : +- Union
: :- *(1) Project [value#1 AS col1#4, 1 AS col2#6] : :- *(1) Project [value#1 AS col1#4, 1 AS col2#6]
: : +- *(1) Filter (isnotnull(value#1) AND (value#1 = 2)) : : +- *(1) Filter (isnotnull(value#1) AND (value#1 = 2))
: : +- *(1) LocalTableScan [value#1] : : +- *(1) LocalTableScan [value#1]
: +- *(2) Project [value#10 AS col1#13, 2 AS col2#15] : +- *(2) Project [value#10 AS col1#13, 2 AS col2#15]
: +- *(2) Filter (isnotnull(value#10) AND (value#10 = 2)) : +- *(2) Filter (isnotnull(value#10) AND (value#10 = 2))
: +- *(2) LocalTableScan [value#10] : +- *(2) LocalTableScan [value#10]
+- Project [col4#34] +- Project [col4#34]
+- Join Inner, (col2#6 = col4#34) +- Join Inner, (col2#6 = col4#34)
:- Project [value#31 AS col4#34] :- Project [value#31 AS col4#34]
: +- LocalRelation [value#31] : +- LocalRelation [value#31]
+- Project [col2#6] +- Project [col2#6]
+- Union false, false +- Union false, false
:- Project [1 AS col2#6] :- Project [1 AS col2#6]
: +- LocalRelation [value#1] : +- LocalRelation [value#1]
+- Project [2 AS col2#15] +- Project [2 AS col2#15]
+- LocalRelation [value#10] +- LocalRelation [value#10]
```
and so the result is wrong:
```
+----+----+
|col2|col4|
+----+----+
| 1|null|
+----+----+
```
After this PR foldable propagation will not happen incorrectly and the result is correct:
```
+----+----+
|col2|col4|
+----+----+
| 2| 2|
+----+----+
```
### Why are the changes needed?
To fix a correctness issue.
### Does this PR introduce _any_ user-facing change?
Yes, fixes a correctness issue.
### How was this patch tested?
Existing and new UTs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] peter-toth commented on pull request #29805: [SPARK-32635][SQL][2.4] Fix foldable propagation
Posted by GitBox <gi...@apache.org>.
peter-toth commented on pull request #29805:
URL: https://github.com/apache/spark/pull/29805#issuecomment-694777537
@dongjoon-hyun, @maropu, this is the 2.4 backport of https://github.com/apache/spark/pull/29771 + https://github.com/apache/spark/pull/29802.
cc @cloud-fan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29805: [SPARK-32635][SQL][2.4] Fix foldable propagation
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29805:
URL: https://github.com/apache/spark/pull/29805#issuecomment-694875462
**[Test build #128874 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128874/testReport)** for PR 29805 at commit [`20ccb7b`](https://github.com/apache/spark/commit/20ccb7b6d7de7b5491c3709aa28dba68898b9b56).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29805: [SPARK-32635][SQL][2.4] Fix foldable propagation
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29805:
URL: https://github.com/apache/spark/pull/29805#issuecomment-694876323
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29805: [SPARK-32635][SQL][2.4] Fix foldable propagation
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29805:
URL: https://github.com/apache/spark/pull/29805#issuecomment-694778060
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29805: [SPARK-32635][SQL][2.4] Fix foldable propagation
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29805:
URL: https://github.com/apache/spark/pull/29805#issuecomment-694777543
**[Test build #128874 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128874/testReport)** for PR 29805 at commit [`20ccb7b`](https://github.com/apache/spark/commit/20ccb7b6d7de7b5491c3709aa28dba68898b9b56).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29805: [SPARK-32635][SQL][2.4] Fix foldable propagation
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29805:
URL: https://github.com/apache/spark/pull/29805#issuecomment-694778060
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #29805: [SPARK-32635][SQL][2.4] Fix foldable propagation
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #29805:
URL: https://github.com/apache/spark/pull/29805#issuecomment-695122960
late LGTM, thanks, @peter-toth !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #29805: [SPARK-32635][SQL][2.4] Fix foldable propagation
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #29805:
URL: https://github.com/apache/spark/pull/29805
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29805: [SPARK-32635][SQL][2.4] Fix foldable propagation
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29805:
URL: https://github.com/apache/spark/pull/29805#issuecomment-694777543
**[Test build #128874 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128874/testReport)** for PR 29805 at commit [`20ccb7b`](https://github.com/apache/spark/commit/20ccb7b6d7de7b5491c3709aa28dba68898b9b56).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29805: [SPARK-32635][SQL][2.4] Fix foldable propagation
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29805:
URL: https://github.com/apache/spark/pull/29805#issuecomment-694876323
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org