You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by wangyum <gi...@git.apache.org> on 2018/11/14 03:22:27 UTC
[GitHub] spark pull request #23028: [SPARK-26053][SQL] Enhance LikeSimplification
GitHub user wangyum opened a pull request:
https://github.com/apache/spark/pull/23028
[SPARK-26053][SQL] Enhance LikeSimplification
## What changes were proposed in this pull request?
This PR enhance `LikeSimplification` in 2 cases:
1. null like col -> null
2. 'str' like col -> col = 'str'
It difficult to handle these cases:
1. 'str%' like col
2. '%str' like col
3. 'str%str' like col
4. '%' like col
for example:
```sql
select '8%' like '8%'; -- true
select '8%' like '%8%'; -- true
select '8%' like '%%8%%'; -- true
select '8%' like '%%5%%8%%'; --false
select '%8' like '%8%'; -- true
select '%8' like '%8%'; -- true
select '%8' like '%%8%'; -- true
select '%8' like '%%5%%8%'; -- false
select '%' like '%'; -- true
select '%' like '%%'; -- true
select '%' like '%%8%'; -- false
```
## How was this patch tested?
unit tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/wangyum/spark SPARK-26053
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/23028.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #23028
----
commit 56a02eaaa63f297d3dbaf0ca183e4248d4882834
Author: Yuming Wang <yu...@...>
Date: 2018-11-14T02:55:25Z
Enhance LikeSimplification
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23028: [SPARK-26053][SQL] Enhance LikeSimplification
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23028
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23028: [SPARK-26053][SQL] Enhance LikeSimplification
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23028
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23028: [SPARK-26053][SQL] Enhance LikeSimplification
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23028
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98809/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23028: [SPARK-26053][SQL] Enhance LikeSimplification
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23028
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5011/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23028: [SPARK-26053][SQL] Enhance LikeSimplification
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23028
**[Test build #98809 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98809/testReport)** for PR 23028 at commit [`56a02ea`](https://github.com/apache/spark/commit/56a02eaaa63f297d3dbaf0ca183e4248d4882834).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #23028: [SPARK-26053][SQL] Enhance LikeSimplification
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum commented on a diff in the pull request:
https://github.com/apache/spark/pull/23028#discussion_r233317201
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala ---
@@ -504,6 +504,19 @@ object LikeSimplification extends Rule[LogicalPlan] {
Like(input, Literal.create(pattern, StringType))
}
}
+
+ case Like(Literal(pattern, StringType), input) =>
+ if (pattern == null) {
+ // If pattern is null, return null value directly, since "null like col" == null.
+ Literal(null, BooleanType)
+ } else {
+ pattern.toString match {
+ case equalTo(str) =>
+ EqualTo(Literal(str), input)
--- End diff --
Yes.
```
select "abc" like "%abc%" -> true
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #23028: [SPARK-26053][SQL] Enhance LikeSimplification
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/23028#discussion_r233315756
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala ---
@@ -504,6 +504,19 @@ object LikeSimplification extends Rule[LogicalPlan] {
Like(input, Literal.create(pattern, StringType))
}
}
+
+ case Like(Literal(pattern, StringType), input) =>
+ if (pattern == null) {
+ // If pattern is null, return null value directly, since "null like col" == null.
+ Literal(null, BooleanType)
+ } else {
+ pattern.toString match {
+ case equalTo(str) =>
+ EqualTo(Literal(str), input)
--- End diff --
This is not correct. E.g., "abc" like "abc%"?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #23028: [SPARK-26053][SQL] Enhance LikeSimplification
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum closed the pull request at:
https://github.com/apache/spark/pull/23028
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #23028: [SPARK-26053][SQL] Enhance LikeSimplification
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/23028#discussion_r233315192
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala ---
@@ -504,6 +504,19 @@ object LikeSimplification extends Rule[LogicalPlan] {
Like(input, Literal.create(pattern, StringType))
}
}
+
+ case Like(Literal(pattern, StringType), input) =>
+ if (pattern == null) {
+ // If pattern is null, return null value directly, since "null like col" == null.
--- End diff --
This is rather like `If string input is null, return null value directly, ... `.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #23028: [SPARK-26053][SQL] Enhance LikeSimplification
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/23028#discussion_r233315067
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala ---
@@ -504,6 +504,19 @@ object LikeSimplification extends Rule[LogicalPlan] {
Like(input, Literal.create(pattern, StringType))
}
}
+
+ case Like(Literal(pattern, StringType), input) =>
--- End diff --
Left input to `Like` is string input. This makes confusing. Right parameter is `pattern`.
```scala
case Like(Literal(input, StringType), pattern) =>
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23028: [SPARK-26053][SQL] Enhance LikeSimplification
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23028
**[Test build #98809 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98809/testReport)** for PR 23028 at commit [`56a02ea`](https://github.com/apache/spark/commit/56a02eaaa63f297d3dbaf0ca183e4248d4882834).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org