You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "LuciferYang (via GitHub)" <gi...@apache.org> on 2026/10/08 14:55:46 UTC
[PR] [SPARK-60028][SQL] Do not split LIKE 'prefix%suffix' over a nondeterministic input [spark]
LuciferYang opened a new pull request, #59300:
URL: https://github.com/apache/spark/pull/59300
### What changes were proposed in this pull request?
`LikeSimplification` rewrites `input LIKE 'prefix%suffix'` into a length check, a `StartsWith` and an `EndsWith`, each over its own copy of `input`. This PR makes that rewrite only when the input is deterministic. A nondeterministic input keeps the `LIKE`. The other single-pattern rewrites read the input once and are unchanged.
### Why are the changes needed?
A nondeterministic input is evaluated once per copy, so the three checks can each see a different value and pass together for a row where no single value matches the pattern:
```scala
val pick = udf(() => if (scala.util.Random.nextBoolean()) "ay" else "bz").asNondeterministic()
spark.udf.register("pick", pick)
spark.sql(
"SELECT count_if(r) FROM (SELECT pick() LIKE 'a%z' AS r FROM range(0, 1000, 1, 1))").show()
// a few hundred on master (239 in one run); 0 with this change
```
The optimized plan on master computes `r` as `((octet_length(pick()) >= 2) AND (StartsWith(pick(), a) AND EndsWith(pick(), z)))`. `pick() ILIKE 'A%Z'` becomes `lower(pick()) LIKE 'a%z'` and fails the same way. The `LIKE ALL` and `LIKE ANY` rewrites already skip an input that is not cheap.
### Does this PR introduce _any_ user-facing change?
Yes. `LIKE` or `ILIKE` with a `'prefix%suffix'` pattern over a nondeterministic input now evaluates the input once per row, so the result is that of matching that one value. Released versions have the same rewrite, with `length` instead of `octet_length` in the length check.
### How was this patch tested?
`LikeSimplificationSuite`: `uuid() LIKE 'abc%def'` is left unchanged, `uuid() LIKE 'abc%'` is still rewritten to `StartsWith`, and `substring(a, 1, 5) LIKE 'abc%def'`, deterministic but not cheap, is still split into the length check, `StartsWith` and `EndsWith`. The first assertion fails without the change.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60028][SQL] Do not split LIKE 'prefix%suffix' over a nondeterministic input [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #59300:
URL: https://github.com/apache/spark/pull/59300#issuecomment-6116980772
> Thanks for the PR, @LuciferYang!
>
> The `prefix%suffix` rewrite is the only one in `simplifyLike` that reads the input more than once. With the guard, a nondeterministic input falls through to `case _ => None` and keeps the `LIKE`. The `LIKE ALL` / `LIKE ANY` paths were already safe. `CollapseProject.isCheap` lets a nondeterministic input through only as a Python UDF, and `ExtractPythonUDFs` maps all its copies to one result. The new test fails at its first assertion with the old rule, and the suite passes here.
>
> ### Minor
> * **1.** _JIRA affected versions:_ [SPARK-60028](https://issues.apache.org/jira/browse/SPARK-60028) lists only the unreleased 5.0.0, but `branch-3.5` through `branch-4.2` have the same rewrite (with `Length`). So the affected versions are 3.5.8, 4.0.4, 4.1.3 and 4.2.0. As a wrong-results bug, it should also get the `correctness` label.
I will submit separate PRs for other branches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60028][SQL] Do not split LIKE 'prefix%suffix' over a nondeterministic input [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #59300:
URL: https://github.com/apache/spark/pull/59300#issuecomment-6116975004
Thank you @HyukjinKwon @peter-toth @uros-b
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60028][SQL] Do not split LIKE 'prefix%suffix' over a nondeterministic input [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #59300:
URL: https://github.com/apache/spark/pull/59300#issuecomment-6098639324
I will merge this PR if there are no new comments within 24 hours.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60028][SQL] Do not split LIKE 'prefix%suffix' over a nondeterministic input [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #59300:
URL: https://github.com/apache/spark/pull/59300#issuecomment-6116947628
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/c8a3f3d6f99bc43b955abdcd27d6256f24860d8c
- merged into branch-4.x https://github.com/apache/spark/commit/28b3faa54db4959a2f71c4514380f1d9feb0a578
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60028][SQL] Do not split LIKE 'prefix%suffix' over a nondeterministic input [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang closed pull request #59300: [SPARK-60028][SQL] Do not split LIKE 'prefix%suffix' over a nondeterministic input
URL: https://github.com/apache/spark/pull/59300
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60028][SQL] Do not split LIKE 'prefix%suffix' over a nondeterministic input [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #59300:
URL: https://github.com/apache/spark/pull/59300#issuecomment-6062640309
cc @dongjoon-hyun @peter-toth @cloud-fan @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org