You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "fmcmac (via GitHub)" <gi...@apache.org> on 2026/10/18 08:37:33 UTC
[PR] feat: support `RLIKE` and `REGEXP` operators [datafusion]
fmcmac opened a new pull request, #26467:
URL: https://github.com/apache/datafusion/pull/26467
## Which issue does this PR close?
No existing issue. This PR states the problem and a reproduction below.
## Rationale for this change
sqlparser parses `expr [NOT] RLIKE pattern` and `expr [NOT] REGEXP pattern` for every dialect, but DataFusion refuses both:
```sql
SELECT 'abc' RLIKE '^a';
-- Error: This feature is not implemented: Unsupported ast node in sqltorel: RLike { negated: false, .. }
```
Expected: `true`. DataFusion already has the semantics as `regexp_like(expr, pattern)`. Spark, Hive and MySQL users write these operators directly.
## What changes are included in this PR?
- `SqlToRel` plans `expr [NOT] RLIKE|REGEXP pattern` as `[NOT] regexp_like(expr, pattern)`.
- `regexp_like` resolves through `ContextProvider::get_function_meta`, the same lookup as an explicit call. A session that overrides `regexp_like` gets its override for all 3 spellings. A session without it gets a planning error naming `regexp_like`.
- Explicit calls and the operators share a new `plan_scalar_udf` helper. Registered `ExprPlanner::plan_scalar` hooks therefore see `RLIKE`, `REGEXP` and their negations, just as they see `regexp_like(...)`. The helper keeps a planner's result distinct from the default call, so a planner's rewrite is returned as is and only the default call gets the alias for a function called by another name.
- It plans as the function, not `Operator::RegexMatch`. The unparser renders `RegexMatch` as `~` for every dialect, while a function call can be renamed per dialect. The built-in's `simplify` still rewrites it to `~` for execution.
- Docs: `RLIKE` and `NOT RLIKE` in the operators guide.
## What is the testing strategy for this PR?
- `regexp/regexp_like.slt`: `RLIKE`, `REGEXP`, `NOT RLIKE` and `NOT REGEXP` over the shared test data, a NULL subject, a NULL pattern, a column pattern, and row-for-row agreement with `regexp_like(...)`.
- `sql_integration.rs`:
- `select_rlike_plans_as_regexp_like`: registry resolution and plan shape.
- `select_rlike_without_regexp_like`: the planning error.
- `select_rlike_with_expr_planner`: an `ExprPlanner` that rewrites `regexp_like` to `~` applies to the explicit call, `RLIKE`, `REGEXP`, `NOT RLIKE` and `NOT REGEXP`.
- `select_expr_planner_rewrite_to_other_function`: an `ExprPlanner` that rewrites `regexp_like` to another scalar function keeps that function's output name, for the explicit call and for `RLIKE`, with no alias back to `regexp_like(...)`.
- All fail on `main`.
## Are there any user-facing changes?
Yes: `RLIKE` and `REGEXP` now plan instead of erroring. Documented in the operators guide. `ExprPlanner::plan_scalar` now also sees these operators. No public API change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support `RLIKE` and `REGEXP` operators [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on PR #26467:
URL: https://github.com/apache/datafusion/pull/26467#issuecomment-6506259094
> Do you think this is a PR worth pursuing @Omega359 ?
No response from author, I'm not pursuing it. While I think it's useful I haven't heard of a request for it previously.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support `RLIKE` and `REGEXP` operators [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on code in PR #26467:
URL: https://github.com/apache/datafusion/pull/26467#discussion_r4340661799
##########
docs/source/user-guide/sql/operators.md:
##########
@@ -336,6 +338,36 @@ Not Regex Case-Insensitive Match
+---------------------------------------------------+
```
+(op_rlike)=
+
+### `RLIKE`
+
+Regex Match, planned as a call to [`regexp_like`](scalar_functions.md#regexp_like). `REGEXP` is a synonym.
+
+```sql
+> SELECT 'datafusion' RLIKE '^datafusion(-cli)*';
++------------------------------------------------------------+
+| regexp_like(Utf8("datafusion"),Utf8("^datafusion(-cli)*")) |
++------------------------------------------------------------+
+| true |
++------------------------------------------------------------+
+```
+
+(op_not_rlike)=
+
+### `NOT RLIKE`
+
+Not Regex Match, planned as `NOT regexp_like`. `NOT REGEXP` is a synonym.
Review Comment:
```suggestion
Not Regex Like, planned as `NOT regexp_like`. `NOT REGEXP` is a synonym.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support `RLIKE` and `REGEXP` operators [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #26467:
URL: https://github.com/apache/datafusion/pull/26467#issuecomment-6506088090
Do you think this is a PR worth pursuing @Omega359 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support `RLIKE` and `REGEXP` operators [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on code in PR #26467:
URL: https://github.com/apache/datafusion/pull/26467#discussion_r4340657383
##########
docs/source/user-guide/sql/operators.md:
##########
@@ -336,6 +338,36 @@ Not Regex Case-Insensitive Match
+---------------------------------------------------+
```
+(op_rlike)=
+
+### `RLIKE`
+
+Regex Match, planned as a call to [`regexp_like`](scalar_functions.md#regexp_like). `REGEXP` is a synonym.
Review Comment:
```suggestion
Regex Like, planned as a call to [`regexp_like`](scalar_functions.md#regexp_like). `REGEXP` is a synonym.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support `RLIKE` and `REGEXP` operators [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on PR #26467:
URL: https://github.com/apache/datafusion/pull/26467#issuecomment-6324628680
We should update datafusion/functions/src/regex/regexplike.rs to mention rlike/regexp and regenerate the scalar_functions.md (dev/update_function_docs.sh)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support `RLIKE` and `REGEXP` operators [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on code in PR #26467:
URL: https://github.com/apache/datafusion/pull/26467#discussion_r4340926657
##########
datafusion/sql/src/expr/mod.rs:
##########
@@ -1155,6 +1164,37 @@ impl<S: ContextProvider> SqlToRel<'_, S> {
)))
}
+ /// Plans `expr [NOT] RLIKE pattern` and `expr [NOT] REGEXP pattern` as a call
+ /// to the `regexp_like` function, resolved through the function registry.
+ fn sql_rlike_to_expr(
+ &self,
+ negated: bool,
+ expr: SQLExpr,
+ pattern: SQLExpr,
+ schema: &DFSchema,
+ planner_context: &mut PlannerContext,
+ ) -> Result<Expr> {
+ let fun = self
+ .context_provider
+ .get_function_meta("regexp_like")
+ .ok_or_else(|| {
+ plan_datafusion_err!(
+ "RLIKE and REGEXP require the 'regexp_like' function, which is not registered"
+ )
+ })?;
+ let expr = self.sql_expr_to_logical_expr(expr, schema, planner_context)?;
+ let pattern = self.sql_expr_to_logical_expr(pattern, schema, planner_context)?;
+ let regexp_like = match self.plan_scalar_udf(fun, vec![expr, pattern], schema)? {
Review Comment:
This is a slight change of behaviour from existing special cases such as TRIM, etc where this allows a downstream planner to override but existing ones do not. I think this is a good thing but I do think we should file an issue to update all the other special cases to use this new function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support `RLIKE` and `REGEXP` operators [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #26467:
URL: https://github.com/apache/datafusion/pull/26467#issuecomment-6506096209
Drafting as it is waiting to resolve reviewer feedback
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support `RLIKE` and `REGEXP` operators [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on code in PR #26467:
URL: https://github.com/apache/datafusion/pull/26467#discussion_r4340963789
##########
datafusion/sql/src/expr/mod.rs:
##########
@@ -1155,6 +1164,37 @@ impl<S: ContextProvider> SqlToRel<'_, S> {
)))
}
+ /// Plans `expr [NOT] RLIKE pattern` and `expr [NOT] REGEXP pattern` as a call
+ /// to the `regexp_like` function, resolved through the function registry.
+ fn sql_rlike_to_expr(
Review Comment:
There are already three sites in `datafusion/sql` that lower SQL syntax to a
registry-resolved function, and they all share the smae shape. This PR adds a fourth
that diverges on two axes:
| Site | Lookup | Runs `plan_scalar` hooks? | Error on missing fn |
|---|---|---|---|
| `sql_fn_name_to_expr` (`expr/function.rs:972`) | `get_function_meta(fn_name)` | no | `internal_datafusion_err!("Unable to find expected '{fn_name}' function")` |
| `sql_trim_to_expr` (`expr/mod.rs:1224`) | same | no | same message |
| `build_binary_expr`, `^` → `power` (`expr/binary_op.rs:78`) | same | no | same message |
| **`sql_rlike_to_expr`** | same | **yes** | `plan_datafusion_err!("RLIKE and REGEXP require …")` |
The lookup half follows precedent exactly. The plan_scalar and the error on missing differ.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support `RLIKE` and `REGEXP` operators [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed pull request #26467: feat: support `RLIKE` and `REGEXP` operators
URL: https://github.com/apache/datafusion/pull/26467
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org