You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Omega359 (via GitHub)" <gi...@apache.org> on 2024/02/05 19:05:46 UTC
[PR] Add regexp_like scalar function [arrow-datafusion]
Omega359 opened a new pull request, #9137:
URL: https://github.com/apache/arrow-datafusion/pull/9137
## Which issue does this PR close?
Closes #9102
## Rationale for this change
Add a new regexp_like function.
## What changes are included in this PR?
Code, tests, documentation, perf test
## Are these changes tested?
Yes,
## Are there any user-facing changes?
A new regexp_like function is available for use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `regexp_like, improve docs and examples for `regexp_match` [arrow-datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on code in PR #9137:
URL: https://github.com/apache/arrow-datafusion/pull/9137#discussion_r1482959000
##########
datafusion/physical-expr/src/regex_expressions.rs:
##########
@@ -332,10 +482,70 @@ pub fn specialize_regexp_replace<T: OffsetSizeTrait>(
#[cfg(test)]
mod tests {
- use super::*;
use arrow::array::*;
+
use datafusion_common::ScalarValue;
+ use super::*;
+
+ #[test]
+ fn test_case_sensitive_regexp_like() {
+ let values = StringArray::from(vec!["abc"; 5]);
+
+ let patterns =
+ StringArray::from(vec!["^(a)", "^(A)", "(b|d)", "(B|D)", "^(b|c)"]);
+
+ let mut expected_builder: BooleanBuilder = BooleanBuilder::new();
+ expected_builder.append_value(true);
+ expected_builder.append_value(false);
+ expected_builder.append_value(true);
+ expected_builder.append_value(false);
+ expected_builder.append_value(false);
+ let expected = expected_builder.finish();
Review Comment:
This is just me copy/pasting from some other bit of code and not thinking too much about it. I am still getting used to Rust's way of doing things :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `regexp_like, improve docs and examples for `regexp_match` [arrow-datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on code in PR #9137:
URL: https://github.com/apache/arrow-datafusion/pull/9137#discussion_r1483138131
##########
datafusion/physical-expr/src/regex_expressions.rs:
##########
@@ -54,7 +54,120 @@ macro_rules! fetch_string_arg {
}};
}
-/// extract a specific group from a string column, using a regular expression
+/// Tests a string using a regular expression returning true if at
+/// least one match, false otherwise.
+///
+/// The full list of supported features and syntax can be found at
+/// <https://docs.rs/regex/latest/regex/#syntax>
+///
+/// Supported flags can be found at
+/// <https://docs.rs/regex/latest/regex/#grouping-and-flags>
+///
+/// # Examples
+///
+/// ```
+/// # use datafusion::prelude::*;
+/// # use datafusion::error::Result;
+/// # #[tokio::main]
+/// # async fn main() -> Result<()> {
+/// let ctx = SessionContext::new();
+/// let df = ctx.read_csv("tests/data/regex.csv", CsvReadOptions::new()).await?;
+///
+/// // use the regexp_like function to test col 'values',
+/// // against patterns in col 'patterns' without flags
+/// let df = df.with_column(
+/// "a",
+/// regexp_like(vec![col("values"), col("patterns")])
+/// )?;
+/// // use the regexp_like function to test col 'values',
+/// // against patterns in col 'patterns' with flags
+/// let df = df.with_column(
+/// "b",
+/// regexp_like(vec![col("values"), col("patterns"), col("flags")])
+/// )?;
+/// // literals can be used as well with dataframe calls
+/// let df = df.with_column(
+/// "c",
+/// regexp_like(vec![lit("foobarbequebaz"), lit("(bar)(beque)")])
+/// )?;
+///
+/// df.show().await?;
Review Comment:
completed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add regexp_like scalar function [arrow-datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #9137:
URL: https://github.com/apache/arrow-datafusion/pull/9137#discussion_r1480449706
##########
datafusion-examples/examples/regexp.rs:
##########
@@ -0,0 +1,300 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
Review Comment:
I'm thinking on doc tests? like in `datafusion/core/src/dataframe/mod.rs`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add regexp_like scalar function [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9137:
URL: https://github.com/apache/arrow-datafusion/pull/9137#discussion_r1482886133
##########
datafusion/physical-expr/src/functions.rs:
##########
@@ -608,6 +608,27 @@ pub fn create_physical_fun(
_ => unreachable!(),
},
}),
+ BuiltinScalarFunction::RegexpLike => Arc::new(|args| match args[0].data_type() {
+ DataType::Utf8 => {
+ let func = invoke_on_array_if_regex_expressions_feature_flag!(
+ regexp_like,
+ i32,
+ "regexp_like"
+ );
+ make_scalar_function_inner(func)(args)
+ }
+ DataType::LargeUtf8 => {
+ let func = invoke_on_array_if_regex_expressions_feature_flag!(
+ regexp_like,
+ i64,
+ "regexp_like"
+ );
+ make_scalar_function_inner(func)(args)
+ }
+ other => {
+ internal_err!("Unsupported data type {other:?} for function regexp_like")
Review Comment:
That would be most appreciated. Cleaning up internal errors is very good improvement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `regexp_like, improve docs and examples for `regexp_match` [arrow-datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on code in PR #9137:
URL: https://github.com/apache/arrow-datafusion/pull/9137#discussion_r1482951315
##########
docs/source/user-guide/sql/scalar_functions.md:
##########
@@ -1785,10 +1893,10 @@ from_unixtime(expression)
- [array_sort](#array_sort)
- [array_cat](#array_cat)
- [array_concat](#array_concat)
-- [array_contains](#array_contains)
Review Comment:
I removed this from the PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `regexp_like, improve docs and examples for `regexp_match` [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #9137:
URL: https://github.com/apache/arrow-datafusion/pull/9137
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `regexp_like, improve docs and examples for `regexp_match` [arrow-datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on code in PR #9137:
URL: https://github.com/apache/arrow-datafusion/pull/9137#discussion_r1482950402
##########
datafusion/physical-expr/src/functions.rs:
##########
@@ -608,6 +608,27 @@ pub fn create_physical_fun(
_ => unreachable!(),
},
}),
+ BuiltinScalarFunction::RegexpLike => Arc::new(|args| match args[0].data_type() {
+ DataType::Utf8 => {
+ let func = invoke_on_array_if_regex_expressions_feature_flag!(
+ regexp_like,
+ i32,
+ "regexp_like"
+ );
+ make_scalar_function_inner(func)(args)
+ }
+ DataType::LargeUtf8 => {
+ let func = invoke_on_array_if_regex_expressions_feature_flag!(
+ regexp_like,
+ i64,
+ "regexp_like"
+ );
+ make_scalar_function_inner(func)(args)
+ }
+ other => {
+ internal_err!("Unsupported data type {other:?} for function regexp_like")
Review Comment:
[Filed](https://github.com/apache/arrow-datafusion/issues/9164)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add regexp_like scalar function [arrow-datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on PR #9137:
URL: https://github.com/apache/arrow-datafusion/pull/9137#issuecomment-1927877337
FYI - the prettier error is not something I see on my machine
```
arrow-datafusion on feature/regexp_like [$!] is 📦 v35.0.0 via 🦀 v1.75.0 on ☁️ (us-east-1)
❯ prettier --version
3.2.4
arrow-datafusion on feature/regexp_like [$!] is 📦 v35.0.0 via 🦀 v1.75.0 on ☁️ (us-east-1)
❯ prettier -w docs/**/scalar_functions.md
docs/source/user-guide/sql/scalar_functions.md 356ms (unchanged)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add regexp_like scalar function [arrow-datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on code in PR #9137:
URL: https://github.com/apache/arrow-datafusion/pull/9137#discussion_r1480333453
##########
datafusion-examples/examples/regexp.rs:
##########
@@ -0,0 +1,300 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
Review Comment:
When I started with DataFusion it was the examples that gave me the initial base to work from - especially since this projects was essentially my first time with Rust code. I personally think having both dataframe and SQL examples for the same thing to be quite helpful for those getting started who are not already fluent with rust or datafusion.
I was actually thinking of spending some time (when I had it) to enhance the scalar docs to add examples where missing and to add whole new sections to the examples as above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add regexp_like scalar function [arrow-datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #9137:
URL: https://github.com/apache/arrow-datafusion/pull/9137#discussion_r1480302825
##########
datafusion/physical-expr/benches/regexp.rs:
##########
@@ -0,0 +1,133 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+extern crate criterion;
+
+use std::iter;
+use std::sync::Arc;
+
+use arrow_array::builder::StringBuilder;
+use arrow_array::{ArrayRef, StringArray};
+use criterion::{black_box, criterion_group, criterion_main, Criterion};
+use rand::distributions::Alphanumeric;
+use rand::rngs::ThreadRng;
+use rand::seq::SliceRandom;
+use rand::Rng;
+
+use datafusion_physical_expr::regex_expressions::{
+ regexp_like, regexp_match, regexp_replace,
+};
+
+fn data(rng: &mut ThreadRng) -> StringArray {
Review Comment:
that bench is really needed taking into account how much probs we had with regex performance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add regexp_like scalar function [arrow-datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on code in PR #9137:
URL: https://github.com/apache/arrow-datafusion/pull/9137#discussion_r1480314705
##########
docs/source/user-guide/sql/scalar_functions.md:
##########
@@ -1785,10 +1893,10 @@ from_unixtime(expression)
- [array_sort](#array_sort)
- [array_cat](#array_cat)
- [array_concat](#array_concat)
-- [array_contains](#array_contains)
Review Comment:
Indeed you are correct, there are likely a couple of places where I fixed things as I saw them. Would you like those pulled into a separate PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `regexp_like, improve docs and examples for `regexp_match` [arrow-datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on code in PR #9137:
URL: https://github.com/apache/arrow-datafusion/pull/9137#discussion_r1482941989
##########
datafusion/proto/proto/datafusion.proto:
##########
@@ -677,6 +677,7 @@ enum ScalarFunction {
InStr = 132;
MakeDate = 133;
ArrayReverse = 134;
+ RegexpLike = 135;
Review Comment:
It's in there - RegexpMatch = 21; To not break any existing persisted proto files or interactions with code generated from other versions of this file you can't renumber these.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `regexp_like, improve docs and examples for `regexp_match` [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9137:
URL: https://github.com/apache/arrow-datafusion/pull/9137#discussion_r1482891745
##########
datafusion-examples/README.md:
##########
@@ -52,21 +52,22 @@ cargo run --example csv_sql
- [`dataframe_output.rs`](examples/dataframe_output.rs): Examples of methods which write data out from a DataFrame
- [`dataframe_in_memory.rs`](examples/dataframe_in_memory.rs): Run a query using a DataFrame against data in memory
- [`deserialize_to_struct.rs`](examples/deserialize_to_struct.rs): Convert query results into rust structs using serde
-- [`expr_api.rs`](examples/expr_api.rs): Create, execute, simplify and anaylze `Expr`s
+- [`expr_api.rs`](examples/expr_api.rs): Create, execute, simplify and analyze `Expr`s
- [`flight_sql_server.rs`](examples/flight/flight_sql_server.rs): Run DataFusion as a standalone process and execute SQL queries from JDBC clients
+- [`make_date.rs`](examples/make_date.rs): Examples of using the make_date function
Review Comment:
Thank you 🙏
##########
datafusion/physical-expr/src/regex_expressions.rs:
##########
@@ -54,7 +54,120 @@ macro_rules! fetch_string_arg {
}};
}
-/// extract a specific group from a string column, using a regular expression
+/// Tests a string using a regular expression returning true if at
+/// least one match, false otherwise.
+///
+/// The full list of supported features and syntax can be found at
+/// <https://docs.rs/regex/latest/regex/#syntax>
+///
+/// Supported flags can be found at
+/// <https://docs.rs/regex/latest/regex/#grouping-and-flags>
+///
+/// # Examples
+///
+/// ```
+/// # use datafusion::prelude::*;
+/// # use datafusion::error::Result;
+/// # #[tokio::main]
+/// # async fn main() -> Result<()> {
+/// let ctx = SessionContext::new();
+/// let df = ctx.read_csv("tests/data/regex.csv", CsvReadOptions::new()).await?;
+///
+/// // use the regexp_like function to test col 'values',
+/// // against patterns in col 'patterns' without flags
+/// let df = df.with_column(
+/// "a",
+/// regexp_like(vec![col("values"), col("patterns")])
+/// )?;
+/// // use the regexp_like function to test col 'values',
+/// // against patterns in col 'patterns' with flags
+/// let df = df.with_column(
+/// "b",
+/// regexp_like(vec![col("values"), col("patterns"), col("flags")])
+/// )?;
+/// // literals can be used as well with dataframe calls
+/// let df = df.with_column(
+/// "c",
+/// regexp_like(vec![lit("foobarbequebaz"), lit("(bar)(beque)")])
+/// )?;
+///
+/// df.show().await?;
Review Comment:
One way to improve these examples would be to bake in here the expected results
For example, using pretty_print on the collected result and asserting they are the same.
Or maybe just copy/pasting the output as a comment
##########
datafusion/proto/proto/datafusion.proto:
##########
@@ -677,6 +677,7 @@ enum ScalarFunction {
InStr = 132;
MakeDate = 133;
ArrayReverse = 134;
+ RegexpLike = 135;
Review Comment:
Doesn't this need RegexpMatch as well? I may be missing something
##########
datafusion/physical-expr/src/regex_expressions.rs:
##########
@@ -332,10 +482,70 @@ pub fn specialize_regexp_replace<T: OffsetSizeTrait>(
#[cfg(test)]
mod tests {
- use super::*;
use arrow::array::*;
+
use datafusion_common::ScalarValue;
+ use super::*;
+
+ #[test]
+ fn test_case_sensitive_regexp_like() {
+ let values = StringArray::from(vec!["abc"; 5]);
+
+ let patterns =
+ StringArray::from(vec!["^(a)", "^(A)", "(b|d)", "(B|D)", "^(b|c)"]);
+
+ let mut expected_builder: BooleanBuilder = BooleanBuilder::new();
+ expected_builder.append_value(true);
+ expected_builder.append_value(false);
+ expected_builder.append_value(true);
+ expected_builder.append_value(false);
+ expected_builder.append_value(false);
+ let expected = expected_builder.finish();
Review Comment:
This probably just follows the exisitng code in this file, but it is a bit more concise to build `BooleanArry`s like:
Something like this (untested):
```suggestion
let expected = BooleanArray::from(vec![true, false, true, false, false]);
```
##########
datafusion/physical-expr/src/regex_expressions.rs:
##########
@@ -54,7 +54,120 @@ macro_rules! fetch_string_arg {
}};
}
-/// extract a specific group from a string column, using a regular expression
+/// Tests a string using a regular expression returning true if at
+/// least one match, false otherwise.
+///
+/// The full list of supported features and syntax can be found at
+/// <https://docs.rs/regex/latest/regex/#syntax>
+///
+/// Supported flags can be found at
+/// <https://docs.rs/regex/latest/regex/#grouping-and-flags>
+///
+/// # Examples
+///
+/// ```
+/// # use datafusion::prelude::*;
+/// # use datafusion::error::Result;
+/// # #[tokio::main]
+/// # async fn main() -> Result<()> {
+/// let ctx = SessionContext::new();
+/// let df = ctx.read_csv("tests/data/regex.csv", CsvReadOptions::new()).await?;
+///
+/// // use the regexp_like function to test col 'values',
+/// // against patterns in col 'patterns' without flags
+/// let df = df.with_column(
+/// "a",
+/// regexp_like(vec![col("values"), col("patterns")])
+/// )?;
+/// // use the regexp_like function to test col 'values',
+/// // against patterns in col 'patterns' with flags
+/// let df = df.with_column(
+/// "b",
+/// regexp_like(vec![col("values"), col("patterns"), col("flags")])
+/// )?;
+/// // literals can be used as well with dataframe calls
+/// let df = df.with_column(
+/// "c",
+/// regexp_like(vec![lit("foobarbequebaz"), lit("(bar)(beque)")])
+/// )?;
+///
+/// df.show().await?;
+///
+/// # Ok(())
+/// # }
+/// ```
+pub fn regexp_like<T: OffsetSizeTrait>(args: &[ArrayRef]) -> Result<ArrayRef> {
+ match args.len() {
+ 2 => {
+ let values = as_generic_string_array::<T>(&args[0])?;
+ let regex = as_generic_string_array::<T>(&args[1])?;
+ let array = arrow_string::regexp::regexp_is_match_utf8(values, regex, None)
+ .map_err(|e| arrow_datafusion_err!(e))?;
+
+ Ok(Arc::new(array) as ArrayRef)
+ }
+ 3 => {
+ let values = as_generic_string_array::<T>(&args[0])?;
+ let regex = as_generic_string_array::<T>(&args[1])?;
+ let flags = as_generic_string_array::<T>(&args[2])?;
+
+ if flags.iter().any(|s| s == Some("g")) {
+ return plan_err!("regexp_like() does not support the \"global\" option");
+ }
+
+ let array = arrow_string::regexp::regexp_is_match_utf8(values, regex, Some(flags))
Review Comment:
👍
##########
datafusion/sqllogictest/test_files/regexp.slt:
##########
@@ -0,0 +1,303 @@
+# Licensed to the Apache Software Foundation (ASF) under one
Review Comment:
Thank you -- this is much better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `regexp_like, improve docs and examples for `regexp_match` [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9137:
URL: https://github.com/apache/arrow-datafusion/pull/9137#discussion_r1482971363
##########
datafusion/physical-expr/src/regex_expressions.rs:
##########
@@ -332,10 +482,70 @@ pub fn specialize_regexp_replace<T: OffsetSizeTrait>(
#[cfg(test)]
mod tests {
- use super::*;
use arrow::array::*;
+
use datafusion_common::ScalarValue;
+ use super::*;
+
+ #[test]
+ fn test_case_sensitive_regexp_like() {
+ let values = StringArray::from(vec!["abc"; 5]);
+
+ let patterns =
+ StringArray::from(vec!["^(a)", "^(A)", "(b|d)", "(B|D)", "^(b|c)"]);
+
+ let mut expected_builder: BooleanBuilder = BooleanBuilder::new();
+ expected_builder.append_value(true);
+ expected_builder.append_value(false);
+ expected_builder.append_value(true);
+ expected_builder.append_value(false);
+ expected_builder.append_value(false);
+ let expected = expected_builder.finish();
Review Comment:
I think following the existing structure is fine, I just figured I would point it out
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add regexp_like scalar function [arrow-datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on code in PR #9137:
URL: https://github.com/apache/arrow-datafusion/pull/9137#discussion_r1480491947
##########
datafusion-examples/examples/regexp.rs:
##########
@@ -0,0 +1,300 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
Review Comment:
That could work though it could be rather large doc. I'll update this PR in the next day or two with that change. I should likely do that with the to_timestamp example as well (in a separate PR of course).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add regexp_like scalar function [arrow-datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on code in PR #9137:
URL: https://github.com/apache/arrow-datafusion/pull/9137#discussion_r1480306240
##########
datafusion/physical-expr/src/functions.rs:
##########
@@ -608,6 +608,27 @@ pub fn create_physical_fun(
_ => unreachable!(),
},
}),
+ BuiltinScalarFunction::RegexpLike => Arc::new(|args| match args[0].data_type() {
+ DataType::Utf8 => {
+ let func = invoke_on_array_if_regex_expressions_feature_flag!(
+ regexp_like,
+ i32,
+ "regexp_like"
+ );
+ make_scalar_function_inner(func)(args)
+ }
+ DataType::LargeUtf8 => {
+ let func = invoke_on_array_if_regex_expressions_feature_flag!(
+ regexp_like,
+ i64,
+ "regexp_like"
+ );
+ make_scalar_function_inner(func)(args)
+ }
+ other => {
+ internal_err!("Unsupported data type {other:?} for function regexp_like")
Review Comment:
There is quite a few instances in the code that have this pattern (I was mirroring the existing regexp_... functions there) - I was going to create a followup PR that fixes them across the whole code base
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add regexp_like scalar function [arrow-datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #9137:
URL: https://github.com/apache/arrow-datafusion/pull/9137#discussion_r1480303798
##########
datafusion/physical-expr/src/functions.rs:
##########
@@ -608,6 +608,27 @@ pub fn create_physical_fun(
_ => unreachable!(),
},
}),
+ BuiltinScalarFunction::RegexpLike => Arc::new(|args| match args[0].data_type() {
+ DataType::Utf8 => {
+ let func = invoke_on_array_if_regex_expressions_feature_flag!(
+ regexp_like,
+ i32,
+ "regexp_like"
+ );
+ make_scalar_function_inner(func)(args)
+ }
+ DataType::LargeUtf8 => {
+ let func = invoke_on_array_if_regex_expressions_feature_flag!(
+ regexp_like,
+ i64,
+ "regexp_like"
+ );
+ make_scalar_function_inner(func)(args)
+ }
+ other => {
+ internal_err!("Unsupported data type {other:?} for function regexp_like")
Review Comment:
```suggestion
exec_err!("Unsupported data type {other:?} for function regexp_like")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add regexp_like scalar function [arrow-datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #9137:
URL: https://github.com/apache/arrow-datafusion/pull/9137#discussion_r1480300405
##########
datafusion-examples/examples/regexp.rs:
##########
@@ -0,0 +1,300 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
Review Comment:
IMHO I like adding examples for introduced functions, but we probably can do that on SQL only? like https://spark.apache.org/docs/latest/api/sql/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add regexp_like scalar function [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9137:
URL: https://github.com/apache/arrow-datafusion/pull/9137#discussion_r1482886133
##########
datafusion/physical-expr/src/functions.rs:
##########
@@ -608,6 +608,27 @@ pub fn create_physical_fun(
_ => unreachable!(),
},
}),
+ BuiltinScalarFunction::RegexpLike => Arc::new(|args| match args[0].data_type() {
+ DataType::Utf8 => {
+ let func = invoke_on_array_if_regex_expressions_feature_flag!(
+ regexp_like,
+ i32,
+ "regexp_like"
+ );
+ make_scalar_function_inner(func)(args)
+ }
+ DataType::LargeUtf8 => {
+ let func = invoke_on_array_if_regex_expressions_feature_flag!(
+ regexp_like,
+ i64,
+ "regexp_like"
+ );
+ make_scalar_function_inner(func)(args)
+ }
+ other => {
+ internal_err!("Unsupported data type {other:?} for function regexp_like")
Review Comment:
That would be most appreciated. Cleaning up internal errors that are possible to trigger via incrrect queries (rather than a bug( is very good improvement
##########
datafusion/physical-expr/src/functions.rs:
##########
@@ -608,6 +608,27 @@ pub fn create_physical_fun(
_ => unreachable!(),
},
}),
+ BuiltinScalarFunction::RegexpLike => Arc::new(|args| match args[0].data_type() {
+ DataType::Utf8 => {
+ let func = invoke_on_array_if_regex_expressions_feature_flag!(
+ regexp_like,
+ i32,
+ "regexp_like"
+ );
+ make_scalar_function_inner(func)(args)
+ }
+ DataType::LargeUtf8 => {
+ let func = invoke_on_array_if_regex_expressions_feature_flag!(
+ regexp_like,
+ i64,
+ "regexp_like"
+ );
+ make_scalar_function_inner(func)(args)
+ }
+ other => {
+ internal_err!("Unsupported data type {other:?} for function regexp_like")
Review Comment:
That would be most appreciated. Cleaning up internal errors that are possible to trigger via incrrect queries (rather than a bug) is very good improvement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `regexp_like, improve docs and examples for `regexp_match` [arrow-datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on code in PR #9137:
URL: https://github.com/apache/arrow-datafusion/pull/9137#discussion_r1482954125
##########
datafusion/physical-expr/src/regex_expressions.rs:
##########
@@ -54,7 +54,120 @@ macro_rules! fetch_string_arg {
}};
}
-/// extract a specific group from a string column, using a regular expression
+/// Tests a string using a regular expression returning true if at
+/// least one match, false otherwise.
+///
+/// The full list of supported features and syntax can be found at
+/// <https://docs.rs/regex/latest/regex/#syntax>
+///
+/// Supported flags can be found at
+/// <https://docs.rs/regex/latest/regex/#grouping-and-flags>
+///
+/// # Examples
+///
+/// ```
+/// # use datafusion::prelude::*;
+/// # use datafusion::error::Result;
+/// # #[tokio::main]
+/// # async fn main() -> Result<()> {
+/// let ctx = SessionContext::new();
+/// let df = ctx.read_csv("tests/data/regex.csv", CsvReadOptions::new()).await?;
+///
+/// // use the regexp_like function to test col 'values',
+/// // against patterns in col 'patterns' without flags
+/// let df = df.with_column(
+/// "a",
+/// regexp_like(vec![col("values"), col("patterns")])
+/// )?;
+/// // use the regexp_like function to test col 'values',
+/// // against patterns in col 'patterns' with flags
+/// let df = df.with_column(
+/// "b",
+/// regexp_like(vec![col("values"), col("patterns"), col("flags")])
+/// )?;
+/// // literals can be used as well with dataframe calls
+/// let df = df.with_column(
+/// "c",
+/// regexp_like(vec![lit("foobarbequebaz"), lit("(bar)(beque)")])
+/// )?;
+///
+/// df.show().await?;
Review Comment:
Good point, I can do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add regexp_like scalar function [arrow-datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on code in PR #9137:
URL: https://github.com/apache/arrow-datafusion/pull/9137#discussion_r1480879284
##########
datafusion-examples/examples/regexp.rs:
##########
@@ -0,0 +1,300 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add regexp_like scalar function [arrow-datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #9137:
URL: https://github.com/apache/arrow-datafusion/pull/9137#discussion_r1480306216
##########
docs/source/user-guide/sql/scalar_functions.md:
##########
@@ -1785,10 +1893,10 @@ from_unixtime(expression)
- [array_sort](#array_sort)
- [array_cat](#array_cat)
- [array_concat](#array_concat)
-- [array_contains](#array_contains)
Review Comment:
this is not related to PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add regexp_like scalar function [arrow-datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on code in PR #9137:
URL: https://github.com/apache/arrow-datafusion/pull/9137#discussion_r1480333453
##########
datafusion-examples/examples/regexp.rs:
##########
@@ -0,0 +1,300 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
Review Comment:
When I started with DataFusion it was the examples that gave me the initial base to work from - especially since this projects was essentially my first time with Rust code. I personally think having both dataframe and SQL examples for the same thing to be quite helpful for those getting started who are not already
I was actually thinking of spending some time (when I had it) to enhance the scalar docs to add examples where missing and to add whole new sections to the examples as above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `regexp_like, improve docs and examples for `regexp_match` [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9137:
URL: https://github.com/apache/arrow-datafusion/pull/9137#issuecomment-1935216474
Thanks again @Omega359
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `regexp_like, improve docs and examples for `regexp_match` [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9137:
URL: https://github.com/apache/arrow-datafusion/pull/9137#discussion_r1483261487
##########
datafusion-examples/examples/regexp.rs:
##########
@@ -49,29 +43,83 @@ async fn main() -> Result<()> {
//regexp_like examples
//
//
- // regexp_like format is (regexp_replace(text, regex[, flags])
+ // regexp_like format is (regexp_like(text, regex[, flags])
//
// use sql and regexp_like function to test col 'values', against patterns in col 'patterns' without flags
- let df = ctx
+ let result = ctx
.sql("select regexp_like(values, patterns) from examples")
+ .await?
+ .collect()
.await?;
- // print the results
- df.show().await?;
-
- // use dataframe and regexp_like function to test col 'values', against patterns in col 'patterns' with flags
- let df = ctx
+ assert_batches_eq!(
Review Comment:
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org