You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Lordworms (via GitHub)" <gi...@apache.org> on 2024/06/11 21:19:35 UTC
[PR] Support for contains function in datafusion substrait consumer [datafusion]
Lordworms opened a new pull request, #10879:
URL: https://github.com/apache/datafusion/pull/10879
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Closes #10861
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add contains function, and support in datafusion substrait consumer [datafusion]
Posted by "Lordworms (via GitHub)" <gi...@apache.org>.
Lordworms commented on code in PR #10879:
URL: https://github.com/apache/datafusion/pull/10879#discussion_r1637401620
##########
datafusion/functions/src/string/contains.rs:
##########
@@ -0,0 +1,143 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::utils::make_scalar_function;
+use arrow::array::{ArrayRef, OffsetSizeTrait};
+use arrow::datatypes::DataType;
+use arrow::datatypes::DataType::Boolean;
+use datafusion_common::cast::as_generic_string_array;
+use datafusion_common::DataFusionError;
+use datafusion_common::Result;
+use datafusion_common::{arrow_datafusion_err, exec_err};
+use datafusion_expr::ScalarUDFImpl;
+use datafusion_expr::TypeSignature::Exact;
+use datafusion_expr::{ColumnarValue, Signature, Volatility};
+use std::any::Any;
+use std::sync::Arc;
+#[derive(Debug)]
+pub struct ContainsFunc {
+ signature: Signature,
+}
+
+impl Default for ContainsFunc {
+ fn default() -> Self {
+ ContainsFunc::new()
+ }
+}
+
+impl ContainsFunc {
+ pub fn new() -> Self {
+ use DataType::*;
+ Self {
+ signature: Signature::one_of(
+ vec![Exact(vec![Utf8, Utf8]), Exact(vec![LargeUtf8, LargeUtf8])],
+ Volatility::Immutable,
+ ),
+ }
+ }
+}
+
+impl ScalarUDFImpl for ContainsFunc {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "contains"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, _: &[DataType]) -> Result<DataType> {
+ Ok(Boolean)
+ }
+
+ fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
+ match args[0].data_type() {
+ DataType::Utf8 => make_scalar_function(contains::<i32>, vec![])(args),
+ DataType::LargeUtf8 => make_scalar_function(contains::<i64>, vec![])(args),
+ other => {
+ exec_err!("unsupported data type {other:?} for function contains")
+ }
+ }
+ }
+}
+
+/// use regexp_is_match_utf8_scalar to do the calculation for contains
Review Comment:
The reason is that in substrait test, we would directly look all the scalar function and find if there is a name which fits, if we just want to implement the semantic of contains I think adding a rewrite is fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add contains function, and support in datafusion substrait consumer [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10879:
URL: https://github.com/apache/datafusion/pull/10879#issuecomment-2170301722
Thanks again @waynexia @Lordworms and @Weijun-H !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support for contains function in datafusion substrait consumer [datafusion]
Posted by "waynexia (via GitHub)" <gi...@apache.org>.
waynexia commented on code in PR #10879:
URL: https://github.com/apache/datafusion/pull/10879#discussion_r1635767599
##########
datafusion/functions/src/string/contains.rs:
##########
@@ -0,0 +1,143 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::utils::make_scalar_function;
+use arrow::array::{ArrayRef, OffsetSizeTrait};
+use arrow::datatypes::DataType;
+use arrow::datatypes::DataType::Boolean;
+use datafusion_common::cast::as_generic_string_array;
+use datafusion_common::DataFusionError;
+use datafusion_common::Result;
+use datafusion_common::{arrow_datafusion_err, exec_err};
+use datafusion_expr::ScalarUDFImpl;
+use datafusion_expr::TypeSignature::Exact;
+use datafusion_expr::{ColumnarValue, Signature, Volatility};
+use std::any::Any;
+use std::sync::Arc;
+#[derive(Debug)]
+pub struct ContainsFunc {
+ signature: Signature,
+}
+
+impl Default for ContainsFunc {
+ fn default() -> Self {
+ ContainsFunc::new()
+ }
+}
+
+impl ContainsFunc {
+ pub fn new() -> Self {
+ use DataType::*;
+ Self {
+ signature: Signature::one_of(
+ vec![Exact(vec![Utf8, Utf8]), Exact(vec![LargeUtf8, LargeUtf8])],
+ Volatility::Immutable,
+ ),
+ }
+ }
+}
+
+impl ScalarUDFImpl for ContainsFunc {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "contains"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, _: &[DataType]) -> Result<DataType> {
+ Ok(Boolean)
+ }
+
+ fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
+ match args[0].data_type() {
+ DataType::Utf8 => make_scalar_function(contains::<i32>, vec![])(args),
+ DataType::LargeUtf8 => make_scalar_function(contains::<i64>, vec![])(args),
+ other => {
+ exec_err!("unsupported data type {other:?} for function contains")
+ }
+ }
+ }
+}
+
+/// use regexp_is_match_utf8_scalar to do the calculation for contains
+pub fn contains<T: OffsetSizeTrait>(
+ args: &[ArrayRef],
+) -> Result<ArrayRef, DataFusionError> {
+ let mod_str = as_generic_string_array::<T>(&args[0])?;
+ let match_str = as_generic_string_array::<T>(&args[1])?;
+ let res = arrow::compute::kernels::comparison::regexp_is_match_utf8(
Review Comment:
Do we need to escape search string as it's used in regexp? Wondering what's the result of `contains("abc.*efg", ".*")`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add contains function, and support in datafusion substrait consumer [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10879:
URL: https://github.com/apache/datafusion/pull/10879#discussion_r1638340942
##########
datafusion/functions/src/string/contains.rs:
##########
@@ -0,0 +1,143 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::utils::make_scalar_function;
+use arrow::array::{ArrayRef, OffsetSizeTrait};
+use arrow::datatypes::DataType;
+use arrow::datatypes::DataType::Boolean;
+use datafusion_common::cast::as_generic_string_array;
+use datafusion_common::DataFusionError;
+use datafusion_common::Result;
+use datafusion_common::{arrow_datafusion_err, exec_err};
+use datafusion_expr::ScalarUDFImpl;
+use datafusion_expr::TypeSignature::Exact;
+use datafusion_expr::{ColumnarValue, Signature, Volatility};
+use std::any::Any;
+use std::sync::Arc;
+#[derive(Debug)]
+pub struct ContainsFunc {
+ signature: Signature,
+}
+
+impl Default for ContainsFunc {
+ fn default() -> Self {
+ ContainsFunc::new()
+ }
+}
+
+impl ContainsFunc {
+ pub fn new() -> Self {
+ use DataType::*;
+ Self {
+ signature: Signature::one_of(
+ vec![Exact(vec![Utf8, Utf8]), Exact(vec![LargeUtf8, LargeUtf8])],
+ Volatility::Immutable,
+ ),
+ }
+ }
+}
+
+impl ScalarUDFImpl for ContainsFunc {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "contains"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, _: &[DataType]) -> Result<DataType> {
+ Ok(Boolean)
+ }
+
+ fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
+ match args[0].data_type() {
+ DataType::Utf8 => make_scalar_function(contains::<i32>, vec![])(args),
+ DataType::LargeUtf8 => make_scalar_function(contains::<i64>, vec![])(args),
+ other => {
+ exec_err!("unsupported data type {other:?} for function contains")
+ }
+ }
+ }
+}
+
+/// use regexp_is_match_utf8_scalar to do the calculation for contains
Review Comment:
makes sense -- thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add contains function, and support in datafusion substrait consumer [datafusion]
Posted by "Lordworms (via GitHub)" <gi...@apache.org>.
Lordworms commented on code in PR #10879:
URL: https://github.com/apache/datafusion/pull/10879#discussion_r1639143722
##########
datafusion/functions/src/string/mod.rs:
##########
@@ -149,6 +149,9 @@ pub mod expr_fn {
),(
uuid,
"returns uuid v4 as a string value",
+ ), (
+ contains,
+ "Return true if search_string is found within string.",
Review Comment:
sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add contains function, and support in datafusion substrait consumer [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10879:
URL: https://github.com/apache/datafusion/pull/10879#discussion_r1641295837
##########
datafusion/functions/src/string/contains.rs:
##########
@@ -0,0 +1,143 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::utils::make_scalar_function;
+use arrow::array::{ArrayRef, OffsetSizeTrait};
+use arrow::datatypes::DataType;
+use arrow::datatypes::DataType::Boolean;
+use datafusion_common::cast::as_generic_string_array;
+use datafusion_common::DataFusionError;
+use datafusion_common::Result;
+use datafusion_common::{arrow_datafusion_err, exec_err};
+use datafusion_expr::ScalarUDFImpl;
+use datafusion_expr::TypeSignature::Exact;
+use datafusion_expr::{ColumnarValue, Signature, Volatility};
+use std::any::Any;
+use std::sync::Arc;
+#[derive(Debug)]
+pub struct ContainsFunc {
+ signature: Signature,
+}
+
+impl Default for ContainsFunc {
+ fn default() -> Self {
+ ContainsFunc::new()
+ }
+}
+
+impl ContainsFunc {
+ pub fn new() -> Self {
+ use DataType::*;
+ Self {
+ signature: Signature::one_of(
+ vec![Exact(vec![Utf8, Utf8]), Exact(vec![LargeUtf8, LargeUtf8])],
+ Volatility::Immutable,
+ ),
+ }
+ }
+}
+
+impl ScalarUDFImpl for ContainsFunc {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "contains"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, _: &[DataType]) -> Result<DataType> {
+ Ok(Boolean)
+ }
+
+ fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
+ match args[0].data_type() {
+ DataType::Utf8 => make_scalar_function(contains::<i32>, vec![])(args),
+ DataType::LargeUtf8 => make_scalar_function(contains::<i64>, vec![])(args),
+ other => {
+ exec_err!("unsupported data type {other:?} for function contains")
+ }
+ }
+ }
+}
+
+/// use regexp_is_match_utf8_scalar to do the calculation for contains
+pub fn contains<T: OffsetSizeTrait>(
+ args: &[ArrayRef],
+) -> Result<ArrayRef, DataFusionError> {
+ let mod_str = as_generic_string_array::<T>(&args[0])?;
+ let match_str = as_generic_string_array::<T>(&args[1])?;
+ let res = arrow::compute::kernels::comparison::regexp_is_match_utf8(
Review Comment:
To keep the review queue clear, I filed https://github.com/apache/datafusion/issues/10929 to follow up on this item and will merge this PR in
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add contains function, and support in datafusion substrait consumer [datafusion]
Posted by "Weijun-H (via GitHub)" <gi...@apache.org>.
Weijun-H commented on code in PR #10879:
URL: https://github.com/apache/datafusion/pull/10879#discussion_r1639203659
##########
docs/source/user-guide/sql/scalar_functions.md:
##########
@@ -1443,6 +1443,19 @@ position(substr in origstr)
- **substr**: The pattern string.
- **origstr**: The model string.
+### `contains`
+
+Return true if search_string is found within string.
+
+```
+contains(substr in origstr)
Review Comment:
```suggestion
contains(string, search_string)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add contains function, and support in datafusion substrait consumer [datafusion]
Posted by "Lordworms (via GitHub)" <gi...@apache.org>.
Lordworms commented on code in PR #10879:
URL: https://github.com/apache/datafusion/pull/10879#discussion_r1639206381
##########
docs/source/user-guide/sql/scalar_functions.md:
##########
@@ -1443,6 +1443,19 @@ position(substr in origstr)
- **substr**: The pattern string.
- **origstr**: The model string.
+### `contains`
+
+Return true if search_string is found within string.
+
+```
+contains(substr in origstr)
Review Comment:
Thanks a lot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add contains function, and support in datafusion substrait consumer [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10879:
URL: https://github.com/apache/datafusion/pull/10879#discussion_r1636789004
##########
datafusion/functions/src/string/contains.rs:
##########
@@ -0,0 +1,143 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::utils::make_scalar_function;
+use arrow::array::{ArrayRef, OffsetSizeTrait};
+use arrow::datatypes::DataType;
+use arrow::datatypes::DataType::Boolean;
+use datafusion_common::cast::as_generic_string_array;
+use datafusion_common::DataFusionError;
+use datafusion_common::Result;
+use datafusion_common::{arrow_datafusion_err, exec_err};
+use datafusion_expr::ScalarUDFImpl;
+use datafusion_expr::TypeSignature::Exact;
+use datafusion_expr::{ColumnarValue, Signature, Volatility};
+use std::any::Any;
+use std::sync::Arc;
+#[derive(Debug)]
+pub struct ContainsFunc {
+ signature: Signature,
+}
+
+impl Default for ContainsFunc {
+ fn default() -> Self {
+ ContainsFunc::new()
+ }
+}
+
+impl ContainsFunc {
+ pub fn new() -> Self {
+ use DataType::*;
+ Self {
+ signature: Signature::one_of(
+ vec![Exact(vec![Utf8, Utf8]), Exact(vec![LargeUtf8, LargeUtf8])],
+ Volatility::Immutable,
+ ),
+ }
+ }
+}
+
+impl ScalarUDFImpl for ContainsFunc {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "contains"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, _: &[DataType]) -> Result<DataType> {
+ Ok(Boolean)
+ }
+
+ fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
+ match args[0].data_type() {
+ DataType::Utf8 => make_scalar_function(contains::<i32>, vec![])(args),
+ DataType::LargeUtf8 => make_scalar_function(contains::<i64>, vec![])(args),
+ other => {
+ exec_err!("unsupported data type {other:?} for function contains")
+ }
+ }
+ }
+}
+
+/// use regexp_is_match_utf8_scalar to do the calculation for contains
Review Comment:
What are the semantics of substring supposed to be? If the idea is that contains only looks for a substring, perhaps instead of implementing `invoke()` directly, we could implement https://docs.rs/datafusion/latest/datafusion/logical_expr/struct.ScalarUDF.html#method.simplify and rewrite from `contains(x, y)` to `x LIKE '%y%` (aka rewrite to `Expr::Like` which is already pretty well optimized I tink
##########
datafusion/functions/src/string/mod.rs:
##########
@@ -149,6 +149,9 @@ pub mod expr_fn {
),(
uuid,
"returns uuid v4 as a string value",
+ ), (
+ contains,
+ "Return true if search_string is found within string.",
Review Comment:
I think we should also mention here that search_string is treated like a regex
##########
datafusion/functions/src/string/mod.rs:
##########
@@ -149,6 +149,9 @@ pub mod expr_fn {
),(
uuid,
"returns uuid v4 as a string value",
+ ), (
+ contains,
+ "Return true if search_string is found within string.",
Review Comment:
I think we should also mention here that search_string is treated like a regex if that is the intention
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add contains function, and support in datafusion substrait consumer [datafusion]
Posted by "Lordworms (via GitHub)" <gi...@apache.org>.
Lordworms commented on PR #10879:
URL: https://github.com/apache/datafusion/pull/10879#issuecomment-2164230455
> Can we also please add an entry to the documentation about this new function? https://datafusion.apache.org/user-guide/sql/scalar_functions.html#scalar-functions
Sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add contains function, and support in datafusion substrait consumer [datafusion]
Posted by "waynexia (via GitHub)" <gi...@apache.org>.
waynexia commented on code in PR #10879:
URL: https://github.com/apache/datafusion/pull/10879#discussion_r1635767599
##########
datafusion/functions/src/string/contains.rs:
##########
@@ -0,0 +1,143 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::utils::make_scalar_function;
+use arrow::array::{ArrayRef, OffsetSizeTrait};
+use arrow::datatypes::DataType;
+use arrow::datatypes::DataType::Boolean;
+use datafusion_common::cast::as_generic_string_array;
+use datafusion_common::DataFusionError;
+use datafusion_common::Result;
+use datafusion_common::{arrow_datafusion_err, exec_err};
+use datafusion_expr::ScalarUDFImpl;
+use datafusion_expr::TypeSignature::Exact;
+use datafusion_expr::{ColumnarValue, Signature, Volatility};
+use std::any::Any;
+use std::sync::Arc;
+#[derive(Debug)]
+pub struct ContainsFunc {
+ signature: Signature,
+}
+
+impl Default for ContainsFunc {
+ fn default() -> Self {
+ ContainsFunc::new()
+ }
+}
+
+impl ContainsFunc {
+ pub fn new() -> Self {
+ use DataType::*;
+ Self {
+ signature: Signature::one_of(
+ vec![Exact(vec![Utf8, Utf8]), Exact(vec![LargeUtf8, LargeUtf8])],
+ Volatility::Immutable,
+ ),
+ }
+ }
+}
+
+impl ScalarUDFImpl for ContainsFunc {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "contains"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, _: &[DataType]) -> Result<DataType> {
+ Ok(Boolean)
+ }
+
+ fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
+ match args[0].data_type() {
+ DataType::Utf8 => make_scalar_function(contains::<i32>, vec![])(args),
+ DataType::LargeUtf8 => make_scalar_function(contains::<i64>, vec![])(args),
+ other => {
+ exec_err!("unsupported data type {other:?} for function contains")
+ }
+ }
+ }
+}
+
+/// use regexp_is_match_utf8_scalar to do the calculation for contains
+pub fn contains<T: OffsetSizeTrait>(
+ args: &[ArrayRef],
+) -> Result<ArrayRef, DataFusionError> {
+ let mod_str = as_generic_string_array::<T>(&args[0])?;
+ let match_str = as_generic_string_array::<T>(&args[1])?;
+ let res = arrow::compute::kernels::comparison::regexp_is_match_utf8(
Review Comment:
Do we need to escape search string as it's used in regexp? Wondering what's the result of `contains("abcdefg", ".*")`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add contains function, and support in datafusion substrait consumer [datafusion]
Posted by "Weijun-H (via GitHub)" <gi...@apache.org>.
Weijun-H commented on PR #10879:
URL: https://github.com/apache/datafusion/pull/10879#issuecomment-2167185570
Please add an entry here
https://github.com/apache/datafusion/blob/1a2a1bf3f96b8ca96d1496d99a0c072d65e1940d/docs/source/user-guide/sql/scalar_functions.md#L645-L646
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add contains function, and support in datafusion substrait consumer [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10879:
URL: https://github.com/apache/datafusion/pull/10879#issuecomment-2168723309
The CI failure seems unrelated to the code in this PR: https://github.com/apache/datafusion/actions/runs/9510703358?pr=10879
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add contains function, and support in datafusion substrait consumer [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #10879:
URL: https://github.com/apache/datafusion/pull/10879
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add contains function, and support in datafusion substrait consumer [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10879:
URL: https://github.com/apache/datafusion/pull/10879#discussion_r1638343130
##########
datafusion/functions/src/string/contains.rs:
##########
@@ -0,0 +1,143 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::utils::make_scalar_function;
+use arrow::array::{ArrayRef, OffsetSizeTrait};
+use arrow::datatypes::DataType;
+use arrow::datatypes::DataType::Boolean;
+use datafusion_common::cast::as_generic_string_array;
+use datafusion_common::DataFusionError;
+use datafusion_common::Result;
+use datafusion_common::{arrow_datafusion_err, exec_err};
+use datafusion_expr::ScalarUDFImpl;
+use datafusion_expr::TypeSignature::Exact;
+use datafusion_expr::{ColumnarValue, Signature, Volatility};
+use std::any::Any;
+use std::sync::Arc;
+#[derive(Debug)]
+pub struct ContainsFunc {
+ signature: Signature,
+}
+
+impl Default for ContainsFunc {
+ fn default() -> Self {
+ ContainsFunc::new()
+ }
+}
+
+impl ContainsFunc {
+ pub fn new() -> Self {
+ use DataType::*;
+ Self {
+ signature: Signature::one_of(
+ vec![Exact(vec![Utf8, Utf8]), Exact(vec![LargeUtf8, LargeUtf8])],
+ Volatility::Immutable,
+ ),
+ }
+ }
+}
+
+impl ScalarUDFImpl for ContainsFunc {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "contains"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, _: &[DataType]) -> Result<DataType> {
+ Ok(Boolean)
+ }
+
+ fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
+ match args[0].data_type() {
+ DataType::Utf8 => make_scalar_function(contains::<i32>, vec![])(args),
+ DataType::LargeUtf8 => make_scalar_function(contains::<i64>, vec![])(args),
+ other => {
+ exec_err!("unsupported data type {other:?} for function contains")
+ }
+ }
+ }
+}
+
+/// use regexp_is_match_utf8_scalar to do the calculation for contains
+pub fn contains<T: OffsetSizeTrait>(
+ args: &[ArrayRef],
+) -> Result<ArrayRef, DataFusionError> {
+ let mod_str = as_generic_string_array::<T>(&args[0])?;
+ let match_str = as_generic_string_array::<T>(&args[1])?;
+ let res = arrow::compute::kernels::comparison::regexp_is_match_utf8(
Review Comment:
@Lordworms can you possibly respond to @waynexia 's comment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org