You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "davidlghellin (via GitHub)" <gi...@apache.org> on 2026/06/10 18:55:17 UTC
[PR] Feat/spark levenshtein [datafusion]
davidlghellin opened a new pull request, #22890:
URL: https://github.com/apache/datafusion/pull/22890
## Which issue does this PR close?
- Part of #15914
## Rationale for this change
Spark's `levenshtein(str1, str2, threshold)` supports an optional third argument: when the computed distance exceeds the threshold, it returns `-1` instead of the actual distance. DataFusion core's `levenshtein` only supports the 2-argument form. This PR adds the Spark-compatible version to `datafusion-spark`.
## What changes are included in this PR?
- Add `SparkLevenshtein` UDF in `datafusion/spark/src/function/string/levenshtein.rs`:
- 2-arg form: `levenshtein(str1, str2)` — standard Levenshtein distance
- 3-arg form: `levenshtein(str1, str2, threshold)` — returns `-1` when distance > threshold
- Supports Utf8, Utf8View, and LargeUtf8 (with automatic type coercion)
- NULL threshold is treated as `0` (Spark behavior)
- NULL string arguments return NULL
- Register the function in `string/mod.rs`
- Comprehensive SLT tests covering basic usage, threshold, NULL handling, Unicode, column expressions, per-row threshold, and mixed type coercion
## Are these changes tested?
Yes:
- 22 Rust unit tests in `levenshtein.rs`
- 20 SQL logic tests in `spark/string/levenshtein.slt`
## Are there any user-facing changes?
No breaking changes. Adds the `levenshtein` function to the Spark function library with optional threshold support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(spark): add levenshtein with optional threshold support [datafusion]
Posted by "davidlghellin (via GitHub)" <gi...@apache.org>.
davidlghellin commented on code in PR #22890:
URL: https://github.com/apache/datafusion/pull/22890#discussion_r3391123030
##########
datafusion/spark/src/function/string/levenshtein.rs:
##########
@@ -0,0 +1,594 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::any::Any;
+use std::sync::Arc;
+
+use arrow::array::{Array, ArrayRef, Int32Array, Int64Array, OffsetSizeTrait};
+use arrow::datatypes::DataType;
+use datafusion_common::cast::{
+ as_generic_string_array, as_int32_array, as_string_view_array,
+};
+use datafusion_common::types::{NativeType, logical_int32, logical_string};
+use datafusion_common::utils::datafusion_strsim;
+use datafusion_common::{Result, ScalarValue, exec_err};
+use datafusion_expr::type_coercion::binary::{
+ binary_to_string_coercion, string_coercion,
+};
+use datafusion_expr::{
+ Coercion, ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl, Signature, TypeSignature,
+ TypeSignatureClass, Volatility,
+};
+use datafusion_functions::utils::make_scalar_function;
+
+/// Spark-compatible `levenshtein` function.
+///
+/// Differs from DataFusion core's `levenshtein` in that it supports an optional
+/// third argument `threshold`. When the computed Levenshtein distance exceeds
+/// the threshold, the function returns -1 instead of the actual distance.
+///
+/// ```sql
+/// levenshtein('kitten', 'sitting') -- returns 3
+/// levenshtein('kitten', 'sitting', 2) -- returns -1 (distance 3 > threshold 2)
+/// levenshtein('kitten', 'sitting', 4) -- returns 3 (distance 3 <= threshold 4)
+/// ```
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkLevenshtein {
+ signature: Signature,
+}
+
+impl Default for SparkLevenshtein {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl SparkLevenshtein {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::one_of(
+ vec![
+ TypeSignature::Coercible(vec![
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ ]),
+ TypeSignature::Coercible(vec![
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ Coercion::new_implicit(
+ TypeSignatureClass::Native(logical_int32()),
+ vec![TypeSignatureClass::Integer],
+ NativeType::Int32,
+ ),
+ ]),
+ ],
+ Volatility::Immutable,
+ ),
+ }
+ }
+}
+
+impl ScalarUDFImpl for SparkLevenshtein {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "levenshtein"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
+ if arg_types.len() != 2 && arg_types.len() != 3 {
+ return exec_err!(
+ "levenshtein expects 2 or 3 arguments, got {}",
+ arg_types.len()
+ );
+ }
+ if let Some(coercion_data_type) = string_coercion(&arg_types[0], &arg_types[1])
+ .or_else(|| binary_to_string_coercion(&arg_types[0], &arg_types[1]))
+ {
+ match coercion_data_type {
+ DataType::LargeUtf8 => Ok(DataType::Int64),
+ DataType::Utf8 | DataType::Utf8View => Ok(DataType::Int32),
+ other => exec_err!(
+ "levenshtein requires Utf8, LargeUtf8 or Utf8View, got {other}"
+ ),
+ }
+ } else {
+ exec_err!(
+ "Unsupported data types for levenshtein. Expected Utf8, LargeUtf8 or Utf8View"
+ )
+ }
+ }
+
+ fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
+ let ScalarFunctionArgs { args, .. } = args;
+
+ // Determine the coerced string type (handles mixed Utf8 + LargeUtf8)
+ let coerced_type = string_coercion(&args[0].data_type(), &args[1].data_type())
+ .or_else(|| {
+ binary_to_string_coercion(&args[0].data_type(), &args[1].data_type())
+ })
+ .unwrap_or(DataType::Utf8);
Review Comment:
Done — replaced `.unwrap_or(DataType::Utf8)` with `.ok_or_else(...)?` that errors with the offending types in the message. Now consistent with `return_type`'s behaviour.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(spark): add levenshtein with optional threshold support [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #22890:
URL: https://github.com/apache/datafusion/pull/22890#issuecomment-5157324153
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/22890?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `76.48402%` with `103 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 80.85%. Comparing base ([`60d9b67`](https://app.codecov.io/gh/apache/datafusion/commit/60d9b67ec9428cac3e5b7ec7f25774f797063b65?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`447f700`](https://app.codecov.io/gh/apache/datafusion/commit/447f700582aac8f7af75f0cdf68628bad9fc2657?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 2 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/22890?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...atafusion/spark/src/function/string/levenshtein.rs](https://app.codecov.io/gh/apache/datafusion/pull/22890?src=pr&el=tree&filepath=datafusion%2Fspark%2Fsrc%2Ffunction%2Fstring%2Flevenshtein.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9zcGFyay9zcmMvZnVuY3Rpb24vc3RyaW5nL2xldmVuc2h0ZWluLnJz) | 76.95% | [26 Missing and 74 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/22890?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/spark/src/function/string/mod.rs](https://app.codecov.io/gh/apache/datafusion/pull/22890?src=pr&el=tree&filepath=datafusion%2Fspark%2Fsrc%2Ffunction%2Fstring%2Fmod.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9zcGFyay9zcmMvZnVuY3Rpb24vc3RyaW5nL21vZC5ycw==) | 25.00% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/22890?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #22890 +/- ##
==========================================
- Coverage 80.86% 80.85% -0.01%
==========================================
Files 1101 1102 +1
Lines 375446 375884 +438
Branches 375446 375884 +438
==========================================
+ Hits 303588 303917 +329
- Misses 53766 53796 +30
- Partials 18092 18171 +79
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/22890?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(spark): add levenshtein with optional threshold support [datafusion]
Posted by "davidlghellin (via GitHub)" <gi...@apache.org>.
davidlghellin commented on code in PR #22890:
URL: https://github.com/apache/datafusion/pull/22890#discussion_r3391138721
##########
datafusion/spark/src/function/string/levenshtein.rs:
##########
@@ -0,0 +1,594 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::any::Any;
+use std::sync::Arc;
+
+use arrow::array::{Array, ArrayRef, Int32Array, Int64Array, OffsetSizeTrait};
+use arrow::datatypes::DataType;
+use datafusion_common::cast::{
+ as_generic_string_array, as_int32_array, as_string_view_array,
+};
+use datafusion_common::types::{NativeType, logical_int32, logical_string};
+use datafusion_common::utils::datafusion_strsim;
+use datafusion_common::{Result, ScalarValue, exec_err};
+use datafusion_expr::type_coercion::binary::{
+ binary_to_string_coercion, string_coercion,
+};
+use datafusion_expr::{
+ Coercion, ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl, Signature, TypeSignature,
+ TypeSignatureClass, Volatility,
+};
+use datafusion_functions::utils::make_scalar_function;
+
+/// Spark-compatible `levenshtein` function.
+///
+/// Differs from DataFusion core's `levenshtein` in that it supports an optional
+/// third argument `threshold`. When the computed Levenshtein distance exceeds
+/// the threshold, the function returns -1 instead of the actual distance.
+///
+/// ```sql
+/// levenshtein('kitten', 'sitting') -- returns 3
+/// levenshtein('kitten', 'sitting', 2) -- returns -1 (distance 3 > threshold 2)
+/// levenshtein('kitten', 'sitting', 4) -- returns 3 (distance 3 <= threshold 4)
+/// ```
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkLevenshtein {
+ signature: Signature,
+}
+
+impl Default for SparkLevenshtein {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl SparkLevenshtein {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::one_of(
+ vec![
+ TypeSignature::Coercible(vec![
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ ]),
+ TypeSignature::Coercible(vec![
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ Coercion::new_implicit(
+ TypeSignatureClass::Native(logical_int32()),
+ vec![TypeSignatureClass::Integer],
+ NativeType::Int32,
+ ),
+ ]),
+ ],
+ Volatility::Immutable,
+ ),
+ }
+ }
+}
+
+impl ScalarUDFImpl for SparkLevenshtein {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "levenshtein"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
+ if arg_types.len() != 2 && arg_types.len() != 3 {
+ return exec_err!(
+ "levenshtein expects 2 or 3 arguments, got {}",
+ arg_types.len()
+ );
+ }
+ if let Some(coercion_data_type) = string_coercion(&arg_types[0], &arg_types[1])
+ .or_else(|| binary_to_string_coercion(&arg_types[0], &arg_types[1]))
+ {
+ match coercion_data_type {
+ DataType::LargeUtf8 => Ok(DataType::Int64),
+ DataType::Utf8 | DataType::Utf8View => Ok(DataType::Int32),
+ other => exec_err!(
+ "levenshtein requires Utf8, LargeUtf8 or Utf8View, got {other}"
+ ),
+ }
+ } else {
+ exec_err!(
+ "Unsupported data types for levenshtein. Expected Utf8, LargeUtf8 or Utf8View"
+ )
+ }
+ }
+
+ fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
+ let ScalarFunctionArgs { args, .. } = args;
+
+ // Determine the coerced string type (handles mixed Utf8 + LargeUtf8)
+ let coerced_type = string_coercion(&args[0].data_type(), &args[1].data_type())
+ .or_else(|| {
+ binary_to_string_coercion(&args[0].data_type(), &args[1].data_type())
+ })
+ .unwrap_or(DataType::Utf8);
+
+ // Spark returns NULL when any scalar argument is NULL.
+ let null_int = |dt: &DataType| match dt {
+ DataType::LargeUtf8 => ColumnarValue::Scalar(ScalarValue::Int64(None)),
+ _ => ColumnarValue::Scalar(ScalarValue::Int32(None)),
+ };
+ for arg in &args {
+ if matches!(arg, ColumnarValue::Scalar(s) if s.is_null()) {
+ return Ok(null_int(&coerced_type));
+ }
+ }
+
+ match coerced_type {
+ DataType::Utf8View | DataType::Utf8 => {
+ make_scalar_function(spark_levenshtein::<i32>, vec![])(&args)
+ }
+ DataType::LargeUtf8 => {
+ make_scalar_function(spark_levenshtein::<i64>, vec![])(&args)
+ }
+ other => {
+ exec_err!("Unsupported data type {other:?} for function levenshtein")
+ }
+ }
+ }
+}
+
+/// Spark-compatible Levenshtein distance with optional threshold.
+fn spark_levenshtein<T: OffsetSizeTrait>(args: &[ArrayRef]) -> Result<ArrayRef> {
+ if args.len() < 2 || args.len() > 3 {
+ return exec_err!("levenshtein expects 2 or 3 arguments, got {}", args.len());
+ }
+
+ let str1 = &args[0];
+ let str2 = &args[1];
+ let threshold = if args.len() == 3 {
+ Some(as_int32_array(&args[2])?)
+ } else {
+ None
+ };
+
+ if let Some(coercion_data_type) = string_coercion(str1.data_type(), str2.data_type())
+ .or_else(|| binary_to_string_coercion(str1.data_type(), str2.data_type()))
+ {
+ let str1 = if str1.data_type() == &coercion_data_type {
+ Arc::clone(str1)
+ } else {
+ arrow::compute::kernels::cast::cast(str1, &coercion_data_type)?
+ };
+ let str2 = if str2.data_type() == &coercion_data_type {
+ Arc::clone(str2)
+ } else {
+ arrow::compute::kernels::cast::cast(str2, &coercion_data_type)?
+ };
+
+ match coercion_data_type {
+ DataType::Utf8View => {
+ let str1_array = as_string_view_array(&str1)?;
+ let str2_array = as_string_view_array(&str2)?;
+ let mut cache = Vec::new();
+
+ let result = str1_array
+ .iter()
+ .zip(str2_array.iter())
+ .enumerate()
+ .map(|(i, (string1, string2))| match (string1, string2) {
+ (Some(string1), Some(string2)) => {
+ let dist = datafusion_strsim::levenshtein_with_buffer(
+ string1, string2, &mut cache,
+ ) as i32;
+ match &threshold {
+ Some(t) => {
+ let thresh =
+ if t.is_null(i) { 0 } else { t.value(i) };
+ if dist > thresh {
+ Some(-1i32)
+ } else {
+ Some(dist)
+ }
+ }
+ None => Some(dist),
+ }
+ }
Review Comment:
Extracted into `apply_threshold_i32` / `apply_threshold_i64` helpers — each branch now just calls `apply_threshold_*(dist, threshold.as_ref().map(|t| (*t, i)))`. Two helpers rather than one generic because the bounds (`D: From<I> + From<i8>`) added more noise than they removed for a tiny body.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(spark): add levenshtein with optional threshold support [datafusion]
Posted by "davidlghellin (via GitHub)" <gi...@apache.org>.
davidlghellin commented on code in PR #22890:
URL: https://github.com/apache/datafusion/pull/22890#discussion_r3391135044
##########
datafusion/spark/src/function/string/mod.rs:
##########
@@ -92,6 +94,11 @@ pub mod expr_fn {
"Returns the character length of string data or number of bytes of binary data. The length of string data includes the trailing spaces. The length of binary data includes binary zeros.",
arg1
));
+ export_functions!((
+ levenshtein,
+ "Returns the Levenshtein distance between two strings. Optionally accepts a threshold; returns -1 if the distance exceeds it.",
+ str1 str2 threshold
+ ));
Review Comment:
Confirmed — the macro generated a fixed 3-arg builder. Replaced it with a manual `pub fn levenshtein(args: Vec<Expr>) -> Expr` wrapper (same pattern as `concat_ws`/`trim` in core), so both 2-arg and 3-arg forms are reachable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(spark): add levenshtein with optional threshold support [datafusion]
Posted by "davidlghellin (via GitHub)" <gi...@apache.org>.
davidlghellin commented on code in PR #22890:
URL: https://github.com/apache/datafusion/pull/22890#discussion_r3391145642
##########
datafusion/spark/src/function/string/levenshtein.rs:
##########
@@ -0,0 +1,594 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::any::Any;
+use std::sync::Arc;
+
+use arrow::array::{Array, ArrayRef, Int32Array, Int64Array, OffsetSizeTrait};
+use arrow::datatypes::DataType;
+use datafusion_common::cast::{
+ as_generic_string_array, as_int32_array, as_string_view_array,
+};
+use datafusion_common::types::{NativeType, logical_int32, logical_string};
+use datafusion_common::utils::datafusion_strsim;
+use datafusion_common::{Result, ScalarValue, exec_err};
+use datafusion_expr::type_coercion::binary::{
+ binary_to_string_coercion, string_coercion,
+};
+use datafusion_expr::{
+ Coercion, ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl, Signature, TypeSignature,
+ TypeSignatureClass, Volatility,
+};
+use datafusion_functions::utils::make_scalar_function;
+
+/// Spark-compatible `levenshtein` function.
+///
+/// Differs from DataFusion core's `levenshtein` in that it supports an optional
+/// third argument `threshold`. When the computed Levenshtein distance exceeds
+/// the threshold, the function returns -1 instead of the actual distance.
+///
+/// ```sql
+/// levenshtein('kitten', 'sitting') -- returns 3
+/// levenshtein('kitten', 'sitting', 2) -- returns -1 (distance 3 > threshold 2)
+/// levenshtein('kitten', 'sitting', 4) -- returns 3 (distance 3 <= threshold 4)
+/// ```
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkLevenshtein {
+ signature: Signature,
+}
+
+impl Default for SparkLevenshtein {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl SparkLevenshtein {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::one_of(
+ vec![
+ TypeSignature::Coercible(vec![
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ ]),
+ TypeSignature::Coercible(vec![
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ Coercion::new_implicit(
+ TypeSignatureClass::Native(logical_int32()),
+ vec![TypeSignatureClass::Integer],
+ NativeType::Int32,
+ ),
+ ]),
+ ],
+ Volatility::Immutable,
+ ),
+ }
+ }
+}
+
+impl ScalarUDFImpl for SparkLevenshtein {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "levenshtein"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
+ if arg_types.len() != 2 && arg_types.len() != 3 {
+ return exec_err!(
+ "levenshtein expects 2 or 3 arguments, got {}",
+ arg_types.len()
+ );
+ }
+ if let Some(coercion_data_type) = string_coercion(&arg_types[0], &arg_types[1])
+ .or_else(|| binary_to_string_coercion(&arg_types[0], &arg_types[1]))
+ {
+ match coercion_data_type {
+ DataType::LargeUtf8 => Ok(DataType::Int64),
+ DataType::Utf8 | DataType::Utf8View => Ok(DataType::Int32),
+ other => exec_err!(
+ "levenshtein requires Utf8, LargeUtf8 or Utf8View, got {other}"
+ ),
+ }
+ } else {
+ exec_err!(
+ "Unsupported data types for levenshtein. Expected Utf8, LargeUtf8 or Utf8View"
+ )
+ }
+ }
+
+ fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
+ let ScalarFunctionArgs { args, .. } = args;
+
+ // Determine the coerced string type (handles mixed Utf8 + LargeUtf8)
+ let coerced_type = string_coercion(&args[0].data_type(), &args[1].data_type())
+ .or_else(|| {
+ binary_to_string_coercion(&args[0].data_type(), &args[1].data_type())
+ })
+ .unwrap_or(DataType::Utf8);
+
+ // Spark returns NULL when any scalar argument is NULL.
+ let null_int = |dt: &DataType| match dt {
+ DataType::LargeUtf8 => ColumnarValue::Scalar(ScalarValue::Int64(None)),
+ _ => ColumnarValue::Scalar(ScalarValue::Int32(None)),
+ };
+ for arg in &args {
+ if matches!(arg, ColumnarValue::Scalar(s) if s.is_null()) {
+ return Ok(null_int(&coerced_type));
+ }
+ }
Review Comment:
Verified against Spark — the asymmetry is intentional. Documented it in two places:
- A block comment at the scalar-NULL check explaining all 3 cases (NULL string propagates, NULL threshold scalar propagates, NULL threshold per row → treated as 0)
- A doc-comment on the `apply_threshold_*` helpers themselves
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(spark): add levenshtein with optional threshold support [datafusion]
Posted by "davidlghellin (via GitHub)" <gi...@apache.org>.
davidlghellin commented on code in PR #22890:
URL: https://github.com/apache/datafusion/pull/22890#discussion_r3391119951
##########
datafusion/spark/src/function/string/levenshtein.rs:
##########
@@ -0,0 +1,594 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::any::Any;
+use std::sync::Arc;
+
+use arrow::array::{Array, ArrayRef, Int32Array, Int64Array, OffsetSizeTrait};
+use arrow::datatypes::DataType;
+use datafusion_common::cast::{
+ as_generic_string_array, as_int32_array, as_string_view_array,
+};
+use datafusion_common::types::{NativeType, logical_int32, logical_string};
+use datafusion_common::utils::datafusion_strsim;
+use datafusion_common::{Result, ScalarValue, exec_err};
+use datafusion_expr::type_coercion::binary::{
+ binary_to_string_coercion, string_coercion,
+};
+use datafusion_expr::{
+ Coercion, ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl, Signature, TypeSignature,
+ TypeSignatureClass, Volatility,
+};
+use datafusion_functions::utils::make_scalar_function;
+
+/// Spark-compatible `levenshtein` function.
+///
+/// Differs from DataFusion core's `levenshtein` in that it supports an optional
+/// third argument `threshold`. When the computed Levenshtein distance exceeds
+/// the threshold, the function returns -1 instead of the actual distance.
+///
+/// ```sql
+/// levenshtein('kitten', 'sitting') -- returns 3
+/// levenshtein('kitten', 'sitting', 2) -- returns -1 (distance 3 > threshold 2)
+/// levenshtein('kitten', 'sitting', 4) -- returns 3 (distance 3 <= threshold 4)
+/// ```
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkLevenshtein {
+ signature: Signature,
+}
+
+impl Default for SparkLevenshtein {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl SparkLevenshtein {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::one_of(
+ vec![
+ TypeSignature::Coercible(vec![
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ ]),
+ TypeSignature::Coercible(vec![
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ Coercion::new_implicit(
+ TypeSignatureClass::Native(logical_int32()),
+ vec![TypeSignatureClass::Integer],
+ NativeType::Int32,
+ ),
+ ]),
+ ],
+ Volatility::Immutable,
+ ),
+ }
+ }
+}
+
+impl ScalarUDFImpl for SparkLevenshtein {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "levenshtein"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
+ if arg_types.len() != 2 && arg_types.len() != 3 {
+ return exec_err!(
+ "levenshtein expects 2 or 3 arguments, got {}",
+ arg_types.len()
+ );
+ }
+ if let Some(coercion_data_type) = string_coercion(&arg_types[0], &arg_types[1])
+ .or_else(|| binary_to_string_coercion(&arg_types[0], &arg_types[1]))
+ {
+ match coercion_data_type {
+ DataType::LargeUtf8 => Ok(DataType::Int64),
+ DataType::Utf8 | DataType::Utf8View => Ok(DataType::Int32),
+ other => exec_err!(
+ "levenshtein requires Utf8, LargeUtf8 or Utf8View, got {other}"
+ ),
+ }
+ } else {
+ exec_err!(
+ "Unsupported data types for levenshtein. Expected Utf8, LargeUtf8 or Utf8View"
+ )
+ }
+ }
+
+ fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
+ let ScalarFunctionArgs { args, .. } = args;
+
+ // Determine the coerced string type (handles mixed Utf8 + LargeUtf8)
+ let coerced_type = string_coercion(&args[0].data_type(), &args[1].data_type())
+ .or_else(|| {
+ binary_to_string_coercion(&args[0].data_type(), &args[1].data_type())
+ })
+ .unwrap_or(DataType::Utf8);
Review Comment:
Done — added an arity check at the start of `invoke_with_args` that errors with `exec_err!` before any `args[0]/[1]` indexing. Same shape as the one in `return_type`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(spark): add levenshtein with optional threshold support [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #22890: feat(spark): add levenshtein with optional threshold support
URL: https://github.com/apache/datafusion/pull/22890
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(spark): add levenshtein with optional threshold support [datafusion]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #22890:
URL: https://github.com/apache/datafusion/pull/22890#discussion_r3390797746
##########
datafusion/spark/src/function/string/levenshtein.rs:
##########
@@ -0,0 +1,594 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::any::Any;
+use std::sync::Arc;
+
+use arrow::array::{Array, ArrayRef, Int32Array, Int64Array, OffsetSizeTrait};
+use arrow::datatypes::DataType;
+use datafusion_common::cast::{
+ as_generic_string_array, as_int32_array, as_string_view_array,
+};
+use datafusion_common::types::{NativeType, logical_int32, logical_string};
+use datafusion_common::utils::datafusion_strsim;
+use datafusion_common::{Result, ScalarValue, exec_err};
+use datafusion_expr::type_coercion::binary::{
+ binary_to_string_coercion, string_coercion,
+};
+use datafusion_expr::{
+ Coercion, ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl, Signature, TypeSignature,
+ TypeSignatureClass, Volatility,
+};
+use datafusion_functions::utils::make_scalar_function;
+
+/// Spark-compatible `levenshtein` function.
+///
+/// Differs from DataFusion core's `levenshtein` in that it supports an optional
+/// third argument `threshold`. When the computed Levenshtein distance exceeds
+/// the threshold, the function returns -1 instead of the actual distance.
+///
+/// ```sql
+/// levenshtein('kitten', 'sitting') -- returns 3
+/// levenshtein('kitten', 'sitting', 2) -- returns -1 (distance 3 > threshold 2)
+/// levenshtein('kitten', 'sitting', 4) -- returns 3 (distance 3 <= threshold 4)
+/// ```
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkLevenshtein {
+ signature: Signature,
+}
+
+impl Default for SparkLevenshtein {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl SparkLevenshtein {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::one_of(
+ vec![
+ TypeSignature::Coercible(vec![
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ ]),
+ TypeSignature::Coercible(vec![
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ Coercion::new_implicit(
+ TypeSignatureClass::Native(logical_int32()),
+ vec![TypeSignatureClass::Integer],
+ NativeType::Int32,
+ ),
+ ]),
+ ],
+ Volatility::Immutable,
+ ),
+ }
+ }
+}
+
+impl ScalarUDFImpl for SparkLevenshtein {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "levenshtein"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
+ if arg_types.len() != 2 && arg_types.len() != 3 {
+ return exec_err!(
+ "levenshtein expects 2 or 3 arguments, got {}",
+ arg_types.len()
+ );
+ }
+ if let Some(coercion_data_type) = string_coercion(&arg_types[0], &arg_types[1])
+ .or_else(|| binary_to_string_coercion(&arg_types[0], &arg_types[1]))
+ {
+ match coercion_data_type {
+ DataType::LargeUtf8 => Ok(DataType::Int64),
+ DataType::Utf8 | DataType::Utf8View => Ok(DataType::Int32),
+ other => exec_err!(
+ "levenshtein requires Utf8, LargeUtf8 or Utf8View, got {other}"
+ ),
+ }
+ } else {
+ exec_err!(
+ "Unsupported data types for levenshtein. Expected Utf8, LargeUtf8 or Utf8View"
+ )
+ }
+ }
+
+ fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
+ let ScalarFunctionArgs { args, .. } = args;
+
+ // Determine the coerced string type (handles mixed Utf8 + LargeUtf8)
+ let coerced_type = string_coercion(&args[0].data_type(), &args[1].data_type())
+ .or_else(|| {
+ binary_to_string_coercion(&args[0].data_type(), &args[1].data_type())
+ })
+ .unwrap_or(DataType::Utf8);
Review Comment:
Defaulting to `DataType::Utf8` when coercion fails is inconsistent with `return_type`, which returns an error for unsupported types. This can mask type errors and produce confusing runtime behavior; return an `exec_err!` when both coercions return `None` (instead of `unwrap_or(DataType::Utf8)`), matching `return_type`’s logic.
##########
datafusion/spark/src/function/string/mod.rs:
##########
@@ -92,6 +94,11 @@ pub mod expr_fn {
"Returns the character length of string data or number of bytes of binary data. The length of string data includes the trailing spaces. The length of binary data includes binary zeros.",
arg1
));
+ export_functions!((
+ levenshtein,
+ "Returns the Levenshtein distance between two strings. Optionally accepts a threshold; returns -1 if the distance exceeds it.",
+ str1 str2 threshold
+ ));
Review Comment:
The `expr_fn` export lists `threshold` as a required parameter name, but the SQL function supports both 2-arg and 3-arg forms. If `export_functions!` generates fixed-arity Rust helpers, this likely exposes only a 3-arg builder (making the 2-arg form unavailable/misleading in the expression API). Consider exporting two arities (separate exports/wrappers) or using whichever macro pattern the codebase uses for optional arguments so both `levenshtein(str1, str2)` and `levenshtein(str1, str2, threshold)` are supported.
##########
datafusion/spark/src/function/string/levenshtein.rs:
##########
@@ -0,0 +1,594 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::any::Any;
+use std::sync::Arc;
+
+use arrow::array::{Array, ArrayRef, Int32Array, Int64Array, OffsetSizeTrait};
+use arrow::datatypes::DataType;
+use datafusion_common::cast::{
+ as_generic_string_array, as_int32_array, as_string_view_array,
+};
+use datafusion_common::types::{NativeType, logical_int32, logical_string};
+use datafusion_common::utils::datafusion_strsim;
+use datafusion_common::{Result, ScalarValue, exec_err};
+use datafusion_expr::type_coercion::binary::{
+ binary_to_string_coercion, string_coercion,
+};
+use datafusion_expr::{
+ Coercion, ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl, Signature, TypeSignature,
+ TypeSignatureClass, Volatility,
+};
+use datafusion_functions::utils::make_scalar_function;
+
+/// Spark-compatible `levenshtein` function.
+///
+/// Differs from DataFusion core's `levenshtein` in that it supports an optional
+/// third argument `threshold`. When the computed Levenshtein distance exceeds
+/// the threshold, the function returns -1 instead of the actual distance.
+///
+/// ```sql
+/// levenshtein('kitten', 'sitting') -- returns 3
+/// levenshtein('kitten', 'sitting', 2) -- returns -1 (distance 3 > threshold 2)
+/// levenshtein('kitten', 'sitting', 4) -- returns 3 (distance 3 <= threshold 4)
+/// ```
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkLevenshtein {
+ signature: Signature,
+}
+
+impl Default for SparkLevenshtein {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl SparkLevenshtein {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::one_of(
+ vec![
+ TypeSignature::Coercible(vec![
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ ]),
+ TypeSignature::Coercible(vec![
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ Coercion::new_implicit(
+ TypeSignatureClass::Native(logical_int32()),
+ vec![TypeSignatureClass::Integer],
+ NativeType::Int32,
+ ),
+ ]),
+ ],
+ Volatility::Immutable,
+ ),
+ }
+ }
+}
+
+impl ScalarUDFImpl for SparkLevenshtein {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "levenshtein"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
+ if arg_types.len() != 2 && arg_types.len() != 3 {
+ return exec_err!(
+ "levenshtein expects 2 or 3 arguments, got {}",
+ arg_types.len()
+ );
+ }
+ if let Some(coercion_data_type) = string_coercion(&arg_types[0], &arg_types[1])
+ .or_else(|| binary_to_string_coercion(&arg_types[0], &arg_types[1]))
+ {
+ match coercion_data_type {
+ DataType::LargeUtf8 => Ok(DataType::Int64),
+ DataType::Utf8 | DataType::Utf8View => Ok(DataType::Int32),
+ other => exec_err!(
+ "levenshtein requires Utf8, LargeUtf8 or Utf8View, got {other}"
+ ),
+ }
+ } else {
+ exec_err!(
+ "Unsupported data types for levenshtein. Expected Utf8, LargeUtf8 or Utf8View"
+ )
+ }
+ }
+
+ fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
+ let ScalarFunctionArgs { args, .. } = args;
+
+ // Determine the coerced string type (handles mixed Utf8 + LargeUtf8)
+ let coerced_type = string_coercion(&args[0].data_type(), &args[1].data_type())
+ .or_else(|| {
+ binary_to_string_coercion(&args[0].data_type(), &args[1].data_type())
+ })
+ .unwrap_or(DataType::Utf8);
Review Comment:
`invoke_with_args` indexes `args[0]` / `args[1]` without validating arity first; if the UDF is invoked with <2 args this will panic. Add the same `2 or 3 arguments` check used in `return_type` (and `spark_levenshtein`) at the start of `invoke_with_args` and return an `exec_err!` on mismatch.
##########
datafusion/sqllogictest/test_files/spark/string/levenshtein.slt:
##########
@@ -21,12 +21,224 @@
# For more information, please see:
# https://github.com/apache/datafusion/issues/15914
-## Original Query: SELECT levenshtein('kitten', 'sitting');
-## PySpark 3.5.5 Result: {'levenshtein(kitten, sitting)': 3, 'typeof(levenshtein(kitten, sitting))': 'int', 'typeof(kitten)': 'string', 'typeof(sitting)': 'string'}
-#query
-#SELECT levenshtein('kitten'::string, 'sitting'::string);
-
-## Original Query: SELECT levenshtein('kitten', 'sitting', 2);
-## PySpark 3.5.5 Result: {'levenshtein(kitten, sitting, 2)': -1, 'typeof(levenshtein(kitten, sitting, 2))': 'int', 'typeof(kitten)': 'string', 'typeof(sitting)': 'string', 'typeof(2)': 'int'}
-#query
-#SELECT levenshtein('kitten'::string, 'sitting'::string, 2::int);
+## ── Basic usage ─────────────────────────────────────────────
+
+## Basic distance
+query I
+SELECT levenshtein('kitten', 'sitting');
+----
+3
+
+## Identical strings
+query I
+SELECT levenshtein('hello', 'hello');
+----
+0
+
+## Empty string vs non-empty
+query I
+SELECT levenshtein('', 'abc');
+----
+3
+
+## Both empty strings
+query I
+SELECT levenshtein('', '');
+----
+0
+
+## Single character difference
+query I
+SELECT levenshtein('abc', 'adc');
+----
+1
+
+## ── Threshold (3-argument form) ─────────────────────────────
+
+## Distance within threshold
+query I
+SELECT levenshtein('kitten', 'sitting', 4);
+----
+3
+
+## Distance exceeds threshold → returns -1
+query I
+SELECT levenshtein('kitten', 'sitting', 2);
+----
+-1
+
+## Distance equals threshold (boundary) → returns distance
+query I
+SELECT levenshtein('kitten', 'sitting', 3);
+----
+3
+
+## Threshold zero with different strings
+query I
+SELECT levenshtein('abc', 'def', 0);
+----
+-1
+
+## Threshold zero with identical strings
+query I
+SELECT levenshtein('abc', 'abc', 0);
+----
+0
+
+## ── Null handling ───────────────────────────────────────────
+
+## First argument null
+query I
+SELECT levenshtein(CAST(NULL AS STRING), 'hello');
+----
+NULL
+
+## Second argument null
+query I
+SELECT levenshtein('hello', CAST(NULL AS STRING));
+----
+NULL
+
+## Both arguments null
+query I
+SELECT levenshtein(CAST(NULL AS STRING), CAST(NULL AS STRING));
+----
+NULL
+
+## Null threshold
+query I
+SELECT levenshtein('kitten', 'sitting', CAST(NULL AS INT));
+----
+NULL
+
+## ── Unicode and special characters ──────────────────────────
+
+## Unicode strings
+query I
+SELECT levenshtein('café', 'cafe');
+----
+1
+
+## Strings with spaces
+query I
+SELECT levenshtein('hello world', 'hello world!');
+----
+1
+
+## ── Column expressions ──────────────────────────────────────
+
+## Levenshtein on columns from inline table
+query I
+SELECT levenshtein(s1, s2) AS result FROM VALUES ('abc', 'abc'), ('abc', 'def'), ('kitten', 'sitting') AS t(s1, s2);
+----
+0
+3
+3
+
+## Threshold on columns from inline table
+query I
+SELECT levenshtein(s1, s2, 2) AS result FROM VALUES ('abc', 'abc'), ('abc', 'def'), ('kitten', 'sitting') AS t(s1, s2);
+----
+0
+-1
+-1
+
+## ── Per-row threshold ───────────────────────────────────────
+
+## Different threshold per row
+query I
+SELECT levenshtein(s1, s2, t) AS result FROM VALUES ('abc', 'def', 2), ('abc', 'def', 5), ('abc', 'def', 3) AS t(s1, s2, t);
Review Comment:
This query uses `t` as both the table alias and a column name (`AS t(s1, s2, t)`), which is unnecessarily confusing and can make failures harder to interpret. Rename either the alias (e.g., `v`) or the threshold column (e.g., `threshold`) to avoid the collision.
##########
datafusion/spark/src/function/string/levenshtein.rs:
##########
@@ -0,0 +1,594 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::any::Any;
+use std::sync::Arc;
+
+use arrow::array::{Array, ArrayRef, Int32Array, Int64Array, OffsetSizeTrait};
+use arrow::datatypes::DataType;
+use datafusion_common::cast::{
+ as_generic_string_array, as_int32_array, as_string_view_array,
+};
+use datafusion_common::types::{NativeType, logical_int32, logical_string};
+use datafusion_common::utils::datafusion_strsim;
+use datafusion_common::{Result, ScalarValue, exec_err};
+use datafusion_expr::type_coercion::binary::{
+ binary_to_string_coercion, string_coercion,
+};
+use datafusion_expr::{
+ Coercion, ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl, Signature, TypeSignature,
+ TypeSignatureClass, Volatility,
+};
+use datafusion_functions::utils::make_scalar_function;
+
+/// Spark-compatible `levenshtein` function.
+///
+/// Differs from DataFusion core's `levenshtein` in that it supports an optional
+/// third argument `threshold`. When the computed Levenshtein distance exceeds
+/// the threshold, the function returns -1 instead of the actual distance.
+///
+/// ```sql
+/// levenshtein('kitten', 'sitting') -- returns 3
+/// levenshtein('kitten', 'sitting', 2) -- returns -1 (distance 3 > threshold 2)
+/// levenshtein('kitten', 'sitting', 4) -- returns 3 (distance 3 <= threshold 4)
+/// ```
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkLevenshtein {
+ signature: Signature,
+}
+
+impl Default for SparkLevenshtein {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl SparkLevenshtein {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::one_of(
+ vec![
+ TypeSignature::Coercible(vec![
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ ]),
+ TypeSignature::Coercible(vec![
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ Coercion::new_implicit(
+ TypeSignatureClass::Native(logical_int32()),
+ vec![TypeSignatureClass::Integer],
+ NativeType::Int32,
+ ),
+ ]),
+ ],
+ Volatility::Immutable,
+ ),
+ }
+ }
+}
+
+impl ScalarUDFImpl for SparkLevenshtein {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "levenshtein"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
+ if arg_types.len() != 2 && arg_types.len() != 3 {
+ return exec_err!(
+ "levenshtein expects 2 or 3 arguments, got {}",
+ arg_types.len()
+ );
+ }
+ if let Some(coercion_data_type) = string_coercion(&arg_types[0], &arg_types[1])
+ .or_else(|| binary_to_string_coercion(&arg_types[0], &arg_types[1]))
+ {
+ match coercion_data_type {
+ DataType::LargeUtf8 => Ok(DataType::Int64),
+ DataType::Utf8 | DataType::Utf8View => Ok(DataType::Int32),
+ other => exec_err!(
+ "levenshtein requires Utf8, LargeUtf8 or Utf8View, got {other}"
+ ),
+ }
+ } else {
+ exec_err!(
+ "Unsupported data types for levenshtein. Expected Utf8, LargeUtf8 or Utf8View"
+ )
+ }
+ }
+
+ fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
+ let ScalarFunctionArgs { args, .. } = args;
+
+ // Determine the coerced string type (handles mixed Utf8 + LargeUtf8)
+ let coerced_type = string_coercion(&args[0].data_type(), &args[1].data_type())
+ .or_else(|| {
+ binary_to_string_coercion(&args[0].data_type(), &args[1].data_type())
+ })
+ .unwrap_or(DataType::Utf8);
+
+ // Spark returns NULL when any scalar argument is NULL.
+ let null_int = |dt: &DataType| match dt {
+ DataType::LargeUtf8 => ColumnarValue::Scalar(ScalarValue::Int64(None)),
+ _ => ColumnarValue::Scalar(ScalarValue::Int32(None)),
+ };
+ for arg in &args {
+ if matches!(arg, ColumnarValue::Scalar(s) if s.is_null()) {
+ return Ok(null_int(&coerced_type));
+ }
+ }
+
+ match coerced_type {
+ DataType::Utf8View | DataType::Utf8 => {
+ make_scalar_function(spark_levenshtein::<i32>, vec![])(&args)
+ }
+ DataType::LargeUtf8 => {
+ make_scalar_function(spark_levenshtein::<i64>, vec![])(&args)
+ }
Review Comment:
The unit tests cover mixed `Utf8 + LargeUtf8` for the 2-arg form, but there are no direct unit tests exercising the `Utf8View` path and no unit test covering the `LargeUtf8` path with the 3-arg threshold form. Adding targeted tests for (1) `Utf8View` inputs and (2) `LargeUtf8` inputs with a threshold would help ensure the per-branch logic remains consistent.
##########
datafusion/spark/src/function/string/levenshtein.rs:
##########
@@ -0,0 +1,594 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::any::Any;
+use std::sync::Arc;
+
+use arrow::array::{Array, ArrayRef, Int32Array, Int64Array, OffsetSizeTrait};
+use arrow::datatypes::DataType;
+use datafusion_common::cast::{
+ as_generic_string_array, as_int32_array, as_string_view_array,
+};
+use datafusion_common::types::{NativeType, logical_int32, logical_string};
+use datafusion_common::utils::datafusion_strsim;
+use datafusion_common::{Result, ScalarValue, exec_err};
+use datafusion_expr::type_coercion::binary::{
+ binary_to_string_coercion, string_coercion,
+};
+use datafusion_expr::{
+ Coercion, ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl, Signature, TypeSignature,
+ TypeSignatureClass, Volatility,
+};
+use datafusion_functions::utils::make_scalar_function;
+
+/// Spark-compatible `levenshtein` function.
+///
+/// Differs from DataFusion core's `levenshtein` in that it supports an optional
+/// third argument `threshold`. When the computed Levenshtein distance exceeds
+/// the threshold, the function returns -1 instead of the actual distance.
+///
+/// ```sql
+/// levenshtein('kitten', 'sitting') -- returns 3
+/// levenshtein('kitten', 'sitting', 2) -- returns -1 (distance 3 > threshold 2)
+/// levenshtein('kitten', 'sitting', 4) -- returns 3 (distance 3 <= threshold 4)
+/// ```
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkLevenshtein {
+ signature: Signature,
+}
+
+impl Default for SparkLevenshtein {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl SparkLevenshtein {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::one_of(
+ vec![
+ TypeSignature::Coercible(vec![
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ ]),
+ TypeSignature::Coercible(vec![
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ Coercion::new_implicit(
+ TypeSignatureClass::Native(logical_int32()),
+ vec![TypeSignatureClass::Integer],
+ NativeType::Int32,
+ ),
+ ]),
+ ],
+ Volatility::Immutable,
+ ),
+ }
+ }
+}
+
+impl ScalarUDFImpl for SparkLevenshtein {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "levenshtein"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
+ if arg_types.len() != 2 && arg_types.len() != 3 {
+ return exec_err!(
+ "levenshtein expects 2 or 3 arguments, got {}",
+ arg_types.len()
+ );
+ }
+ if let Some(coercion_data_type) = string_coercion(&arg_types[0], &arg_types[1])
+ .or_else(|| binary_to_string_coercion(&arg_types[0], &arg_types[1]))
+ {
+ match coercion_data_type {
+ DataType::LargeUtf8 => Ok(DataType::Int64),
+ DataType::Utf8 | DataType::Utf8View => Ok(DataType::Int32),
+ other => exec_err!(
+ "levenshtein requires Utf8, LargeUtf8 or Utf8View, got {other}"
+ ),
+ }
+ } else {
+ exec_err!(
+ "Unsupported data types for levenshtein. Expected Utf8, LargeUtf8 or Utf8View"
+ )
+ }
+ }
+
+ fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
+ let ScalarFunctionArgs { args, .. } = args;
+
+ // Determine the coerced string type (handles mixed Utf8 + LargeUtf8)
+ let coerced_type = string_coercion(&args[0].data_type(), &args[1].data_type())
+ .or_else(|| {
+ binary_to_string_coercion(&args[0].data_type(), &args[1].data_type())
+ })
+ .unwrap_or(DataType::Utf8);
+
+ // Spark returns NULL when any scalar argument is NULL.
+ let null_int = |dt: &DataType| match dt {
+ DataType::LargeUtf8 => ColumnarValue::Scalar(ScalarValue::Int64(None)),
+ _ => ColumnarValue::Scalar(ScalarValue::Int32(None)),
+ };
+ for arg in &args {
+ if matches!(arg, ColumnarValue::Scalar(s) if s.is_null()) {
+ return Ok(null_int(&coerced_type));
+ }
+ }
+
+ match coerced_type {
+ DataType::Utf8View | DataType::Utf8 => {
+ make_scalar_function(spark_levenshtein::<i32>, vec![])(&args)
+ }
+ DataType::LargeUtf8 => {
+ make_scalar_function(spark_levenshtein::<i64>, vec![])(&args)
+ }
+ other => {
+ exec_err!("Unsupported data type {other:?} for function levenshtein")
+ }
+ }
+ }
+}
+
+/// Spark-compatible Levenshtein distance with optional threshold.
+fn spark_levenshtein<T: OffsetSizeTrait>(args: &[ArrayRef]) -> Result<ArrayRef> {
+ if args.len() < 2 || args.len() > 3 {
+ return exec_err!("levenshtein expects 2 or 3 arguments, got {}", args.len());
+ }
+
+ let str1 = &args[0];
+ let str2 = &args[1];
+ let threshold = if args.len() == 3 {
+ Some(as_int32_array(&args[2])?)
+ } else {
+ None
+ };
+
+ if let Some(coercion_data_type) = string_coercion(str1.data_type(), str2.data_type())
+ .or_else(|| binary_to_string_coercion(str1.data_type(), str2.data_type()))
+ {
+ let str1 = if str1.data_type() == &coercion_data_type {
+ Arc::clone(str1)
+ } else {
+ arrow::compute::kernels::cast::cast(str1, &coercion_data_type)?
+ };
+ let str2 = if str2.data_type() == &coercion_data_type {
+ Arc::clone(str2)
+ } else {
+ arrow::compute::kernels::cast::cast(str2, &coercion_data_type)?
+ };
+
+ match coercion_data_type {
+ DataType::Utf8View => {
+ let str1_array = as_string_view_array(&str1)?;
+ let str2_array = as_string_view_array(&str2)?;
+ let mut cache = Vec::new();
+
+ let result = str1_array
+ .iter()
+ .zip(str2_array.iter())
+ .enumerate()
+ .map(|(i, (string1, string2))| match (string1, string2) {
+ (Some(string1), Some(string2)) => {
+ let dist = datafusion_strsim::levenshtein_with_buffer(
+ string1, string2, &mut cache,
+ ) as i32;
+ match &threshold {
+ Some(t) => {
+ let thresh =
+ if t.is_null(i) { 0 } else { t.value(i) };
+ if dist > thresh {
+ Some(-1i32)
+ } else {
+ Some(dist)
+ }
+ }
+ None => Some(dist),
+ }
+ }
Review Comment:
The per-row threshold handling logic is duplicated across the `Utf8View`, `Utf8`, and `LargeUtf8` branches (with only minor type differences). This increases the chance of subtle behavior drift between branches; consider extracting a small helper that maps `(dist, threshold_value_or_none)` to `Option<dist_or_-1>` so each branch shares exactly the same decision logic.
##########
datafusion/spark/src/function/string/levenshtein.rs:
##########
@@ -0,0 +1,594 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::any::Any;
+use std::sync::Arc;
+
+use arrow::array::{Array, ArrayRef, Int32Array, Int64Array, OffsetSizeTrait};
+use arrow::datatypes::DataType;
+use datafusion_common::cast::{
+ as_generic_string_array, as_int32_array, as_string_view_array,
+};
+use datafusion_common::types::{NativeType, logical_int32, logical_string};
+use datafusion_common::utils::datafusion_strsim;
+use datafusion_common::{Result, ScalarValue, exec_err};
+use datafusion_expr::type_coercion::binary::{
+ binary_to_string_coercion, string_coercion,
+};
+use datafusion_expr::{
+ Coercion, ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl, Signature, TypeSignature,
+ TypeSignatureClass, Volatility,
+};
+use datafusion_functions::utils::make_scalar_function;
+
+/// Spark-compatible `levenshtein` function.
+///
+/// Differs from DataFusion core's `levenshtein` in that it supports an optional
+/// third argument `threshold`. When the computed Levenshtein distance exceeds
+/// the threshold, the function returns -1 instead of the actual distance.
+///
+/// ```sql
+/// levenshtein('kitten', 'sitting') -- returns 3
+/// levenshtein('kitten', 'sitting', 2) -- returns -1 (distance 3 > threshold 2)
+/// levenshtein('kitten', 'sitting', 4) -- returns 3 (distance 3 <= threshold 4)
+/// ```
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkLevenshtein {
+ signature: Signature,
+}
+
+impl Default for SparkLevenshtein {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl SparkLevenshtein {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::one_of(
+ vec![
+ TypeSignature::Coercible(vec![
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ ]),
+ TypeSignature::Coercible(vec![
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ Coercion::new_implicit(
+ TypeSignatureClass::Native(logical_int32()),
+ vec![TypeSignatureClass::Integer],
+ NativeType::Int32,
+ ),
+ ]),
+ ],
+ Volatility::Immutable,
+ ),
+ }
+ }
+}
+
+impl ScalarUDFImpl for SparkLevenshtein {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "levenshtein"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
+ if arg_types.len() != 2 && arg_types.len() != 3 {
+ return exec_err!(
+ "levenshtein expects 2 or 3 arguments, got {}",
+ arg_types.len()
+ );
+ }
+ if let Some(coercion_data_type) = string_coercion(&arg_types[0], &arg_types[1])
+ .or_else(|| binary_to_string_coercion(&arg_types[0], &arg_types[1]))
+ {
+ match coercion_data_type {
+ DataType::LargeUtf8 => Ok(DataType::Int64),
+ DataType::Utf8 | DataType::Utf8View => Ok(DataType::Int32),
+ other => exec_err!(
+ "levenshtein requires Utf8, LargeUtf8 or Utf8View, got {other}"
+ ),
+ }
+ } else {
+ exec_err!(
+ "Unsupported data types for levenshtein. Expected Utf8, LargeUtf8 or Utf8View"
+ )
+ }
+ }
+
+ fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
+ let ScalarFunctionArgs { args, .. } = args;
+
+ // Determine the coerced string type (handles mixed Utf8 + LargeUtf8)
+ let coerced_type = string_coercion(&args[0].data_type(), &args[1].data_type())
+ .or_else(|| {
+ binary_to_string_coercion(&args[0].data_type(), &args[1].data_type())
+ })
+ .unwrap_or(DataType::Utf8);
+
+ // Spark returns NULL when any scalar argument is NULL.
+ let null_int = |dt: &DataType| match dt {
+ DataType::LargeUtf8 => ColumnarValue::Scalar(ScalarValue::Int64(None)),
+ _ => ColumnarValue::Scalar(ScalarValue::Int32(None)),
+ };
+ for arg in &args {
+ if matches!(arg, ColumnarValue::Scalar(s) if s.is_null()) {
+ return Ok(null_int(&coerced_type));
+ }
+ }
Review Comment:
The code explicitly documents NULL propagation for *scalar* arguments, but in the array path NULL thresholds are treated as `0` (see `if t.is_null(i) { 0 } ...`). This mixed behavior is surprising; please add an in-code comment explaining why Spark’s behavior differs between scalar-NULL vs row-NULL threshold (and ideally link to a Spark issue/test/reference), or adjust the array behavior if Spark null-propagates per row.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(spark): add levenshtein with optional threshold support [datafusion]
Posted by "davidlghellin (via GitHub)" <gi...@apache.org>.
davidlghellin commented on code in PR #22890:
URL: https://github.com/apache/datafusion/pull/22890#discussion_r3391147956
##########
datafusion/spark/src/function/string/levenshtein.rs:
##########
@@ -0,0 +1,594 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::any::Any;
+use std::sync::Arc;
+
+use arrow::array::{Array, ArrayRef, Int32Array, Int64Array, OffsetSizeTrait};
+use arrow::datatypes::DataType;
+use datafusion_common::cast::{
+ as_generic_string_array, as_int32_array, as_string_view_array,
+};
+use datafusion_common::types::{NativeType, logical_int32, logical_string};
+use datafusion_common::utils::datafusion_strsim;
+use datafusion_common::{Result, ScalarValue, exec_err};
+use datafusion_expr::type_coercion::binary::{
+ binary_to_string_coercion, string_coercion,
+};
+use datafusion_expr::{
+ Coercion, ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl, Signature, TypeSignature,
+ TypeSignatureClass, Volatility,
+};
+use datafusion_functions::utils::make_scalar_function;
+
+/// Spark-compatible `levenshtein` function.
+///
+/// Differs from DataFusion core's `levenshtein` in that it supports an optional
+/// third argument `threshold`. When the computed Levenshtein distance exceeds
+/// the threshold, the function returns -1 instead of the actual distance.
+///
+/// ```sql
+/// levenshtein('kitten', 'sitting') -- returns 3
+/// levenshtein('kitten', 'sitting', 2) -- returns -1 (distance 3 > threshold 2)
+/// levenshtein('kitten', 'sitting', 4) -- returns 3 (distance 3 <= threshold 4)
+/// ```
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkLevenshtein {
+ signature: Signature,
+}
+
+impl Default for SparkLevenshtein {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl SparkLevenshtein {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::one_of(
+ vec![
+ TypeSignature::Coercible(vec![
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ ]),
+ TypeSignature::Coercible(vec![
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ Coercion::new_exact(TypeSignatureClass::Native(logical_string())),
+ Coercion::new_implicit(
+ TypeSignatureClass::Native(logical_int32()),
+ vec![TypeSignatureClass::Integer],
+ NativeType::Int32,
+ ),
+ ]),
+ ],
+ Volatility::Immutable,
+ ),
+ }
+ }
+}
+
+impl ScalarUDFImpl for SparkLevenshtein {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "levenshtein"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
+ if arg_types.len() != 2 && arg_types.len() != 3 {
+ return exec_err!(
+ "levenshtein expects 2 or 3 arguments, got {}",
+ arg_types.len()
+ );
+ }
+ if let Some(coercion_data_type) = string_coercion(&arg_types[0], &arg_types[1])
+ .or_else(|| binary_to_string_coercion(&arg_types[0], &arg_types[1]))
+ {
+ match coercion_data_type {
+ DataType::LargeUtf8 => Ok(DataType::Int64),
+ DataType::Utf8 | DataType::Utf8View => Ok(DataType::Int32),
+ other => exec_err!(
+ "levenshtein requires Utf8, LargeUtf8 or Utf8View, got {other}"
+ ),
+ }
+ } else {
+ exec_err!(
+ "Unsupported data types for levenshtein. Expected Utf8, LargeUtf8 or Utf8View"
+ )
+ }
+ }
+
+ fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
+ let ScalarFunctionArgs { args, .. } = args;
+
+ // Determine the coerced string type (handles mixed Utf8 + LargeUtf8)
+ let coerced_type = string_coercion(&args[0].data_type(), &args[1].data_type())
+ .or_else(|| {
+ binary_to_string_coercion(&args[0].data_type(), &args[1].data_type())
+ })
+ .unwrap_or(DataType::Utf8);
+
+ // Spark returns NULL when any scalar argument is NULL.
+ let null_int = |dt: &DataType| match dt {
+ DataType::LargeUtf8 => ColumnarValue::Scalar(ScalarValue::Int64(None)),
+ _ => ColumnarValue::Scalar(ScalarValue::Int32(None)),
+ };
+ for arg in &args {
+ if matches!(arg, ColumnarValue::Scalar(s) if s.is_null()) {
+ return Ok(null_int(&coerced_type));
+ }
+ }
+
+ match coerced_type {
+ DataType::Utf8View | DataType::Utf8 => {
+ make_scalar_function(spark_levenshtein::<i32>, vec![])(&args)
+ }
+ DataType::LargeUtf8 => {
+ make_scalar_function(spark_levenshtein::<i64>, vec![])(&args)
+ }
Review Comment:
Added 5 unit tests: 3 for the Utf8View path (2-arg, threshold-exceeds, threshold-within) and 2 for LargeUtf8 + threshold (exceeds, within), plus `test_largeutf8_with_null_threshold_row_treated_as_zero` which exercises both the LargeUtf8 path AND the NULL-threshold-per-row branch from comment #6. 31/31 passing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(spark): add levenshtein with optional threshold support [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22890:
URL: https://github.com/apache/datafusion/pull/22890#issuecomment-4673482216
<!-- semver-check-comment -->
Thank you for opening this pull request!
Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
<details>
<summary>Details</summary>
```
Cloning apache/main
Building datafusion-spark v54.0.0 (current)
error: running cargo-doc on crate 'datafusion-spark' failed with output:
-----
Compiling proc-macro2 v1.0.106
Compiling unicode-ident v1.0.24
Compiling quote v1.0.45
Compiling libc v0.2.186
Checking cfg-if v1.0.4
Compiling autocfg v1.5.1
Compiling libm v0.2.16
Checking memchr v2.8.1
Compiling num-traits v0.2.19
Compiling zerocopy v0.8.52
Compiling syn v2.0.117
Compiling version_check v0.9.5
Compiling getrandom v0.3.4
Checking bytes v1.11.1
Checking equivalent v1.0.2
Checking itoa v1.0.18
Checking allocator-api2 v0.2.21
Checking once_cell v1.21.4
Checking foldhash v0.2.0
Compiling serde_core v1.0.228
Checking hashbrown v0.17.1
Compiling zmij v1.0.21
Checking num-integer v0.1.46
Compiling serde_json v1.0.150
Checking num-bigint v0.4.6
Checking indexmap v2.14.0
Checking siphasher v1.0.3
Checking iana-time-zone v0.1.65
Checking stable_deref_trait v1.2.1
Checking phf_shared v0.12.1
Checking chrono v0.4.45
Compiling ahash v0.8.12
Compiling chrono-tz v0.10.4
Checking phf v0.12.1
Compiling jobserver v0.1.34
Compiling shlex v2.0.1
Compiling find-msvc-tools v0.1.9
Compiling synstructure v0.13.2
Checking arrow-schema v59.0.0
Compiling cc v1.2.63
Checking num-complex v0.4.6
Checking lexical-util v1.0.7
Checking pin-project-lite v0.2.17
Checking smallvec v1.15.1
Checking litemap v0.8.2
Checking writeable v0.6.3
Compiling pkg-config v0.3.33
Compiling zerocopy-derive v0.8.52
Compiling zerofrom-derive v0.1.7
Compiling yoke-derive v0.8.2
Checking zerofrom v0.1.8
Compiling zerovec-derive v0.11.3
Checking yoke v0.8.3
Compiling displaydoc v0.2.6
Checking futures-core v0.3.32
Checking futures-sink v0.3.32
Checking zerovec v0.11.6
Checking zerotrie v0.2.4
Compiling zstd-sys v2.0.16+zstd.1.5.7
Checking utf8_iter v1.0.4
Checking tinystr v0.8.3
Checking potential_utf v0.1.5
Compiling icu_properties_data v2.2.0
Checking icu_locale_core v2.2.0
Compiling icu_normalizer_data v2.2.0
Checking icu_collections v2.2.0
Checking icu_provider v2.2.0
Checking lexical-write-integer v1.0.6
Checking lexical-parse-integer v1.0.6
Checking rand_core v0.9.5
Compiling semver v1.0.28
Compiling rustc_version v0.4.1
Checking lexical-parse-float v1.0.6
Checking lexical-write-float v1.0.6
Checking futures-channel v0.3.32
Compiling futures-macro v0.3.32
Compiling tokio-macros v2.7.0
Checking half v2.7.1
Checking ppv-lite86 v0.2.21
Checking aho-corasick v1.1.4
Checking arrow-buffer v59.0.0
Checking rand_chacha v0.9.0
Checking ryu v1.0.23
Checking slab v0.4.12
Checking unicode-width v0.2.2
Checking arrow-data v59.0.0
Checking futures-io v0.3.32
Compiling parking_lot_core v0.9.12
Checking unicode-segmentation v1.13.3
Compiling zstd-safe v7.2.4
Checking futures-task v0.3.32
Checking regex-syntax v0.8.11
Checking bitflags v2.13.0
Checking arrow-array v59.0.0
Checking futures-util v0.3.32
Checking regex-automata v0.4.14
Checking arrow-select v59.0.0
Checking comfy-table v7.2.2
Checking rand v0.9.4
Checking arrow-ord v59.0.0
Checking tokio v1.52.3
Checking icu_normalizer v2.2.0
Checking lexical-core v1.0.6
Checking icu_properties v2.2.0
Compiling flatbuffers v25.12.19
Checking atoi v2.0.0
Compiling getrandom v0.4.2
Checking base64 v0.22.1
Checking scopeguard v1.2.0
Checking lock_api v0.4.14
Checking arrow-cast v59.0.0
Checking idna_adapter v1.2.2
Checking twox-hash v2.1.2
Checking regex v1.12.4
Checking csv-core v0.1.13
Compiling thiserror v2.0.18
Checking percent-encoding v2.3.2
Checking csv v1.4.0
Checking form_urlencoded v1.2.2
Checking lz4_flex v0.13.1
Checking idna v1.1.0
Compiling tracing-attributes v0.1.31
Compiling thiserror-impl v2.0.18
Checking tracing-core v0.1.36
Checking same-file v1.0.6
Checking either v1.16.0
Checking simdutf8 v0.1.5
Checking arrow-json v59.0.0
Checking tracing v0.1.44
Checking itertools v0.14.0
Checking walkdir v2.5.0
Checking url v2.5.8
Checking arrow-csv v59.0.0
Checking arrow-string v59.0.0
Checking parking_lot v0.12.5
Checking arrow-arith v59.0.0
Checking arrow-row v59.0.0
Compiling async-trait v0.1.89
Checking http v1.4.2
Checking typenum v1.20.1
Checking log v0.4.32
Checking humantime v2.3.0
Checking uuid v1.23.3
Checking object_store v0.13.2
Compiling pin-project-internal v1.1.13
Checking hybrid-array v0.4.12
Compiling generic-array v0.14.7
Checking datafusion-doc v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/doc)
Compiling rustix v1.1.4
Compiling crossbeam-utils v0.8.21
Checking pin-project v1.1.13
Checking foldhash v0.1.5
Checking hashbrown v0.15.5
Checking crypto-common v0.2.2
Checking block-buffer v0.12.0
Checking futures-executor v0.3.32
Checking linux-raw-sys v0.12.1
Checking const-oid v0.10.2
Checking fixedbitset v0.5.7
Checking digest v0.11.3
Checking zstd v0.13.3
Checking futures v0.3.32
Checking petgraph v0.8.3
Checking arrow-ipc v59.0.0
Checking cpufeatures v0.3.0
Checking fastrand v2.4.1
Checking hashbrown v0.14.5
Checking arrow v59.0.0
Checking datafusion-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/common)
Checking dashmap v6.2.1
Compiling datafusion-macros v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/macros)
Checking tempfile v3.27.0
Checking block-buffer v0.10.4
Checking crypto-common v0.1.7
Compiling blake3 v1.8.5
Checking subtle v2.6.1
Checking digest v0.10.7
Checking constant_time_eq v0.4.2
Checking arrayref v0.3.9
Checking arrayvec v0.7.6
Checking blake2 v0.10.6
Checking sha2 v0.11.0
Checking md-5 v0.11.0
Checking hex v0.4.3
Checking datafusion-common-runtime v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/common-runtime)
Checking glob v0.3.3
Checking tokio-util v0.7.18
Checking tokio-stream v0.1.18
Compiling bigdecimal v0.4.10
Compiling crc32fast v1.5.0
Checking sha1 v0.11.0
Checking datafusion-expr-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/expr-common)
Checking datafusion-physical-expr-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-expr-common)
Checking datafusion-functions-window-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-window-common)
Checking datafusion-functions-aggregate-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-aggregate-common)
Checking datafusion-expr v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/expr)
Checking datafusion-physical-expr v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-expr)
Checking datafusion-execution v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/execution)
Checking datafusion-functions v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions)
Checking datafusion-functions-aggregate v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-aggregate)
Checking datafusion-optimizer v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/optimizer)
Checking datafusion-functions-window v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-window)
Checking datafusion-physical-plan v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-plan)
Checking datafusion-physical-expr-adapter v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-expr-adapter)
Checking datafusion-functions-nested v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-nested)
Checking datafusion-session v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/session)
Checking datafusion-datasource v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/datasource)
Checking datafusion-catalog v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/catalog)
Checking datafusion-pruning v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/pruning)
Checking datafusion-datasource-csv v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/datasource-csv)
Checking datafusion-datasource-json v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/datasource-json)
Checking datafusion-physical-optimizer v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-optimizer)
Checking datafusion-datasource-arrow v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/datasource-arrow)
Checking datafusion-catalog-listing v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/catalog-listing)
Checking datafusion-functions-table v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-table)
Checking datafusion v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/core)
Documenting datafusion-spark v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/spark)
error[E0407]: method `as_any` is not a member of trait `ScalarUDFImpl`
--> /home/runner/work/datafusion/datafusion/datafusion/spark/src/function/string/levenshtein.rs:86:5
|
86 | / fn as_any(&self) -> &dyn Any {
87 | | self
88 | | }
| |_____^ not a member of trait `ScalarUDFImpl`
For more information about this error, try `rustc --explain E0407`.
error: could not document `datafusion-spark`
-----
error: failed to build rustdoc for crate datafusion-spark v54.0.0
note: this is usually due to a compilation error in the crate,
and is unlikely to be a bug in cargo-semver-checks
note: the following command can be used to reproduce the error:
cargo new --lib example &&
cd example &&
echo '[workspace]' >> Cargo.toml &&
cargo add --path /home/runner/work/datafusion/datafusion/datafusion/spark --features core,datafusion,default &&
cargo check &&
cargo doc
Building datafusion-sqllogictest v54.0.0 (current)
error: running cargo-doc on crate 'datafusion-sqllogictest' failed with output:
-----
Compiling proc-macro2 v1.0.106
Compiling quote v1.0.45
Compiling unicode-ident v1.0.24
Compiling libc v0.2.186
Checking cfg-if v1.0.4
Checking bytes v1.11.1
Checking memchr v2.8.1
Compiling serde_core v1.0.228
Compiling syn v2.0.117
Compiling autocfg v1.5.1
Compiling shlex v2.0.1
Compiling jobserver v0.1.34
Compiling find-msvc-tools v0.1.9
Checking itoa v1.0.18
Compiling cc v1.2.63
Checking once_cell v1.21.4
Checking equivalent v1.0.2
Checking allocator-api2 v0.2.21
Checking foldhash v0.2.0
Compiling libm v0.2.16
Compiling num-traits v0.2.19
Checking hashbrown v0.17.1
Checking pin-project-lite v0.2.17
Checking indexmap v2.14.0
Compiling zmij v1.0.21
Checking futures-core v0.3.32
Compiling zerocopy v0.8.52
Checking futures-sink v0.3.32
Checking errno v0.3.14
Checking mio v1.2.1
Checking signal-hook-registry v1.4.8
Checking socket2 v0.6.4
Compiling version_check v0.9.5
Compiling serde v1.0.228
Compiling getrandom v0.3.4
Checking num-integer v0.1.46
Compiling serde_json v1.0.150
Checking slab v0.4.12
Checking futures-channel v0.3.32
Checking smallvec v1.15.1
Checking num-bigint v0.4.6
Compiling synstructure v0.13.2
Checking http v1.4.2
Checking base64 v0.22.1
Checking futures-io v0.3.32
Checking futures-task v0.3.32
Checking iana-time-zone v0.1.65
Checking chrono v0.4.45
Checking tracing-core v0.1.36
Compiling zerocopy-derive v0.8.52
Compiling tokio-macros v2.7.0
Checking tokio v1.52.3
Compiling serde_derive v1.0.228
Compiling futures-macro v0.3.32
Checking futures-util v0.3.32
Compiling tracing-attributes v0.1.31
Compiling zerofrom-derive v0.1.7
Checking siphasher v1.0.3
Checking tracing v0.1.44
Compiling yoke-derive v0.8.2
Checking stable_deref_trait v1.2.1
Checking num-complex v0.4.6
Compiling zerovec-derive v0.11.3
Checking zerofrom v0.1.8
Checking getrandom v0.2.17
Compiling pkg-config v0.3.33
Checking half v2.7.1
Checking phf_shared v0.12.1
Compiling displaydoc v0.2.6
Compiling ahash v0.8.12
Checking yoke v0.8.3
Compiling thiserror v2.0.18
Checking percent-encoding v2.3.2
Checking zerovec v0.11.6
Compiling chrono-tz v0.10.4
Checking phf v0.12.1
Checking arrow-buffer v59.0.0
Checking ppv-lite86 v0.2.21
Compiling thiserror-impl v2.0.18
Checking arrow-schema v59.0.0
Compiling ring v0.17.14
Checking arrow-data v59.0.0
Compiling semver v1.0.28
Checking rand_core v0.10.1
Checking log v0.4.32
Compiling getrandom v0.4.2
Checking tinystr v0.8.3
Checking litemap v0.8.2
Checking untrusted v0.9.0
Checking bitflags v2.13.0
Checking writeable v0.6.3
Checking icu_locale_core v2.2.0
Checking potential_utf v0.1.5
Checking zerotrie v0.2.4
Compiling zstd-sys v2.0.16+zstd.1.5.7
Compiling async-trait v0.1.89
Checking rand_core v0.9.5
Compiling icu_properties_data v2.2.0
Checking utf8_iter v1.0.4
Compiling icu_normalizer_data v2.2.0
Checking icu_collections v2.2.0
Checking rand_chacha v0.9.0
Checking icu_provider v2.2.0
Checking rand v0.9.4
Checking aho-corasick v1.1.4
Checking regex-syntax v0.8.11
Checking arrow-array v59.0.0
Checking ryu v1.0.23
Checking lexical-util v1.0.7
Compiling object v0.37.3
Compiling zstd-safe v7.2.4
Checking regex-automata v0.4.14
Checking icu_normalizer v2.2.0
Checking arrow-select v59.0.0
Checking icu_properties v2.2.0
Compiling rustix v1.1.4
Checking idna_adapter v1.2.2
Checking form_urlencoded v1.2.2
Checking either v1.16.0
Checking regex v1.12.4
Compiling parking_lot_core v0.9.12
Compiling crc32fast v1.5.0
Checking unicode-width v0.2.2
Checking typenum v1.20.1
Checking idna v1.1.0
Checking lexical-write-integer v1.0.6
Checking lexical-parse-integer v1.0.6
Compiling rustc_version v0.4.1
Checking futures-executor v0.3.32
Compiling pin-project-internal v1.1.13
Checking adler2 v2.0.1
Checking simd-adler32 v0.3.9
Checking scopeguard v1.2.0
Checking lock_api v0.4.14
Checking miniz_oxide v0.8.9
Checking futures v0.3.32
Compiling flatbuffers v25.12.19
Checking lexical-parse-float v1.0.6
Checking pin-project v1.1.13
Checking lexical-write-float v1.0.6
Checking url v2.5.8
Checking zlib-rs v0.6.3
Compiling ar_archive_writer v0.5.2
Checking unicode-segmentation v1.13.3
Checking comfy-table v7.2.2
Checking lexical-core v1.0.6
Checking itertools v0.14.0
Compiling psm v0.1.31
Checking flate2 v1.1.9
Checking arrow-ord v59.0.0
Checking twox-hash v2.1.2
Checking atoi v2.0.0
Compiling stacker v0.1.24
Checking hex v0.4.3
Compiling snap v1.1.1
Checking alloc-no-stdlib v2.0.4
Checking arrow-cast v59.0.0
Checking alloc-stdlib v0.2.2
Checking lz4_flex v0.13.1
Checking parking_lot v0.12.5
Checking csv-core v0.1.13
Checking simdutf8 v0.1.5
Checking humantime v2.3.0
Compiling paste v1.0.15
Checking same-file v1.0.6
Checking walkdir v2.5.0
Checking csv v1.4.0
Checking brotli-decompressor v5.0.1
Compiling recursive-proc-macro-impl v0.1.1
Checking subtle v2.6.1
Checking recursive v0.1.1
Checking arrow-json v59.0.0
Checking arrow-csv v59.0.0
Checking brotli v8.0.3
Checking object_store v0.13.2
Checking arrow-string v59.0.0
Checking arrow-row v59.0.0
Checking arrow-arith v59.0.0
Checking uuid v1.23.3
Checking tokio-util v0.7.18
Checking zstd v0.13.3
Checking arrow-ipc v59.0.0
Compiling sqlparser_derive v0.5.0
Compiling seq-macro v0.3.6
Checking cpufeatures v0.3.0
Checking hybrid-array v0.4.12
Checking sqlparser v0.62.0
Checking linux-raw-sys v0.12.1
Checking cmov v0.5.4
Checking arrow v59.0.0
Checking parquet v59.0.0
Checking ctutils v0.4.2
Checking crypto-common v0.2.2
Checking block-buffer v0.12.0
Compiling generic-array v0.14.7
Checking const-oid v0.10.2
Checking digest v0.11.3
Checking datafusion-doc v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/doc)
Checking foldhash v0.1.5
Compiling crossbeam-utils v0.8.21
Checking hashbrown v0.15.5
Checking crypto-common v0.1.7
Checking block-buffer v0.10.4
Checking fixedbitset v0.5.7
Checking fastrand v2.4.1
Compiling heck v0.5.0
Checking tempfile v3.27.0
Checking petgraph v0.8.3
Checking digest v0.10.7
Checking sha2 v0.11.0
Checking md-5 v0.11.0
Checking hashbrown v0.14.5
Checking dashmap v6.2.1
Compiling datafusion-macros v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/macros)
Compiling blake3 v1.8.5
Compiling anyhow v1.0.102
Checking arrayvec v0.7.6
Checking arrayref v0.3.9
Checking constant_time_eq v0.4.2
Checking blake2 v0.10.6
Compiling liblzma-sys v0.4.6
Checking libbz2-rs-sys v0.2.5
Checking bzip2 v0.6.1
Checking datafusion-common-runtime v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/common-runtime)
Compiling httparse v1.10.1
Checking compression-core v0.4.32
Compiling prost-derive v0.14.4
Checking http-body v1.0.1
Checking glob v0.3.3
Checking tower-service v0.3.3
Checking atomic-waker v1.1.2
Checking fnv v1.0.7
Checking try-lock v0.2.5
Checking want v0.3.1
Checking h2 v0.4.14
Checking tokio-stream v0.1.18
Checking zeroize v1.8.2
Checking httpdate v1.0.3
Checking hyper v1.10.1
Checking rustls-pki-types v1.14.1
Checking datafusion-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/common)
Compiling prost v0.14.4
Checking hyper-util v0.1.20
Checking http-body-util v0.1.3
Checking sync_wrapper v1.0.2
Checking tower-layer v0.3.3
Compiling prettyplease v0.2.37
Compiling rustls v0.23.40
Compiling prost-types v0.14.4
Checking liblzma v0.4.6
Checking compression-codecs v0.4.38
Checking async-compression v0.4.42
Checking datafusion-expr-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/expr-common)
Checking rustls-webpki v0.103.13
Compiling serde_derive_internals v0.29.1
Compiling schemars v0.8.22
Checking datafusion-physical-expr-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-expr-common)
Checking mime v0.3.17
Checking axum-core v0.5.6
Compiling schemars_derive v0.8.22
Compiling hashbrown v0.16.1
Checking datafusion-functions-aggregate-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-aggregate-common)
Checking datafusion-functions-window-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-window-common)
Checking datafusion-expr v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/expr)
Checking tower v0.5.3
Checking matchit v0.8.4
Compiling dyn-clone v1.0.20
Compiling multimap v0.10.1
Checking axum v0.8.9
Compiling prost-build v0.14.4
Compiling regress v0.10.5
Checking datafusion-physical-expr v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-expr)
Checking datafusion-execution v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/execution)
Checking datafusion-functions v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions)
Compiling pbjson-build v0.8.0
Checking hyper-timeout v0.5.2
Checking chacha20 v0.10.0
Compiling ident_case v1.0.1
Compiling portable-atomic v1.13.1
Compiling strsim v0.11.1
Compiling typify-impl v0.5.0
Compiling darling_core v0.23.0
Checking rand v0.10.1
Checking tonic v0.14.6
Compiling serde_tokenstream v0.2.3
Checking datafusion-physical-plan v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-plan)
Checking datafusion-physical-expr-adapter v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-expr-adapter)
Checking ureq-proto v0.6.0
Compiling bigdecimal v0.4.10
Checking tinyvec_macros v0.1.1
Compiling bollard-buildkit-proto v0.7.0
Checking crc-catalog v2.5.0
Checking utf8-zero v0.8.1
Checking utf8parse v0.2.2
Checking powerfmt v0.2.0
Checking ureq v3.3.0
Checking deranged v0.5.8
Checking anstyle-parse v1.0.0
Compiling darling_macro v0.23.0
Checking crc v3.4.0
Checking tinyvec v1.11.0
Checking tonic-prost v0.14.6
Checking datafusion-functions-aggregate v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-aggregate)
Compiling typify-macro v0.5.0
Compiling strum_macros v0.28.0
Compiling structmeta-derive v0.3.0
Compiling unsafe-libyaml v0.2.11
Checking time-core v0.1.8
Checking colorchoice v1.0.5
Checking anstyle v1.0.14
Checking anstyle-query v1.1.5
Checking num-conv v0.2.2
Checking is_terminal_polyfill v1.70.2
Checking time v0.3.47
Checking anstream v1.0.0
Compiling serde_yaml v0.9.34+deprecated
Checking arrow-avro v59.0.0
Compiling structmeta v0.3.0
Checking datafusion-functions-nested v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-nested)
Compiling typify v0.5.0
Checking datafusion-session v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/session)
Checking unicode-normalization v0.1.25
Checking datafusion-datasource v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/datasource)
Compiling darling v0.23.0
Compiling pbjson-types v0.8.0
Checking tokio-rustls v0.26.4
Checking datafusion-pruning v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/pruning)
Checking datafusion-catalog v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/catalog)
Compiling serde_repr v0.1.20
Compiling async-stream-impl v0.3.6
Checking num-rational v0.4.2
Checking num-iter v0.1.45
Checking unicode-properties v0.1.4
Checking openssl-probe v0.2.1
Checking unicode-bidi v0.3.18
Checking clap_lex v1.1.0
Checking num v0.4.3
Checking rustls-native-certs v0.8.4
Checking clap_builder v4.6.0
Checking async-stream v0.3.6
Checking bollard-stubs v1.52.1-rc.29.1.3
Checking stringprep v0.1.5
Checking datafusion-datasource-parquet v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/datasource-parquet)
Checking datafusion-physical-optimizer v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-optimizer)
Checking datafusion-sql v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/sql)
Checking hyper-rustls v0.27.9
Checking datafusion-datasource-csv v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/datasource-csv)
Checking datafusion-datasource-json v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/datasource-json)
Checking datafusion-datasource-avro v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/datasource-avro)
Checking datafusion-datasource-arrow v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/datasource-arrow)
Compiling serde_with_macros v3.21.0
Compiling substrait v0.63.0
Compiling parse-display-derive v0.9.1
Checking datafusion-optimizer v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/optimizer)
Checking datafusion-functions-window v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-window)
Checking hyperlocal v0.9.1
Compiling clap_derive v4.6.1
Checking xattr v1.6.1
Checking hmac v0.13.0
Checking serde_urlencoded v0.7.1
Checking pbjson v0.8.0
Checking rand_core v0.6.4
Compiling enum-ordinalize-derive v4.3.2
Compiling fs-err v3.3.0
Checking filetime v0.2.29
Compiling owo-colors v4.3.0
Checking home v0.5.12
Checking fallible-iterator v0.2.0
Checking byteorder v1.5.0
Checking rustc-hash v2.1.2
Checking astral-tokio-tar v0.6.2
Checking bollard v0.20.2
Checking postgres-protocol v0.6.11
Checking clap v4.6.1
Compiling enum-ordinalize v4.3.2
Checking rand_chacha v0.3.1
Checking parse-display v0.9.1
Checking serde_with v3.21.0
Checking ferroid v2.0.0
Compiling postgres-derive v0.4.8
Checking docker_credential v1.4.0
Checking phf_shared v0.13.1
Checking etcetera v0.11.0
Checking escape8259 v0.5.3
Checking unicode-width v0.1.14
Checking libtest-mimic v0.8.2
Checking subst v0.3.8
Checking postgres-types v0.2.13
Checking phf v0.13.1
Checking rand v0.8.6
Checking testcontainers v0.27.3
Compiling educe v0.6.0
Checking md-5 v0.10.6
Checking sha1 v0.11.0
Checking console v0.16.3
Checking itertools v0.13.0
Compiling async-recursion v1.1.1
Checking whoami v2.1.2
Checking unit-prefix v0.5.2
Checking similar v2.7.0
Checking indicatif v0.18.4
Checking sqllogictest v0.29.1
Checking tokio-postgres v0.7.17
Checking testcontainers-modules v0.15.0
Checking datafusion-catalog-listing v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/catalog-listing)
Checking datafusion-functions-table v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-table)
Checking datafusion v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/core)
Checking datafusion-substrait v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/substrait)
Checking datafusion-spark v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/spark)
error[E0407]: method `as_any` is not a member of trait `ScalarUDFImpl`
--> /home/runner/work/datafusion/datafusion/datafusion/spark/src/function/string/levenshtein.rs:86:5
|
86 | / fn as_any(&self) -> &dyn Any {
87 | | self
88 | | }
| |_____^ not a member of trait `ScalarUDFImpl`
For more information about this error, try `rustc --explain E0407`.
error: could not compile `datafusion-spark` (lib) due to 1 previous error
warning: build failed, waiting for other jobs to finish...
-----
error: failed to build rustdoc for crate datafusion-sqllogictest v54.0.0
note: this is usually due to a compilation error in the crate,
and is unlikely to be a bug in cargo-semver-checks
note: the following command can be used to reproduce the error:
cargo new --lib example &&
cd example &&
echo '[workspace]' >> Cargo.toml &&
cargo add --path /home/runner/work/datafusion/datafusion/datafusion/sqllogictest --features avro,backtrace,bytes,chrono,datafusion-substrait,parquet_encryption,postgres,postgres-types,substrait,testcontainers-modules,tokio-postgres &&
cargo check &&
cargo doc
error: aborting due to failure to build rustdoc for crate datafusion-spark v54.0.0
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(spark): add levenshtein with optional threshold support [datafusion]
Posted by "davidlghellin (via GitHub)" <gi...@apache.org>.
davidlghellin commented on code in PR #22890:
URL: https://github.com/apache/datafusion/pull/22890#discussion_r3391141423
##########
datafusion/sqllogictest/test_files/spark/string/levenshtein.slt:
##########
@@ -21,12 +21,224 @@
# For more information, please see:
# https://github.com/apache/datafusion/issues/15914
-## Original Query: SELECT levenshtein('kitten', 'sitting');
-## PySpark 3.5.5 Result: {'levenshtein(kitten, sitting)': 3, 'typeof(levenshtein(kitten, sitting))': 'int', 'typeof(kitten)': 'string', 'typeof(sitting)': 'string'}
-#query
-#SELECT levenshtein('kitten'::string, 'sitting'::string);
-
-## Original Query: SELECT levenshtein('kitten', 'sitting', 2);
-## PySpark 3.5.5 Result: {'levenshtein(kitten, sitting, 2)': -1, 'typeof(levenshtein(kitten, sitting, 2))': 'int', 'typeof(kitten)': 'string', 'typeof(sitting)': 'string', 'typeof(2)': 'int'}
-#query
-#SELECT levenshtein('kitten'::string, 'sitting'::string, 2::int);
+## ── Basic usage ─────────────────────────────────────────────
+
+## Basic distance
+query I
+SELECT levenshtein('kitten', 'sitting');
+----
+3
+
+## Identical strings
+query I
+SELECT levenshtein('hello', 'hello');
+----
+0
+
+## Empty string vs non-empty
+query I
+SELECT levenshtein('', 'abc');
+----
+3
+
+## Both empty strings
+query I
+SELECT levenshtein('', '');
+----
+0
+
+## Single character difference
+query I
+SELECT levenshtein('abc', 'adc');
+----
+1
+
+## ── Threshold (3-argument form) ─────────────────────────────
+
+## Distance within threshold
+query I
+SELECT levenshtein('kitten', 'sitting', 4);
+----
+3
+
+## Distance exceeds threshold → returns -1
+query I
+SELECT levenshtein('kitten', 'sitting', 2);
+----
+-1
+
+## Distance equals threshold (boundary) → returns distance
+query I
+SELECT levenshtein('kitten', 'sitting', 3);
+----
+3
+
+## Threshold zero with different strings
+query I
+SELECT levenshtein('abc', 'def', 0);
+----
+-1
+
+## Threshold zero with identical strings
+query I
+SELECT levenshtein('abc', 'abc', 0);
+----
+0
+
+## ── Null handling ───────────────────────────────────────────
+
+## First argument null
+query I
+SELECT levenshtein(CAST(NULL AS STRING), 'hello');
+----
+NULL
+
+## Second argument null
+query I
+SELECT levenshtein('hello', CAST(NULL AS STRING));
+----
+NULL
+
+## Both arguments null
+query I
+SELECT levenshtein(CAST(NULL AS STRING), CAST(NULL AS STRING));
+----
+NULL
+
+## Null threshold
+query I
+SELECT levenshtein('kitten', 'sitting', CAST(NULL AS INT));
+----
+NULL
+
+## ── Unicode and special characters ──────────────────────────
+
+## Unicode strings
+query I
+SELECT levenshtein('café', 'cafe');
+----
+1
+
+## Strings with spaces
+query I
+SELECT levenshtein('hello world', 'hello world!');
+----
+1
+
+## ── Column expressions ──────────────────────────────────────
+
+## Levenshtein on columns from inline table
+query I
+SELECT levenshtein(s1, s2) AS result FROM VALUES ('abc', 'abc'), ('abc', 'def'), ('kitten', 'sitting') AS t(s1, s2);
+----
+0
+3
+3
+
+## Threshold on columns from inline table
+query I
+SELECT levenshtein(s1, s2, 2) AS result FROM VALUES ('abc', 'abc'), ('abc', 'def'), ('kitten', 'sitting') AS t(s1, s2);
+----
+0
+-1
+-1
+
+## ── Per-row threshold ───────────────────────────────────────
+
+## Different threshold per row
+query I
+SELECT levenshtein(s1, s2, t) AS result FROM VALUES ('abc', 'def', 2), ('abc', 'def', 5), ('abc', 'def', 3) AS t(s1, s2, t);
Review Comment:
Renamed alias `t → v` and column `t → threshold` in the four affected cases. Failures now point at unambiguous names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(spark): add levenshtein with optional threshold support [datafusion]
Posted by "davidlghellin (via GitHub)" <gi...@apache.org>.
davidlghellin commented on PR #22890:
URL: https://github.com/apache/datafusion/pull/22890#issuecomment-4673451233
https://github.com/apache/datafusion/pull/20927
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(spark): add levenshtein with optional threshold support [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22890:
URL: https://github.com/apache/datafusion/pull/22890#issuecomment-5944088866
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org