You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "sarutak (via GitHub)" <gi...@apache.org> on 2026/06/03 21:14:36 UTC

[PR] [SPARK-57253][SQL] Add `jaro_winkler_similarity` built-in function [spark]

sarutak opened a new pull request, #56310:
URL: https://github.com/apache/spark/pull/56310

   ### What changes were proposed in this pull request?
   This PR adds a new built-in string function `jaro_winkler_similarity(str1, str2)` that computes the Jaro-Winkler similarity between two strings. The result is a double between 0.0 (no similarity) and 1.0 (identical strings).
   
   The Jaro-Winkler metric is an extension of the Jaro similarity that gives a bonus for common prefixes (up to 4 characters), making it especially suited for short strings such as names.
   
   Changes:
   - `UTF8String.jaroWinklerSimilarity()`  (core algorithm implementation)
   - `JaroWinkler` expression with codegen support
   - Registration as `jaro_winkler_similarity` in `FunctionRegistry`
   - Unit tests covering basic cases, symmetry, multi-byte strings, and null handling
   
   ### Why are the changes needed?
   Jaro-Winkler is one of the most commonly used string similarity metrics for record linkage, deduplication, and fuzzy matching. It is available as a built-in function in DuckDB, SQL Server 2025, and various other engines, but Spark users currently need to implement it as a UDF.
   
   A built-in function provides:
   - Codegen support (no UDF overhead in whole-stage codegen pipelines)
   - Consistency with the existing `levenshtein` function
   - Better discoverability for SQL users
   
   ### Does this PR introduce _any_ user-facing change?
   Yes. A new SQL function `jaro_winkler_similarity` is added.
   
   ### How was this patch tested?
   Added new tests.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   Kiro CLI / Claude


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-57253][SQL] Add `jaro_winkler_similarity` built-in function [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #56310:
URL: https://github.com/apache/spark/pull/56310#discussion_r3356927103


##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -15577,6 +15577,40 @@ def levenshtein(
         )
 
 
+@_try_remote_functions
+def jaro_winkler_similarity(left: "ColumnOrName", right: "ColumnOrName") -> Column:
+    """Computes the Jaro-Winkler similarity between the two given strings.
+
+    The result is a double between 0.0 (no similarity) and 1.0 (identical strings).
+
+    .. versionadded:: 4.3.0
+
+    Parameters
+    ----------
+    left : :class:`~pyspark.sql.Column` or column name
+        first column value.
+    right : :class:`~pyspark.sql.Column` or column name
+        second column value.
+
+    Returns
+    -------
+    :class:`~pyspark.sql.Column`
+        Jaro-Winkler similarity as a double value.
+
+    Examples
+    --------
+    >>> from pyspark.sql import functions as sf
+    >>> df = spark.createDataFrame([('MARTHA', 'MARHTA')], ['l', 'r'])
+    >>> df.select('*', sf.jaro_winkler_similarity('l', 'r')).show()
+    +------+------+-------------------------------+
+    |     l|     r|jaro_winkler_similarity(l, r)|

Review Comment:
   **Blocking:** this `.show()` table looks hand-written and will fail the doctest. The third column's separator rows carry 31 dashes, but the column header `jaro_winkler_similarity(l, r)` is 29 characters, so the real rendered column width is 29. The doctest runs under `ELLIPSIS | NORMALIZE_WHITESPACE` (see `_test()`), which tolerates the value-cell space padding but **not** the dash-count mismatch (dashes aren't whitespace), so the comparison fails.
   
   Please regenerate the block by actually running the example. The third column should be 29 dashes wide, e.g.:
   ```
   +------+------+-----------------------------+
   |     l|     r|jaro_winkler_similarity(l, r)|
   +------+------+-----------------------------+
   |MARTHA|MARHTA|           0.9611111111111111|
   +------+------+-----------------------------+
   ```



##########
common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:
##########
@@ -1981,6 +1981,67 @@ public boolean semanticEquals(final UTF8String other, int collationId) {
     return CollationFactory.fetchCollation(collationId).equalsFunction.apply(this, other);
   }
 
+  /**
+   * Returns the Jaro-Winkler similarity between this string and another, as a double in [0, 1].
+   * A score of 1.0 means the strings are identical, 0.0 means no similarity.
+   * The Jaro-Winkler metric gives a bonus for common prefixes (up to 4 characters).
+   */
+  public double jaroWinklerSimilarity(UTF8String other) {
+    // Decode to char arrays for character-level comparison

Review Comment:
   Nit: the code creates `String` objects (`s1`, `s2`), not char arrays, and compares via `charAt()`.
   ```suggestion
       // Decode to Strings for character-level comparison
   ```



##########
python/docs/source/reference/pyspark.sql/functions.rst:
##########
@@ -184,6 +184,7 @@ String Functions
     left
     length
     levenshtein
+    jaro_winkler_similarity

Review Comment:
   Nit: this list is alphabetical (`left`, `length`, `levenshtein`, `locate`, `lower`, `lpad`), so `jaro_winkler_similarity` belongs before the `l*` entries rather than after `levenshtein`. The same ordering issue exists in `python/pyspark/sql/functions/__init__.py`'s `__all__`. (The Scala `FunctionRegistry` / `functions.scala` placements next to `levenshtein` are fine — those lists aren't alphabetical.)



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala:
##########
@@ -2722,6 +2722,52 @@ case class Levenshtein(
   }
 }
 
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+  usage = "_FUNC_(str1, str2) - Returns the Jaro-Winkler similarity between the two given strings. The result is a double between 0 and 1, where 1 means identical.",
+  examples = """
+    Examples:
+      > SELECT _FUNC_('MARTHA', 'MARHTA');
+       0.9611111111111111
+      > SELECT _FUNC_('kitten', 'sitting');
+       0.746031746031746
+      > SELECT _FUNC_('ABC', 'XYZ');
+       0.0
+  """,
+  since = "4.3.0",
+  group = "string_funcs")
+// scalastyle:on line.size.limit
+case class JaroWinkler(left: Expression, right: Expression)
+  extends BinaryExpression
+  with ImplicitCastInputTypes {
+
+  override def nullIntolerant: Boolean = true
+
+  override def inputTypes: Seq[AbstractDataType] = Seq(
+    StringTypeWithCollation(supportsTrimCollation = true),
+    StringTypeWithCollation(supportsTrimCollation = true))
+
+  override def dataType: DataType = DoubleType
+
+  override def prettyName: String = "jaro_winkler_similarity"
+
+  override protected def withNewChildrenInternal(
+      newLeft: Expression, newRight: Expression): JaroWinkler =
+    copy(left = newLeft, right = newRight)
+
+  // Note: This implementation uses String.charAt() which operates on UTF-16 code units.
+  // Strings containing supplementary characters (surrogate pairs) may produce
+  // inaccurate results, consistent with the existing levenshtein() implementation.

Review Comment:
   This claim is inaccurate: `levenshteinDistance` does **not** use `String.charAt()`. It iterates UTF-8 character boundaries directly on the bytes (`numChars()` / `getByte()` / `numBytesForFirstByte()`), so it treats a supplementary character as a single unit — whereas `jaroWinklerSimilarity` decodes via `toString()` + `charAt()` and splits a supplementary character into two UTF-16 units. So the two are not consistent, and the `toString()` path also allocates two Java `String`s per row.
   
   The cleanest fix is to implement `jaroWinklerSimilarity` on the `UTF8String` representation the way `levenshteinDistance` does — that removes the per-row allocation, makes supplementary-character handling genuinely consistent with `levenshtein`, and makes this note true. If you prefer to keep the `charAt` approach, please at least reword the comment so it doesn't assert consistency with `levenshtein`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-57253][SQL] Add `jaro_winkler_similarity` built-in function [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #56310: [SPARK-57253][SQL] Add `jaro_winkler_similarity` built-in function
URL: https://github.com/apache/spark/pull/56310


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-57253][SQL] Add `jaro_winkler_similarity` built-in function [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #56310:
URL: https://github.com/apache/spark/pull/56310#discussion_r3376672718


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala:
##########
@@ -2721,6 +2721,53 @@ case class Levenshtein(
   }
 }
 
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+  usage = "_FUNC_(str1, str2) - Returns the Jaro-Winkler similarity between the two given strings. The result is a double between 0 and 1, where 1 means identical.",
+  examples = """
+    Examples:
+      > SELECT _FUNC_('MARTHA', 'MARHTA');
+       0.9611111111111111
+      > SELECT _FUNC_('kitten', 'sitting');
+       0.746031746031746
+      > SELECT _FUNC_('ABC', 'XYZ');
+       0.0
+  """,
+  since = "4.3.0",
+  group = "string_funcs")
+// scalastyle:on line.size.limit
+case class JaroWinkler(left: Expression, right: Expression)

Review Comment:
   Consider implementing this as a `RuntimeReplaceable` expression delegating to a `StaticInvoke`, rather than hand-rolling `nullSafeEval` + `doGenCode` on top of a `UTF8String` method.
   
   This copies `Levenshtein`'s pattern, but `Levenshtein` only hand-rolls codegen because of its `threshold` short-circuit (early exit) — `jaro_winkler_similarity` has no such optimization, so there's no reason to write custom eval/codegen. The established pattern for "delegate the algorithm to a JVM helper" is `RuntimeReplaceable` + `StaticInvoke`, used by the immediately-adjacent registry neighbor `luhn_check` (`Luhncheck`) and 17 other expressions in this file. `BinaryPredicate` (a few hundred lines up) is the two-arg template:
   
   ```scala
   case class JaroWinkler(left: Expression, right: Expression)
     extends RuntimeReplaceable with ImplicitCastInputTypes with BinaryLike[Expression] {
   
     override lazy val replacement: Expression = StaticInvoke(
       classOf[ExpressionImplUtils],
       DoubleType,
       "jaroWinklerSimilarity",
       Seq(left, right),
       inputTypes)
   
     override def inputTypes: Seq[AbstractDataType] = Seq(
       StringTypeWithCollation(supportsTrimCollation = true),
       StringTypeWithCollation(supportsTrimCollation = true))
   
     override def prettyName: String = "jaro_winkler_similarity"
   
     override protected def withNewChildrenInternal(
         newLeft: Expression, newRight: Expression): Expression =
       copy(left = newLeft, right = newRight)
   }
   ```
   
   Benefits:
   - `RuntimeReplaceable` makes `eval`/`doGenCode` `final` (delegating to `replacement`), so you drop both — and with them the interpreted-vs-codegen dual path.
   - `StaticInvoke`'s default `propagateNull = true` gives null-intolerance, and `dataType` is derived from the replacement, so the `nullIntolerant` and `dataType` overrides also go away.
   
   I'd also move the algorithm into `ExpressionImplUtils` (where `isLuhnNumber` lives — it already imports `UTF8String`) as a static `jaroWinklerSimilarity(UTF8String, UTF8String): double`. The current `UTF8String.jaroWinklerSimilarity` has no other caller and immediately `toString()`s both operands, so it never uses the byte representation that `UTF8String` methods exist for (unlike `levenshteinDistance`, which genuinely iterates the bytes) — it doesn't really belong on `UTF8String`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-57253][SQL] Add `jaro_winkler_similarity` built-in function [spark]

Posted by "shrirangmhalgi (via GitHub)" <gi...@apache.org>.
shrirangmhalgi commented on PR #56310:
URL: https://github.com/apache/spark/pull/56310#issuecomment-4620335879

   The existing `levenshtein()` function is exposed in both `functions.scala` (Scala/Java DataFrame API) and `pyspark.sql.functions` (PySpark API), in addition to the SQL registration. This PR only registers `jaro_winkler_similarity` in `FunctionRegistry` - so it's accessible via SQL but not via `df.select(jaro_winkler_similarity(...))` in Scala or `F.jaro_winkler_similarity(...)` in PySpark. 
   
   Is the DataFrame API planned as a follow-up, or should it be included here for consistency?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-57253][SQL] Add `jaro_winkler_similarity` built-in function [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #56310:
URL: https://github.com/apache/spark/pull/56310#issuecomment-4655919969

   spark connect test failure unrelated, thanks, merging to master/4.x


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-57253][SQL] Add `jaro_winkler_similarity` built-in function [spark]

Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on PR #56310:
URL: https://github.com/apache/spark/pull/56310#issuecomment-4657316534

   Thank you all for reviewing!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-57253][SQL] Add `jaro_winkler_similarity` built-in function [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #56310:
URL: https://github.com/apache/spark/pull/56310#issuecomment-4616974865

   cc @cloud-fan 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org