You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@auron.apache.org by "weiqingy (via GitHub)" <gi...@apache.org> on 2026/06/12 06:33:45 UTC

Re: [PR] [AURON #2279] Implement native Levenshtein with threshold support for Spark 4.0+ [auron]

weiqingy commented on code in PR #2280:
URL: https://github.com/apache/auron/pull/2280#discussion_r3401208997


##########
native-engine/datafusion-ext-functions/src/spark_strings.rs:
##########
@@ -395,6 +594,124 @@ mod test {
         }
     }
 
+    #[test]
+    fn test_spark_levenshtein_array() -> Result<()> {
+        let r = spark_levenshtein(&vec![
+            ColumnarValue::Array(Arc::new(StringArray::from_iter(vec![
+                Some("kitten".to_string()),
+                Some("frog".to_string()),
+                Some("千世".to_string()),
+                None,
+            ]))),
+            ColumnarValue::Array(Arc::new(StringArray::from_iter(vec![
+                Some("sitting".to_string()),
+                Some("fog".to_string()),
+                Some("世界千世".to_string()),
+                Some("abc".to_string()),
+            ]))),
+        ])?;
+        let s = r.into_array(4)?;
+        assert_eq!(
+            as_int32_array(&s)?.into_iter().collect::<Vec<_>>(),
+            vec![Some(3), Some(1), Some(2), None]
+        );
+        Ok(())
+    }
+
+    #[test]
+    fn test_spark_levenshtein_threshold() -> Result<()> {
+        let r = spark_levenshtein(&vec![
+            ColumnarValue::Array(Arc::new(StringArray::from_iter(vec![
+                Some("kitten".to_string()),
+                Some("kitten".to_string()),
+                Some("abc".to_string()),
+                Some("abc".to_string()),
+                Some("".to_string()),
+                Some("abc".to_string()),
+            ]))),
+            ColumnarValue::Array(Arc::new(StringArray::from_iter(vec![
+                Some("sitting".to_string()),
+                Some("sitting".to_string()),
+                Some("abc".to_string()),
+                Some("xyz".to_string()),
+                Some("abc".to_string()),
+                Some("abc".to_string()),
+            ]))),
+            ColumnarValue::Array(Arc::new(Int32Array::from_iter(vec![
+                Some(2),
+                Some(3),
+                Some(0),
+                None,
+                Some(3),
+                Some(-1),
+            ]))),
+        ])?;
+        let s = r.into_array(6)?;
+        assert_eq!(
+            as_int32_array(&s)?.into_iter().collect::<Vec<_>>(),

Review Comment:
   Every threshold case here runs on short strings (≤7 chars), so none of them exercises the band on a string long enough that a wrong `min-1` / `max+1` sentinel would diverge from the naive DP — the band stays wide enough to cover the whole row. The boundary-poisoning logic is the part most exposed to a future edit, and the current cases wouldn't catch a regression in it.
   
   Would you add a long-string case with a tight band — something like `levenshtein("abcdefghij", "abXdefghij", 1)` (true distance 1, just inside the band) paired with the same inputs at threshold `0` (→ -1)? That pair specifically stresses the out-of-band sentinel handling that the short cases don't reach.



##########
native-engine/auron-planner/proto/auron.proto:
##########
@@ -282,7 +282,7 @@ enum ScalarFunction {
   Hex=66;
   Power=67;
   IsNaN=69;
-  Levenshtein=80;
+  // Levenshtein=80;

Review Comment:
   Commenting out the line frees number 80 without a `reserved 80;` guard, so a future enum value could reclaim it. The practical risk is small — Auron plans are built and consumed in-process rather than persisted across versions, and the enum already has unguarded gaps (68, 70–79), so this matches existing house style.
   
   Is leaving a commented-out enum line the deliberate convention here, or worth tightening to the following while the change is fresh?
   
   ```proto
     reserved 80;
     reserved "Levenshtein";
   ```
   
   That makes the intent explicit and lets the compiler reject any accidental reuse of 80 or the name.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org