You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "viirya (via GitHub)" <gi...@apache.org> on 2026/09/30 18:20:57 UTC

[I] instr, substring_index, trim with a trim string, greatest and least ignore string collation [datafusion-comet]

viirya opened a new issue, #6472:
URL: https://github.com/apache/datafusion-comet/issues/6472

   ### Describe the bug
   
   On Spark 4.x with the default `spark.comet.exec.scalaUDF.codegen.enabled=true`, several native string expressions return wrong results when their input has a non-`UTF8_BINARY` collation. A cast to a collated string type runs through the JVM codegen dispatcher, but these expressions above it still run natively and compare or search strings by raw bytes, so they ignore the collation:
   
   - `instr`, `substring_index` (`UTF8_LCASE`, `UNICODE_CI`)
   - `trim`, `ltrim`, `rtrim` with an explicit trim string (`UTF8_LCASE`, `UNICODE_CI`; `rtrim` also `UTF8_BINARY_RTRIM`)
   - `greatest`, `least` (`UTF8_LCASE`, `UNICODE_CI`; `least` also `UTF8_BINARY_RTRIM`)
   
   This is the string counterpart of #6470, which covers the array functions.
   
   ### Steps to reproduce
   
   Parquet table `t` with columns `_1`, `_2` and rows `("a", "A")`, `("x ", "x")`, `("b", "c")`, `("Hello World", "hello")`, `("a,B,c", "b")`. With `a = CAST(_1 AS STRING COLLATE UTF8_LCASE)` and `b = CAST(_2 AS STRING COLLATE UTF8_LCASE)`:
   
   | Expression | Row | Spark | Comet |
   | --- | --- | --- | --- |
   | `instr(a, b)` | `('a', 'A')` | 1 | 0 |
   | `instr(a, b)` | `('a,B,c', 'b')` | 3 | 0 |
   | `substring_index(a, b, 1)` | `('Hello World', 'hello')` | `''` | `'Hello World'` |
   | `trim(BOTH b FROM a)` | `('Hello World', 'hello')` | `' World'` | `'Hello World'` |
   | `greatest(a, b)` | `('Hello World', 'hello')` | `'Hello World'` | `'hello'` |
   | `least(a, b)` | `('a', 'A')` | `'a'` | `'A'` |
   
   The plan shows `cast` dispatched and the string expression native. The mismatches reproduce on Spark 4.0 and 4.1.
   
   ### Expected behavior
   
   The results match Spark. Other collation-sensitive string expressions already route collated input through the codegen dispatcher, for example `contains`, `startswith`, `endswith`, `locate`, `find_in_set`, `replace` and `levenshtein`.
   
   ### Additional context
   
   `trim(a)` without a trim string, `upper`, `concat_ws`, `hash` and `xxhash64` already match Spark on the same data.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] instr, substring_index, trim with a trim string, greatest and least ignore string collation [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #6472:
URL: https://github.com/apache/datafusion-comet/issues/6472#issuecomment-5917149307

   I am working on a fix for this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] instr, substring_index, trim with a trim string, greatest and least ignore string collation [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #6472: instr, substring_index, trim with a trim string, greatest and least ignore string collation
URL: https://github.com/apache/datafusion-comet/issues/6472


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org