[I] [BUG] spark: lower()/upper() data skipping prunes files that hold matching rows [hudi]

["wombatu-kun (via GitHub)" <gi...@apache.org>]

wombatu-kun opened a new issue, #20227:
URL: https://github.com/apache/hudi/issues/20227

   ### Bug Description
   
   **What happened:**
   
   With the column stats index enabled, a `lower(col)` or `upper(col)` filter can prune a file that holds a matching row. Data skipping treats `lower`/`upper` as order-preserving and applies them to the file's min/max (`BaseHoodieCatalystExpressionUtils`), but case folding does not preserve the binary ordering of strings: a file holding `'Zebra'` and `'apple'` has min `'Zebra'` and max `'apple'`, and `lower('apple')` sorts before `'zebra'`, so the file is pruned.
   
   **What you expected:**
   
   The matching rows returned.
   
   **Steps to reproduce:**
   
   ```sql
   CREATE TABLE t (id INT, name STRING) USING hudi
   TBLPROPERTIES (primaryKey = 'id', 'hoodie.metadata.index.column.stats.enable' = 'true');
   
   INSERT INTO t VALUES (1, 'Zebra'), (2, 'apple');
   
   SELECT id, name FROM t WHERE lower(name) = 'zebra';
   -- expected: 1, Zebra
   -- actual: no rows
   
   SELECT id, name FROM t WHERE upper(name) = 'APPLE';
   -- expected: 2, apple
   -- actual: no rows
   ```
   
   Expression indexes on `lower(col)`/`upper(col)` are not affected: their filters are rewritten to the indexed column before translation.
   
   ### Environment
   
   **Hudi version:** master (1.3.0-SNAPSHOT)
   **Query engine:** Spark 3.5
   **Relevant configs:** `hoodie.metadata.index.column.stats.enable=true`, `hoodie.enable.data.skipping=true` (default)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [BUG] spark: lower()/upper() data skipping prunes files that hold matching rows [hudi]

["voonhous (via GitHub)" <gi...@apache.org>]

voonhous commented on issue #20227:
URL: https://github.com/apache/hudi/issues/20227#issuecomment-6209968098

   Okay, this is a serious correctness issue that we need to fix ASAP, thank you for catching this!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [BUG] spark: lower()/upper() data skipping prunes files that hold matching rows [hudi]

["voonhous (via GitHub)" <gi...@apache.org>]

voonhous closed issue #20227: [BUG] spark: lower()/upper() data skipping prunes files that hold matching rows
URL: https://github.com/apache/hudi/issues/20227


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org