[I] [BUG] Spark data skipping: integral overflow in Add, Multiply and ShiftLeft prunes files with matching rows [hudi]

["voonhous (via GitHub)" <gi...@apache.org>]

voonhous opened a new issue, #20460:
URL: https://github.com/apache/hudi/issues/20460

   ### Describe the problem
   
   Data skipping treats `Add`, `Multiply` and `ShiftLeft` with a constant as order-preserving and re-applies them to a column's min/max stats. Without ANSI, integral arithmetic wraps around on overflow, so the transformed max can land below the transformed min and files holding matching rows are pruned. A positive factor does not help: `Long.MaxValue * 2L` wraps to `-2`.
   
   ### Repro
   
   | Filter | File values | Translated bound | Matching row |
   | --- | --- | --- | --- |
   | `l * 2L > 100` (bigint) | `{51, Long.MaxValue}` | `max * 2L = -2 > 100`: pruned | `51 * 2 = 102` |
   | `i + 1 > 0` (int) | `{0, Int.MaxValue}` | `max + 1 = Int.MinValue > 0`: pruned | `0 + 1 = 1` |
   | `shiftleft(i, 1) > 0` (int) | `{1, Int.MaxValue}` | `max << 1 = -2 > 0`: pruned | `1 << 1 = 2` |
   
   The oracle test in #20439 reproduces this as the `add_overflow`, `multiply_overflow` and `shiftleft_overflow` families (column and partition stats).
   
   ### Suggested fix
   
   In `BaseHoodieCatalystExpressionUtils.OrderPreservingTransformation`, match `Add`, `Multiply` and `ShiftLeft` only when the result cannot wrap: floating-point results saturate to infinity and keep the ordering. Integral results, and decimal results that can overflow into NULL, fall back to no pruning. This costs pruning for filters such as `int_col + 1 > 5`, and `ShiftLeft` (integral only) stops pruning entirely.
   
   Split out of #19445 (one issue per failure mechanism, see #20449). The sign check for `Multiply`/`Divide` is in #20457; this fix builds on it.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [BUG] Spark data skipping: overflow in Add, Multiply, ShiftLeft and decimal Divide prunes files with matching rows [hudi]

["voonhous (via GitHub) via commits" <co...@hudi.apache.org>]

voonhous closed issue #20460: [BUG] Spark data skipping: overflow in Add, Multiply, ShiftLeft and decimal Divide prunes files with matching rows
URL: https://github.com/apache/hudi/issues/20460


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org