You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "cshuo (via GitHub)" <gi...@apache.org> on 2026/10/28 01:16:36 UTC

[I] Track Spark data skipping correctness issues [hudi]

cshuo opened a new issue, #20449:
URL: https://github.com/apache/hudi/issues/20449

   ## Description
   
   Several Spark data skipping issues can cause matching rows to be incorrectly pruned when predicates are evaluated against column or partition statistics. This issue tracks the related fixes and test coverage.
   
   The oracle-based tests in #20439 compare pruning results against the original predicates evaluated on raw rows. Any file or partition containing a matching row must remain a candidate.
   
   ## Related issues
   
   - [ ] Missing column statistics: #20296 (fix: #20328)
   - [ ] Prefix `LIKE` predicates: #20430
   - [ ] Order-breaking casts and negative arithmetic factors: #19445
   - [ ] Date/time transformations and format-sensitive parsing: #19481
   - [ ] Partition stats pruning for time travel queries: #20443
   - [ ] Oracle-based regression coverage for file and partition pruning: #20439
   
   ## Gaps to track
   
   Create or link separate issues as needed for confirmed cases that are not fully covered above:
   
   - Arithmetic overflow and bitwise operations that do not preserve ordering.
   - Date/time transformations across DST transitions or with extreme values.
   - Transforms that produce NULL bounds from real values, such as logarithms on negative values.
   
   The related issues can be assigned and addressed independently, with the oracle tests helping validate the fixes.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Umbrella] Track Spark data skipping correctness issues [hudi]

Posted by "wombatu-kun (via GitHub) via commits" <co...@hudi.apache.org>.
wombatu-kun commented on issue #20449:
URL: https://github.com/apache/hudi/issues/20449#issuecomment-6654220058

   Follow-up from the #20519 review: `TimeZoneShiftRule` now needs a fixed-offset zone, so region zones whose offset last changed decades ago stop pruning too, such as `Asia/Shanghai` (1991), `Asia/Tokyo` (1951) and `Asia/Kolkata` (1945). Once #19474 keeps files on NULL verdicts, a `fileGuard` can restore pruning for zones without transition rules: past the zone's last transition the shift is a constant offset, so the guard only needs the file's `lo` to be after it (as an instant for `from_utc_timestamp`, as a wall-clock time for `to_utc_timestamp`).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Umbrella] Track Spark data skipping correctness issues [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on issue #20449:
URL: https://github.com/apache/hudi/issues/20449#issuecomment-6466941036

   ### Plan: one issue and one fix PR per failure mechanism
   
   Each issue covers one failure mechanism and lists every operator it affects; data types are test cases inside it. A fix PR closes one issue and turns one bug family in the #20439 oracle green. Splitting by type would scatter one fix (integral overflow spans int, bigint and smallint), and splitting by operator would bundle unrelated fixes (Multiply has both a sign bug and an overflow bug).
   
   | Failure mechanism | Oracle families | Issue | Fix PR |
   | --- | --- | --- | --- |
   | Casts that wrap around or overflow into NULL | `narrowing_cast`, `decimal_cast` | #19445 | #19475 |
   | Negative constant reverses order in Multiply/Divide | `negative_factor` | #19445 | #20457 |
   | Collation-changing string casts (Spark 4) | not generated | #19445 | none |
   | `LIKE 'prefix%'` translated as equality on the prefix | `like_prefix` | #20430 | #19169 |
   | Missing column or partition stats | `missing_stats_compound`, `missing_partition_stats` | #20296 | #20328 |
   | String parsing and non-sortable date formats | `string_to_datetime`, `non_sortable_format`, `to_unix_timestamp` | #19481 | none |
   | Integral overflow without ANSI in Add, Multiply, ShiftLeft | `add_overflow`, `multiply_overflow`, `shiftleft_overflow` | to file | planned |
   | BitwiseOr with a constant is not monotonic | `bitwise_or` | to file | none |
   | Logarithms of zero or negative bounds give NULL | `log_null_bound` | to file | none |
   | Time-zone conversions across DST transitions | `dst_fall_back`, `to_utc_timestamp_gap` | to file | none |
   | `from_unixtime` for years outside 0000-9999 | `from_unixtime_years` | to file | none |
   
   #19445 spans three mechanisms, so it closes when the last of #19475, #20457 and the collation fix lands.
   
   ### How the oracle verifies it
   
   - Runs each predicate over the raw rows with Spark; any file or partition holding a TRUE row must survive column stats and partition stats pruning.
   - 9 column types with edge values, nulls and missing stats; every whitelisted transform in every comparison shape and in AND/OR/NOT trees; ANSI on/off x 3 time zones; fixed seed.
   - One test per family, one for unattributed violations (a new mechanism fails the build), one sanity test. Each failure prints its smallest counterexample.
   
   ### Gaps in the oracle itself
   
   | Gap | Ticket |
   | --- | --- |
   | Partition stats exceptions are only logged, though they fail the query | none |
   | No BOOLEAN/BINARY cast targets; no template that uses one column twice | none |
   | float, boolean, binary, nested and collated columns; NaN | #18754 / #18792 (NaN) |
   | Expression, record-level and bloom indexes | #20255 / #20256 (expression index) |
   | MOR log files, time travel, bootstrap | #20443 / #20451, #20266 / #20267 |
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org