Re: [I] csi will cause data loss during sql query [hudi]

["nsivabalan (via GitHub)" <gi...@apache.org>]

nsivabalan commented on issue #16360:
URL: https://github.com/apache/hudi/issues/16360#issuecomment-4795591791

   This appears to have been fixed on master. The `IsNotNull(col)` predicate in `DataSkippingUtils.scala` now translates to `nullCount IS NULL OR valueCount IS NULL OR nullCount < valueCount`, so a file is **not** skipped when column stats are null/missing — which is the exact scenario described here for complex columns like `array<struct<...>>` whose parent column has no min/max/nullCount in the MDT col_stats partition.
   
   Relevant fix: commit b2aef89a8737 ([HUDI-7246] #10389), with follow-on hardening in 3040aa5b4b36 ([HUDI-8663] #12575) which routes the lookup through `indexedCols` so unindexed/complex columns don't reach the predicate at all.
   
   See `hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/DataSkippingUtils.scala` lines 262-273 on master. If you can still reproduce the data loss on a recent build, please attach a minimal repro and reopen — otherwise this can likely be closed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org