You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "anoopj (via GitHub)" <gi...@apache.org> on 2026/09/22 20:53:01 UTC

[I] Page-index pruning can drop rows for INT32/INT64-encoded decimals [iceberg-rust]

anoopj opened a new issue, #3264:
URL: https://github.com/apache/iceberg-rust/issues/3264

   ### Apache Iceberg Rust version
   
   Latest main
   
   ### Describe the bug
   
   Surfaced during https://github.com/apache/iceberg-rust/pull/3247#pullrequestreview-5282630063
   
   `PageIndexEvaluator::apply_predicate_to_column_index` constructs INT32 and INT64 page bounds as `PrimitiveLiteral::Int` and `PrimitiveLiteral::Long`, regardless of the Iceberg field type.
   
     For decimal fields, query datums use `PrimitiveLiteral::Int128`. `Datum::partial_cmp` only compares decimal datums when both literals are Int128, so comparisons between these page bounds and query datums return None.
   
     This can silently prune matching pages when page-index row selection is enabled:
   
     - Inequality predicates (<, <=, >, >=) treat the failed comparison as false and reject the page.
     - IN can conclude that no literal is within the page bounds and reject the page.
     - Equality currently fails open and retains the page.
   
     The row-group statistics path already handles these encodings correctly: `get_parquet_stat_min_as_datum` (and max) convert decimal `Statistics::Int32` and `Statistics::Int64` values to `PrimitiveLiteral::Int128`.
   
     Parquet commonly encodes decimal precision ≤9 as INT32 and precision 10–18 as INT64, including files produced by the current Arrow/Parquet writer.
   
     Fix direction: when the Iceberg field type is decimal, convert `INT32/INT64` page-index bounds to `Int128`, matching the row-group statistics path. Add regression coverage for inequality and IN predicates with row selection enabled.
   
   ### To Reproduce
   
   _No response_
   
   ### Expected behavior
   
   _No response_
   
   ### Willingness to contribute
   
   I can contribute a fix for this bug independently


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Page-index pruning can drop rows for INT32/INT64-encoded decimals [iceberg-rust]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko closed issue #3264: Page-index pruning can drop rows for INT32/INT64-encoded decimals
URL: https://github.com/apache/iceberg-rust/issues/3264


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org