[I] Missing global index metadata can silently prune matching rows [paimon-rust]

["XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>]

XiaoHongbo-Hope opened a new issue, #577:
URL: https://github.com/apache/paimon-rust/issues/577

   ### Search before asking
   
   - [x] I searched the existing issues and found nothing similar.
   
   ### Please describe the bug
   
   On the latest main commit a9a8f6b, GlobalIndexScanner converts missing or unparseable sorted global-index metadata into BTreeIndexMeta(None, None, false). BTreeIndexMeta then interprets that value as an only-null index.
   
   For equality and range predicates, file-level pruning reports no match without opening the physical index file. The scanner returns an indexed result such as Some([]) instead of None, so rows covered by that index shard can be silently omitted in fast, full, and detail search modes.
   
   A minimal reproduction uses two BTree shards for the same field: one with valid metadata and one with missing metadata. Querying a key present in both shards returns only the valid shard result instead of falling back to the normal table scan.
   
   Malformed key lengths are related: BTreeIndexMeta::deserialize currently slices with unchecked lengths and can panic before the scanner can handle the parse error.
   
   ### Solution
   
   Preserve missing or unparseable metadata as unavailable and return None when evaluating that field, allowing the caller to use the normal table scan. Validate serialized key lengths before slicing so malformed metadata returns InvalidData instead of panicking.
   
   ### Are you willing to submit a PR?
   
   - [x] I am willing to submit a PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Missing global index metadata can silently prune matching rows [paimon-rust]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi closed issue #577: Missing global index metadata can silently prune matching rows
URL: https://github.com/apache/paimon-rust/issues/577


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org