You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "wangyong9999 (via GitHub)" <gi...@apache.org> on 2026/09/13 05:51:14 UTC

[I] Reduce repeated Parquet OffsetIndex parsing and page planning [paimon-cpp]

wangyong9999 opened a new issue, #328:
URL: https://github.com/apache/paimon-cpp/issues/328

   ### Problem
   
   Selective Parquet reads repeat CPU work within one file reader: Arrow 17 reparses OffsetIndex on each lookup, page planning scans page locations for sparse row selections, and decoding rebuilds a direct-read plan just to determine whether compressed row coordinates apply.
   
   The byte cache in #272 reduces repeated I/O but does not eliminate this parsing and planning work.
   
   ### Proposed improvement
   
   - Reuse parsed OffsetIndex objects within each retained row-group reader.
   - Seek across page gaps for sparse selections and visit each selected page once.
   - Reuse the direct-plan decision during decoding.
   
   Preserve index validation, dictionary handling, per-leaf row coordinates and missing-index fallbacks. Keep parsed objects reader-local without introducing a shared data cache or public option.
   
   ### Validation
   
   Check selected values and index lifetimes, including sparse/dense selections and the row-group retention limit. Compare against main with the format benchmark, reporting latency and storage bytes without assuming a general speedup.
   
   Implementation: #314.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Reuse Parquet OffsetIndex objects and direct-plan decisions [paimon-cpp]

Posted by "lxy-9602 (via GitHub)" <gi...@apache.org>.
lxy-9602 closed issue #328: Reuse Parquet OffsetIndex objects and direct-plan decisions
URL: https://github.com/apache/paimon-cpp/issues/328


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org