[PR] perf(manifest): selectively decode bucket entries with bounded block … [paimon-cpp]

["gripleaf (via GitHub)" <gi...@apache.org>]

gripleaf opened a new pull request, #310:
URL: https://github.com/apache/paimon-cpp/pull/310

   ### Purpose
   
     Linked issue: #240 
   
     Reduce unnecessary decoding and Arrow materialization when reading a single bucket from a mixed-bucket manifest.
   
     - When manifest bytes are retained in memory, first read the version and bucket fields, validate every entry’s version, and collect matching row IDs. Then materialize only matching entries using precise bitmap selection.
     - Support precise bitmap selection in the Avro reader while preserving original file row IDs.
     - Reuse reader-local block indexes to skip blocks without matching rows. Limit indexes to 65,536 blocks and fall back to sequential selection when the limit is exceeded or the initial pass is incomplete.
     - Preserve single-pass reads when bytes are not retained, precise selection is unavailable, or manifest metadata identifies a single matching bucket.
   
     This change does not include the separate Avro skip-plan optimization. No isolated end-to-end latency improvement is claimed.
   
     ### Tests
   
     Added coverage for:
   
     - Empty, sparse, full, and out-of-range bitmap selections.
     - Nested nullable values, original row IDs, schema resets, and repeated EOF.
     - Cross-block selection after complete and incomplete projection passes.
     - Sequential fallback after exceeding the block-index limit.
     - Cached and uncached bucket reads.
     - Invalid versions and null version/bucket fields outside the selected bucket.
   
     Local validation:
   
     - Avro tests: 73/73 passed.
     - Manifest and scan tests: 43/43 passed.
     - Read integration tests: 294/294 passed.
     - Core tests: 1,899/1,901 passed. Two commit-path tests failed because their fixtures select ORC manifests, which the writer rejects as read-only. A separate clean-main reproduction was not run.
     - Changed-file pre-commit checks and git diff --check passed.
   
     For local GCC 13 validation, -Wmaybe-uninitialized was downgraded from an error only for the unchanged arrow_realtime_store_test.cpp generated build rule. No warning-policy changes are included.
   
     ### API and Format
   
     No public API signatures, storage formats, or protocols are changed.
   
     The Avro reader now honors the existing selection bitmap API and reports precise bitmap-selection support. Block indexes are transient, bounded, reader-local state.
   
     ### Documentation
   
     No new user-facing configuration is introduced. Implementation comments describe the retained-bytes requirement, bounded indexes, and fallback behavior.
   
     ### Generative AI tooling
   
     Generated-by: OpenAI Codex (GPT-5).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] perf(manifest): selectively decode bucket entries with bounded block … [paimon-cpp]

["gripleaf (via GitHub)" <gi...@apache.org>]

gripleaf closed pull request #310: perf(manifest): selectively decode bucket entries with bounded block …
URL: https://github.com/apache/paimon-cpp/pull/310


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org