You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "dwsmith1983 (via GitHub)" <gi...@apache.org> on 2026/09/19 13:09:27 UTC

[I] Nothing guards page skipping in the native Iceberg scan [datafusion-comet]

dwsmith1983 opened a new issue, #6039:
URL: https://github.com/apache/datafusion-comet/issues/6039

   ### What is the problem the feature request solves?
   
   The native Iceberg scan skips Parquet pages: `IcebergScanExec` enables iceberg-rust's row selection and passes the residual predicate, so iceberg-rust loads the page index and builds a page-level row selection. Measured locally on one data file with one row group and 158 `id` pages, `WHERE id BETWEEN 1000 AND 1100` reads about 550 KB of a 21 MB file.
   
   Nothing in the test suite would notice if that stopped. The behaviour hinges on one builder flag in the scan operator and on iceberg-rust internals that change with every pin bump, and `CometIcebergNativeScanExec` reports no page-index counters (iceberg-rust's scan metrics expose only bytes read), so a regression would show up only as slower queries.
   
   ### Describe the potential solution
   
   A regression test in `CometIcebergNativeSuite` that writes a single-row-group, many-page table, verifies that layout from the footer so the fixture cannot degrade into a shape where file-level or row-group pruning explains the savings, and asserts that a narrow range reads a small fraction of the full read's `bytes_scanned`.
   
   ### Additional context
   
   Page-index counters on the Iceberg node, matching the Parquet scan's `page_index_pages_pruned`, need iceberg-rust to expose them first. Related: #6026.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Nothing guards page skipping in the native Iceberg scan [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #6039: Nothing guards page skipping in the native Iceberg scan
URL: https://github.com/apache/datafusion-comet/issues/6039


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org