You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "haohuaijin (via GitHub)" <gi...@apache.org> on 2026/07/25 14:07:01 UTC

[I] Preserve bitmap-backed `RowSelection` in `ParquetAccessPlan` [datafusion]

haohuaijin opened a new issue, #23883:
URL: https://github.com/apache/datafusion/issues/23883

   ### Is your feature request related to a problem or challenge?
   
   [arrow-rs#10141](https://github.com/apache/arrow-rs/pull/10141) allows `RowSelection` to be backed by a `BooleanBuffer`.
   
   However, `ParquetAccessPlan::into_overall_row_selection` currently calls `RowSelection::iter()` and converts each selection into a `Vec<RowSelector>`. This materializes mask-backed selections as RLE and loses the bitmap backing before it reaches the Parquet reader.
   
   This issue is blocked until DataFusion upgrades to an Arrow/Parquet release containing `arrow-rs#10141`.
   
   ### Describe the solution you'd like
   
   - Validate selection length using `row_count() + skipped_row_count()` instead of `iter()`.
   - When all row-group selections are mask-backed, concatenate their `BooleanBuffer`s directly.
   - Represent scanned row groups as all-set bitmap ranges and omit skipped row groups.
   - Keep the existing selector path as the fallback for selector-backed or mixed inputs.
   
   ### Describe alternatives you've considered
   
   Converting selectors back to a bitmap in the Parquet reader does not help because the large intermediate `Vec<RowSelector>` has already been allocated.
   
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Preserve bitmap-backed `RowSelection` in `ParquetAccessPlan` [datafusion]

Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on issue #23883:
URL: https://github.com/apache/datafusion/issues/23883#issuecomment-5383965599

   update: waiting for https://github.com/apache/arrow-rs/issues/10624, then rethink this part.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Preserve bitmap-backed `RowSelection` in `ParquetAccessPlan` [datafusion]

Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on issue #23883:
URL: https://github.com/apache/datafusion/issues/23883#issuecomment-5078781905

   take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Preserve bitmap-backed `RowSelection` in `ParquetAccessPlan` [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #23883:
URL: https://github.com/apache/datafusion/issues/23883#issuecomment-5084183099

   This is great. FYI I plan to make a arrow-rs release later this week so hopefully the bitmap backed selection will be available bynext week


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Preserve bitmap-backed `RowSelection` in `ParquetAccessPlan` [datafusion]

Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on issue #23883:
URL: https://github.com/apache/datafusion/issues/23883#issuecomment-5087228132

   > This is great. FYI I plan to make a arrow-rs release later this week so hopefully the bitmap backed selection will be available bynext week
   
   Great 🚀


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Preserve bitmap-backed `RowSelection` in `ParquetAccessPlan` [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #23883: Preserve bitmap-backed `RowSelection` in `ParquetAccessPlan`
URL: https://github.com/apache/datafusion/issues/23883


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org