You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/11/14 16:53:26 UTC
[I] [Parquet] Better heuristics to pick between RowSelection and Mask filter representation [arrow-rs]
alamb opened a new issue, #8846:
URL: https://github.com/apache/arrow-rs/issues/8846
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
After the great work from @hhhizzz in https://github.com/apache/arrow-rs/pull/8733, we will (finally) have the ability to use a Bitmask filter representation when applying filters *during* Parquet decode.
At the moment, the code relies on a simple threshold strategy to pick between representations
https://github.com/apache/arrow-rs/blob/911331aafa13f5e230440cf5d02feb245985c64e/parquet/src/arrow/arrow_reader/read_plan.rs#L107-L130
However, as @hhhizzz mentions in https://github.com/apache/arrow-rs/pull/8733#discussion_r2506343981
> Yes, my charts indicate that there are many rules for setting the RowSelectionStrategy, like the column type, column count, string length, and their combinations... We can create tickets and collaborate on improving these over time.
**Describe the solution you'd like**
I would like better heuristics for selecting between the stratgies
**Describe alternatives you've considered**
@hhhizzz has some good suggestions, and the charts from https://github.com/apache/arrow-rs/pull/8733#issuecomment-3468441165 offer some good ideas:
> For how I get the the average length to use the mask, here's some statistic, you can checkout to (https://github.com/hhhizzz/arrow-rs/tree/rowselectionempty-charts) and run `python3 dev/row_selection_analysis.py` on your local machine, this is the results on my x86 PC:
> # One column `int32`, different distribution type:
> <img width="768" height="576" alt="scenario-dense80-dense80" src="https://github.com/user-attachments/assets/30582512-ba85-43d7-9c20-02632020a08b" />
> <img width="768" height="576" alt="scenario-sparse20-sparse20" src="https://github.com/user-attachments/assets/dcbcf7ff-2a4a-4039-bcf4-3d821f8d39a1" />
> <img width="768" height="576" alt="scenario-spread50-spread50" src="https://github.com/user-attachments/assets/ff585f4e-959a-40cf-a580-911720c5fef8" />
> <img width="768" height="576" alt="scenario-uniform50-uniform50" src="https://github.com/user-attachments/assets/a581877a-15e4-445c-9b04-f4b9d25a4749" />
>
> # Different column type:
> <img width="768" height="576" alt="dtype-int32-uniform50" src="https://github.com/user-attachments/assets/732befae-e6d2-4d01-9c88-f7d1e0a31269" />
> <img width="768" height="576" alt="dtype-utf8view-uniform50" src="https://github.com/user-attachments/assets/d097c6e2-3099-4001-8ec7-3e6b4ec9be79" />
> <img width="768" height="576" alt="dtype-float64-uniform50" src="https://github.com/user-attachments/assets/ca738a33-750a-472e-ae65-627a4e8387ef" />
>
> # Different column counts:
> <img width="768" height="576" alt="columns-C02-uniform50" src="https://github.com/user-attachments/assets/d0230ed3-a80e-470c-b9f8-72d012486430" />
> <img width="768" height="576" alt="columns-C04-uniform50" src="https://github.com/user-attachments/assets/7f5a252d-5ce3-44b8-a7af-c98bc9426dd5" />
> <img width="768" height="576" alt="columns-C08-uniform50" src="https://github.com/user-attachments/assets/ff658cec-6f6f-41ea-87fd-599096ab6d41" />
> <img width="768" height="576" alt="columns-C16-uniform50" src="https://github.com/user-attachments/assets/d903cf45-0336-4fdd-a1f0-c3b6bd331bdd" />
> <img width="768" height="576" alt="columns-C32-uniform50" src="https://github.com/user-attachments/assets/4bde5045-4391-446e-a678-675109b5e193" />
**Additional context**
<!--
Add any other context or screenshots about the feature request here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Parquet] Adaptively to pick between RowSelection and Mask filter representation [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on issue #8846:
URL: https://github.com/apache/arrow-rs/issues/8846#issuecomment-5235308225
Related data point from a completed selected-decoding investigation ([report](https://github.com/hhhizzz/arrow-rs/blob/exp/v21-rle-selected-fill-20260807/experiments/shape-aware-selected-decoding/README.md)): when I pushed a row selection down into dictionary decoding, selector morphology alone did not predict query-level benefit. Two further things had to hold — the optimization had to be *reachable* under a correctness-preserving admission rule (0/99 TPC-DS and 4/42 ClickBench queries qualified), and the affected work had to be a meaningful share of query CPU (3.0–3.6% of suite runtime for the covered queries). This seems consistent with the multi-level adaptivity direction in this issue: shape is a necessary input to the decision but not a sufficient one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Parquet] Better heuristics to pick between RowSelection and Mask filter representation [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8846:
URL: https://github.com/apache/arrow-rs/issues/8846#issuecomment-4769814523
Context from @hhhizzz in https://github.com/apache/arrow-rs/pull/9956#issuecomment-4769646397
I think the best model is **multi-level adaptivity**.
DataFusion has more **high-level context**, such as query semantics, file / row-group statistics, projected columns across the whole scan, and cross-file predicate selectivity. That makes it a good place to decide whether a `RowFilter` should be used at all, or to override the default behavior when it has stronger information.
However, the part this PR is trying to handle is **lower-level**. The actual shape of the final `RowSelection` is only known after the Parquet reader has decoded the predicate columns for a specific row group and evaluated the `RowFilter`. For example, whether the result is clustered, fragmented, mostly selected, or sparse is not available to DataFusion at logical / physical planning time without duplicating Parquet reader internals.
So I see the responsibilities as:
- **DataFusion** decides the higher-level scan / filtering strategy.
- **arrow-rs** decides how to execute a row filter once it sees the actual row-group-level selection shape and loaded page ranges.
The explicit policy APIs are important because they still allow DataFusion to override the automatic choice. But I think the default `Auto` behavior belongs in arrow-rs, because the information it uses is produced inside the Parquet reader during execution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Parquet] Better heuristics to pick between RowSelection and Mask filter representation [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8846:
URL: https://github.com/apache/arrow-rs/issues/8846#issuecomment-4769809793
Context from @adriangb in https://github.com/apache/arrow-rs/pull/9956#issuecomment-4769463376:
> I think we had been planning to put more of the adaptivity at a higher level (DataFusion specifically) as it has more information about things like statistics, and cross file predicate selectivity.
The pattern that seems to work well is:
- Seed with cheap heuristics
- Adapt once runtime information is available
- Have multiple levels of adaptivity
That last case specifically applies here:
- Datafusion can decide what even is a row filter or not
- arrow-rs can optimize how it applies those row filters (e.g. when it pays the price of flattening / rolling over a mask, etc.) at a more fine-grained level
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org