You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "pdeaudney (via GitHub)" <gi...@apache.org> on 2026/10/14 02:54:04 UTC
[PR] [feature](be) Add Roaring bitmap candidate filters for resource IDs [doris]
pdeaudney opened a new pull request, #69048:
URL: https://github.com/apache/doris/pull/69048
### What problem does this PR solve?
Issue Number: draft for discussion
Related PR: draft for duscussion
Problem Summary: Queries over large shared datasets need a way to supply a posting list of candidate `resource_id` values at query time. The list may represent facets, categories, access rules, or another application-defined candidate set.
This PR adds `bitmap_from_roaring_base64` for portable Roaring32 and Go-compatible portable Roaring64 input. It also adds early bitmap membership filtering in native scans and an opt-in compressed runtime filter for subquery-produced bitmaps. The exact bitmap semi join remains the final membership check.
The benchmark uses 20 million rows in the selected `org_id` partition, 10 million rows in another partition, and candidate lists of 250,000 to 10 million IDs. In the Iceberg rerun, a 1-million-ID broad-text query averaged 11,225 ms with the default bitmap join and 674 ms with the scan filter. A selective-text query averaged 180 ms and 191 ms respectively. Iceberg still read the same Parquet row groups and bytes, so the measured gain is from avoiding downstream join work.
A Go client also generated portable Roaring64 outside Doris and sent it through the MySQL query interface. Its 1-million-ID Base64 payload was 2.67 MB; the broad-text query averaged 130 ms on native storage and 207 ms on Iceberg from that local client. These are separate query shapes and runs, not a direct comparison with the subquery runtime filter.
The runtime filter is opt-in through `enable_bitmap_subquery_runtime_filter`. Enable it only after FE and BE nodes are upgraded. Distributed transfer, memory use, and remote client latency still need evaluation.
### Release note
Add `bitmap_from_roaring_base64` to decode portable Roaring bitmap input for bitmap queries. Add an opt-in compressed bitmap runtime filter for positive bitmap subqueries.
### Check List (For Author)
- Test:
- [x] Regression test: native and Iceberg bitmap suites, including result checks against ordinary `IN` queries
- [x] Manual test: Go-produced portable Roaring64 at 250,000, 1 million, 5 million, and 10 million candidate IDs on native and Iceberg tables
- [x] Go test and vet for the benchmark client
- Behavior changed:
- [x] Yes. Adds a SQL function and bitmap scan paths; the subquery runtime filter requires an explicit session setting.
- Does this need documentation?
- [x] Yes. The PR includes a Go sample README and benchmark results. User-facing Doris documentation remains to be prepared.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Add Roaring bitmap candidate filters for resource IDs [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69048:
URL: https://github.com/apache/doris/pull/69048#issuecomment-6159829093
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Add Roaring bitmap candidate filters for resource IDs [doris]
Posted by "pdeaudney (via GitHub)" <gi...@apache.org>.
pdeaudney commented on PR #69048:
URL: https://github.com/apache/doris/pull/69048#issuecomment-6159911660
# Comparison against sub-query IN with local data
The fixture had 20 million rows for `org_id = 101` and 10 million for another org. Candidate bitmaps contained 250,000 to 10 million `resource_id` values. Queries also applied `payload LIKE '%needle%'`. Each shape was warmed and timed five times; bitmap and `IN` queries returned the same final count and `SUM(resource_id)`.
**Direct Base64 bitmap versus an ordinary `IN` subquery** (mean query latency, ms):
| Candidate IDs | Native bitmap | Native `IN` | Iceberg bitmap | Iceberg `IN` |
| ---: | ---: | ---: | ---: | ---: |
| 250k | 45.4 | 25.9 | 119.1 | 106.7 |
| 1m | 123.5 | 37.2 | 217.5 | 148.1 |
| 5m | 106.2 | 92.1 | 196.1 | 204.8 |
| 10m | 107.9 | 137.9 | 194.1 | 244.7 |
At 10m IDs, the bitmap query was about 22% faster than `IN` on native storage and 21% faster on Iceberg. At 250k and 1m IDs, `IN` was faster. Bitmap construction happened before the timed bitmap query; the `IN` query read the allowlist table, so these are useful query-shape results rather than a controlled comparison of filter delivery.
**Subquery-produced bitmap on native storage** (mean query latency, ms):
| Candidate IDs | Default bitmap join | Compressed scan runtime filter |
| ---: | ---: | ---: |
| 250k | 97.4 | 54.4 |
| 1m | 102.0 | 73.6 |
| 5m | 122.0 | 81.4 |
| 10m | 125.0 | 94.0 |
At 250k IDs, the compressed filter's scan profile reported 19.75 million rows rejected by bitmap membership, leaving 250,000 rows for later expressions. These were warm, single-host macOS arm64 measurements. The scattered matching IDs did not reduce fact-column bytes read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Add Roaring bitmap candidate filters for resource IDs [doris]
Posted by "pdeaudney (via GitHub)" <gi...@apache.org>.
pdeaudney commented on PR #69048:
URL: https://github.com/apache/doris/pull/69048#issuecomment-6159859131
### Bitmap candidate-filter benchmarks
Setup: one local macOS arm64 FE/BE in Release mode; 20 million rows for the selected org and 10 million for another org. Each query had one warm-up and five timed executions. Counts and SUM(resource_id) were verified.
**Go-produced portable Roaring64:** The client generated the bitmap outside Doris and sent it through the MySQL query interface. Times are local client-to-result means.
| Candidate IDs | Base64 payload | Native, broad text | Iceberg, broad text |
| ---: | ---: | ---: | ---: |
| 250k | 670 KB | 60 ms | 158 ms |
| 1m | 2.67 MB | 130 ms | 207 ms |
| 5m | 3.34 MB | 101 ms | 198 ms |
| 10m | 3.35 MB | 101 ms | 208 ms |
At 1m IDs, Go generation and encoding added about 9 ms, giving a 216 ms generation-plus-query mean on Iceberg. Candidate retrieval and remote network latency were not measured.
**Subquery-produced bitmap on Iceberg:** With broad text, the opt-in scan runtime filter reduced work passed to the bitmap join.
| Candidate IDs | Default join | Scan filter |
| ---: | ---: | ---: |
| 250k | 11,121 ms | 220 ms |
| 1m | 11,225 ms | 674 ms |
| 5m | 11,243 ms | 2,967 ms |
| 10m | 11,277 ms | 5,855 ms |
At 1m IDs with selective text, the scan filter averaged 191 ms versus 180 ms for the default join. Both plans still read all 20 million selected-org rows, all 18 Parquet row groups, and 44.82 MB. The measured gain was from avoiding downstream join work, not reducing Parquet I/O.
The Go constant-bitmap query and subquery runtime-filter query use different plans, so their timings are not a controlled comparison of filter delivery.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org