You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "ArnavBalyan (via GitHub)" <gi...@apache.org> on 2026/05/27 08:47:47 UTC
[PR] [POC] Add Support for Bloom Filter for Mosaic [paimon-mosaic]
ArnavBalyan opened a new pull request, #47:
URL: https://github.com/apache/paimon-mosaic/pull/47
- Add bloom filter support, today all scans/point lookups pushdown ont he reader force a rowgroup scan.
- Add bloom filter to per column chunk, which can be used to check for presence of data in the col chunk.
- There are 2 main additions:
1) Each column chunk gets it's own bloom filter
2) Each row group gets additional metadata on the location of the bloom filter.
- Benchmark : Nyc Taxi dataset (https://www.kaggle.com/datasets/elemento/nyc-yellow-taxi-trip-data)
- File size: No bloom: 51.10 MiB, With bloom: 56.85 MiB
- Small bench was created to test the bytes skipped on a sample query:
- Query IO without bloom: 51.10 MiB
- Query I/O with bloom: 3.19 KB (16000x reduction for absent value lookup)
Bloom filter structure:
<img width="1404" height="1075" alt="image" src="https://github.com/user-attachments/assets/ab5181d2-15c2-4ae1-8c34-d393eb95d78a" />
<br>
<br>
<br>
<br>
Changes in the file format:
<img width="2040" height="1777" alt="image" src="https://github.com/user-attachments/assets/de557724-381b-483e-88ee-8f92b0a4df2b" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Support for Bloom Filter for mosaic files [paimon-mosaic]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed pull request #47: Add Support for Bloom Filter for mosaic files
URL: https://github.com/apache/paimon-mosaic/pull/47
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Support for Bloom Filter for mosaic files [paimon-mosaic]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #47:
URL: https://github.com/apache/paimon-mosaic/pull/47#issuecomment-4697155973
In the Paimon ecosystem, we recommend the file index of a standalone Bloom filter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Support for Bloom Filter for mosaic files [paimon-mosaic]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on PR #47:
URL: https://github.com/apache/paimon-mosaic/pull/47#issuecomment-4587083302
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Support for Bloom Filter for mosaic files [paimon-mosaic]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #47:
URL: https://github.com/apache/paimon-mosaic/pull/47#issuecomment-4697157463
Another question is: Is this format compatible? Otherwise, it needs to be placed in V2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [POC] Add Support for Bloom Filter for Mosaic [paimon-mosaic]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #47:
URL: https://github.com/apache/paimon-mosaic/pull/47#issuecomment-4552974513
cc @JingsongLi @XiaoHongbo-Hope ty
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Support for Bloom Filter for mosaic files [paimon-mosaic]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #47:
URL: https://github.com/apache/paimon-mosaic/pull/47#issuecomment-4588989129
Thanks a lot! cc @JingsongLi could you also pls review when time permits
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Support for Bloom Filter for mosaic files [paimon-mosaic]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #47:
URL: https://github.com/apache/paimon-mosaic/pull/47#issuecomment-4564503041
Fixed format issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Support for Bloom Filter for mosaic files [paimon-mosaic]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #47:
URL: https://github.com/apache/paimon-mosaic/pull/47#issuecomment-4586757039
cc @JingsongLi @XiaoHongbo-Hope gentle reminder could you pls take a look when possible thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Support for Bloom Filter for mosaic files [paimon-mosaic]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on PR #47:
URL: https://github.com/apache/paimon-mosaic/pull/47#issuecomment-4586760295
> cc @JingsongLi @XiaoHongbo-Hope gentle reminder could you pls take a look when possible thanks!
sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Support for Bloom Filter for mosaic files [paimon-mosaic]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #47:
URL: https://github.com/apache/paimon-mosaic/pull/47#issuecomment-5367661652
I don't think there's a need to support Bloom filters right now; they are too heavy—let's save that for later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org