You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "Tan-JiaLiang (via GitHub)" <gi...@apache.org> on 2024/11/06 08:13:48 UTC
[PR] [core] Support bsi file index [paimon]
Tan-JiaLiang opened a new pull request, #4464:
URL: https://github.com/apache/paimon/pull/4464
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
<!-- What is the purpose of the change -->
Support bit-slice bitmap index, use for numeric accelerate range query and it can be combined with bitmap index.
The comparison algorithm refers to [the bsi module of roaringbitmap](https://github.com/RoaringBitmap/RoaringBitmap/blob/62782af4201d42358176039caad9512a7a29b42b/bsi/src/main/java/org/roaringbitmap/bsi/RoaringBitmapSliceIndex.java#L432) and the paper [Improved Query Performance with Variant Indexes](https://dl.acm.org/doi/pdf/10.1145/253262.253268) in Algorithm 4.2. Also, this [blog ](https://medium.com/pulseque/bit-map-and-bit-slice-indexing-f6c44a61628d)can help us understand BSI better.
### Tests
<!-- List UT and IT cases to verify this change -->
org.apache.paimon.fileindex.bsi.BitSliceIndexBitmapFileIndexTest
org.apache.paimon.utils.BitSliceIndexRoaringBitmapTest
org.apache.paimon.table.AppendOnlyFileStoreTableTest#testBSIAndBitmapIndexInMemory
org.apache.paimon.table.AppendOnlyFileStoreTableTest#testBSIAndBitmapIndexInDisk
org.apache.paimon.spark.SparkFileIndexITCase#testReadWriteTableWithBitSliceIndex
### API and Format
<!-- Does this change affect API or storage format -->
Nothing.
### Documentation
<!-- Does this change introduce a new feature -->
docs/content/concepts/spec/fileindex.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support bsi file index [paimon]
Posted by "hang8929201 (via GitHub)" <gi...@apache.org>.
hang8929201 commented on code in PR #4464:
URL: https://github.com/apache/paimon/pull/4464#discussion_r1833774197
##########
paimon-common/src/main/java/org/apache/paimon/fileindex/bitmap/BitmapIndexResultLazy.java:
##########
@@ -38,9 +38,13 @@ public boolean remain() {
public FileIndexResult and(FileIndexResult fileIndexResult) {
if (fileIndexResult instanceof BitmapIndexResultLazy) {
return new BitmapIndexResultLazy(
- () ->
- RoaringBitmap32.and(
- get(), ((BitmapIndexResultLazy) fileIndexResult).get()));
+ () -> {
+ if (!remain()) {
Review Comment:
This can indeed be optimized, just return SKIP instead of get();
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support bsi file index [paimon]
Posted by "hang8929201 (via GitHub)" <gi...@apache.org>.
hang8929201 commented on code in PR #4464:
URL: https://github.com/apache/paimon/pull/4464#discussion_r1832186993
##########
paimon-common/src/main/java/org/apache/paimon/fileindex/bitmap/BitmapIndexResultLazy.java:
##########
@@ -38,9 +38,13 @@ public boolean remain() {
public FileIndexResult and(FileIndexResult fileIndexResult) {
if (fileIndexResult instanceof BitmapIndexResultLazy) {
return new BitmapIndexResultLazy(
- () ->
- RoaringBitmap32.and(
- get(), ((BitmapIndexResultLazy) fileIndexResult).get()));
+ () -> {
+ if (!remain()) {
Review Comment:
good, but if this index result is skip, the 'and' result can simply be 'SKIP', because there is no negate operation outside, 'SKIP' and/or with bitmap is always accurate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support bsi file index [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #4464:
URL: https://github.com/apache/paimon/pull/4464
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support bsi file index [paimon]
Posted by "Tan-JiaLiang (via GitHub)" <gi...@apache.org>.
Tan-JiaLiang commented on code in PR #4464:
URL: https://github.com/apache/paimon/pull/4464#discussion_r1833833454
##########
paimon-common/src/main/java/org/apache/paimon/fileindex/bitmap/BitmapIndexResultLazy.java:
##########
@@ -38,9 +38,13 @@ public boolean remain() {
public FileIndexResult and(FileIndexResult fileIndexResult) {
if (fileIndexResult instanceof BitmapIndexResultLazy) {
return new BitmapIndexResultLazy(
- () ->
- RoaringBitmap32.and(
- get(), ((BitmapIndexResultLazy) fileIndexResult).get()));
+ () -> {
+ if (!remain()) {
Review Comment:
I take a look the file index framework, it will fast skip when current predicate is fasle. See https://github.com/apache/paimon/blob/32d05f31e1aae54e7d7504fc5f66cbde4a111a91/paimon-common/src/main/java/org/apache/paimon/fileindex/FileIndexPredicate.java#L164
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support bsi file index [paimon]
Posted by "Tan-JiaLiang (via GitHub)" <gi...@apache.org>.
Tan-JiaLiang commented on code in PR #4464:
URL: https://github.com/apache/paimon/pull/4464#discussion_r1833722049
##########
paimon-common/src/main/java/org/apache/paimon/fileindex/bitmap/BitmapIndexResultLazy.java:
##########
@@ -38,9 +38,13 @@ public boolean remain() {
public FileIndexResult and(FileIndexResult fileIndexResult) {
if (fileIndexResult instanceof BitmapIndexResultLazy) {
return new BitmapIndexResultLazy(
- () ->
- RoaringBitmap32.and(
- get(), ((BitmapIndexResultLazy) fileIndexResult).get()));
+ () -> {
+ if (!remain()) {
Review Comment:
thanks, rollbacked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org