[PR] [core] support file index filter push down [paimon]

["Tan-JiaLiang (via GitHub)" <gi...@apache.org>]

Tan-JiaLiang opened a new pull request, #4684:
URL: https://github.com/apache/paimon/pull/4684

   <!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
   
   ### Purpose
   
   <!-- Linking this pull request to the issue -->
   Linked issue: #4530
   
   The filter will push down if the index matches the predicate, if the index does not exist at runtime, it will fallback to using the paimon predicate filter.
   
   For now, we support bloom-filter/bitmap/bsi index, and the bitmap/bsi index supports filter push down and they will use `ApplyBitmapIndexRecordReader` to filter by position.
   
   <!-- What is the purpose of the change -->
   
   ### Tests
   
   <!-- List UT and IT cases to verify this change -->
   org.apache.paimon.table.AppendOnlyFileStoreTableTest#testFileIndexFilterPushDownFallback
   org.apache.paimon.table.PrimaryKeyFileStoreTableTest#testDeletionVectorsWithFileIndexFilterPushDownFallback
   org.apache.paimon.flink.source.FilterPushDownITCase#testPartitionAndIndexConditionConsuming
   
   ### API and Format
   
   <!-- Does this change affect API or storage format -->
   No
   
   ### Documentation
   
   <!-- Does this change introduce a new feature -->
   No


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [core] Support file index filter push down [paimon]

["Tan-JiaLiang (via GitHub)" <gi...@apache.org>]

Tan-JiaLiang commented on PR #4684:
URL: https://github.com/apache/paimon/pull/4684#issuecomment-2537908380

   I've found that the performance of the file index filter is not as good as I expected, so I'm marking it as a draft for now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [core] Support file index filter push down [paimon]

["Tan-JiaLiang (via GitHub)" <gi...@apache.org>]

Tan-JiaLiang closed pull request #4684: [core] Support file index filter push down
URL: https://github.com/apache/paimon/pull/4684


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org