[GitHub] [incubator-druid] clintropolis opened a new pull request #8209: add mechanism to control filter optimization in historical query processing

[GitBox <gi...@apache.org>]

clintropolis opened a new pull request #8209: add mechanism to control filter optimization in historical query processing
URL: https://github.com/apache/incubator-druid/pull/8209
 
 
   ### Description
   
   This PR introduces the ability for most `DimFilter` implementations to control whether or not `QueryableIndexStorageAdapter` will use a bitmap index while processing queries. I consider this to be laying the groundwork for allowing `QueryableIndexStorageAdapter` the opportunity for more sophisticated decision making when deciding when to use bitmap indexes or not, but the first step of this is to enable easy experimentation so that we can shake out what good "automatic" decisions might be.
   
   This is done by adding a new type, `FilterTuning`, which most `DimFilter` implementations will accept if specified in the filter json, which is supplied to all `Filter` implementations. The `Filter` interface itself has been modified with a few new methods:
   *  `getManualTuning` which is intended to return the `FilterTuning` supplied by the user via the `DimFilter`
   * `computeTuning` which will allow a `Filter` implementation to automatically populate a `FilterTuning` based on a `BitmapIndexSelector`
   * `shouldUseIndex` which `QueryableIndexStorageAdapter` and `FilteringOffset` call now instead of `supportsBitmapIndex` directly, to determine if a bitmap index should be used.
   
   `FilterTuning` currently has 3 properties:
   
   | property | type | description | default |
   | --- | --- | --- | --- |
   | `useIndex`| boolean | allow manually skipping bitmap index even if it exists | true
   | `useIndexMaxCardinalityThreshold` | integer | allow using a bitmap index only if cardinality is _under_ max value| `Integer.MAX_VALUE` |
   | `useIndexMinCardinalityThreshold` | integer | allow using bitmap index only if cardinality is _over_ min value (this is probably useless, but i wanted levers to pull to see what happens). | 0 |
   
   Since this is an advanced feature intended primarily for experimentation, I am leaving it undocumented for now.
   
   <hr>
   
   This PR has:
   - [ ] been self-reviewed.
   - [ ] added documentation for new or modified features or behaviors.
   - [x] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
   - [x] added unit tests or modified existing tests to cover new code paths.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org