[GitHub] [druid] clintropolis commented on a change in pull request #9647: document useFilterCNF query context parameter

[GitBox <gi...@apache.org>]

clintropolis commented on a change in pull request #9647: document useFilterCNF query context parameter
URL: https://github.com/apache/druid/pull/9647#discussion_r405999848
 
 

 ##########
 File path: docs/querying/query-context.md
 ##########
 @@ -45,6 +45,7 @@ The query context is used for various query configuration parameters. The follow
 |parallelMergeParallelism|`druid.processing.merge.pool.parallelism`|Maximum number of parallel threads to use for parallel result merging on the Broker. See [Broker configuration](../configuration/index.html#broker) for more details.|
 |parallelMergeInitialYieldRows|`druid.processing.merge.task.initialYieldNumRows`|Number of rows to yield per ForkJoinPool merge task for parallel result merging on the Broker, before forking off a new task to continue merging sequences. See [Broker configuration](../configuration/index.html#broker) for more details.|
 |parallelMergeSmallBatchRows|`druid.processing.merge.task.smallBatchNumRows`|Size of result batches to operate on in ForkJoinPool merge tasks for parallel result merging on the Broker. See [Broker configuration](../configuration/index.html#broker) for more details.|
+|useFilterCNF|`false`| If true, Druid will attempt to convert the query filter to Conjunctive Normal Form (CNF). During query processing, columns can be pre-filtered by intersecting the bitmap indexes of all values that match the eligible filters, often greatly reducing the raw number of rows which need to be scanned. But this effect only happens for the top level filter, or individual clauses of a top level 'and' filter. As such, filters in CNF potentially have a higher chance to utilize a large amount of bitmap indexes on string columns during pre-filtering. However, this setting should be used with great caution, as it can sometimes have a negative effect on performance, and in some cases, the act of computing CNF of a filter can be expensive. We recommend hand tuning your filters to produce an optimal form if possible, or at least verifying through experimentation that using this parameter actually improves your query performance with no ill-effects.|
 
 Review comment:
   Hmm, I think how filters work with query processing and how mechanically filters are split into pre and post filters should be documented _somewhere_, but I don't think this setting is quite the correct avenue. Additionally, since which filters can and can't use bitmaps isn't exactly documented anywhere, I'm not sure how much the examples would help. 
   
   If we added this general description of query processing and how filters are involved could link to this setting, and also could link to documentation we would add for the `filterTuning` added in #8209, as ways the user can help influence how filter processing behaves. Maybe `segments.md` would be an appropriate place since it mentions bitmaps and their role in filtering, or `segment-optimization.md` since it involves how to tune segment sizes? Or perhaps we need an `advanced-tuning.md` to put this and other stuff that users shouldn't really mess with unless they are prepared to roll up their sleeves and experimentally verify the settings to fine tune to their workload?
   
   Should this be part of this PR, or done as a follow-up? It sort of blows up the scope a bit of what I was looking to do as part of this PR, but it also seems useful so I'm fine either way.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org