[jira] [Comment Edited] (SOLR-13890) Add postfilter support to {!terms} queries

["Jason Gerlowski (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988801#comment-16988801 ] 

Jason Gerlowski edited comment on SOLR-13890 at 12/5/19 1:21 PM:
-----------------------------------------------------------------

bq. I haven't investigated how feasible it is but I wonder if Solr even needs PostFilter given TwoPhaseIterator exists. For another day.

It's a good question.  I'd imagine that two things are necessary if we wanted to replace Solr's postfilter:
# We'd need to make sure that TPI implementations provide the same performance gains as postfilter ones.  I wouldn't have doubted this previously.  But knowing that DVTQ already has TPI, and recalling the gains we saw with our postfilter in (unpublished) perf tests is enough to plant a seed of doubt for me.
# We'd need to see whether users are fine ceding control of when this special execution mode is triggered.  TPI being heuristic-triggered seems double-edged to me if a user finds themselves fighting those heuristics on a query they know would benefit from TPI.  Though maybe there's an override flag I'm just not aware of that forces TPI to be used/ignored.

That said, "for another day" for sure.


was (Author: gerlowskija):
bq. I haven't investigated how feasible it is but I wonder if Solr even needs PostFilter given TwoPhaseIterator exists. For another day.

It's a good question.  I'd imagine that two things are necessary if we wanted to replace Solr's postfilter:
# We'd need to make sure that TPI implementations provide the same performance gains as postfilter ones.  I wouldn't have considered this previously.  But knowing that DVTQ already has TPI, and recalling the gains we saw with our postfilter in (unpublished) perf tests is enough to plant a seed of doubt for me.
# We'd need to see whether users are fine ceding control of when this special execution mode is triggered.  TPI being heuristic-triggered seems double-edged to me if a user finds themselves fighting those heuristics on a query they know would benefit from TPI.  Though maybe there's an override flag I'm just not aware of that forces TPI to be used/ignored.

That said, "for another day" for sure.

> Add postfilter support to {!terms} queries
> ------------------------------------------
>
>                 Key: SOLR-13890
>                 URL: https://issues.apache.org/jira/browse/SOLR-13890
>             Project: Solr
>          Issue Type: Improvement
>      Security Level: Public(Default Security Level. Issues are Public) 
>          Components: query parsers
>    Affects Versions: master (9.0)
>            Reporter: Jason Gerlowski
>            Assignee: Jason Gerlowski
>            Priority: Major
>         Attachments: SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch
>
>
> There are some use-cases where it'd be nice if the "terms" qparser created a query that could be run as a postfilter.  Particularly, when users are checking for hundreds or thousands of terms, a postfilter implementation can be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the {{docValuesTermsFilter}} "method".  Postfilter creation can use a SortedSetDocValues object to populate a DV bitset with the "terms" being checked for.  Each document run through the post-filter can look at their doc-values for the field in question and check them efficiently against the constructed bitset.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org