[jira] [Created] (SOLR-5894) Speed up high-cardinality facets with sparse counters

["Toke Eskildsen (JIRA)" <ji...@apache.org>]

Toke Eskildsen created SOLR-5894:
------------------------------------

             Summary: Speed up high-cardinality facets with sparse counters
                 Key: SOLR-5894
                 URL: https://issues.apache.org/jira/browse/SOLR-5894
             Project: Solr
          Issue Type: Improvement
          Components: SearchComponents - other
    Affects Versions: 4.7, 4.6.1
            Reporter: Toke Eskildsen
            Priority: Minor
             Fix For: 4.6.1


Field based faceting in Solr has two phases: Collecting counts for tags in facets and extracting the requested tags.

The execution time for the collecting phase is approximately linear to the number of hits and the number of references from hits to tags. This phase is not the focus here.

The extraction time scales with the number of unique tags in the search result, but is also heavily influenced by the total number of unique tags in the facet as every counter, 0 or not, is visited by the extractor (at least for count order). For fields with millions of unique tag values this means 10s of milliseconds added to the minimum response time (see https://sbdevel.wordpress.com/2014/03/18/sparse-facet-counting-on-a-real-index/ for a test on a corpus with 7M unique values in the facet).

The extractor needs to visit every counter due to the current counter structure being a plain int-array of size #unique_tags. Switching to a sparse structure, where only the tag counters > 0 are visited, makes the extraction time linear to the number of unique tags in the result set.

Unfortunately the number of unique tags in the result set is unknown at collect time, so it is not possible to reliably select sparse counting vs. full counting up front. Luckily there exists solutions for sparse sets that has the property of switching to non-sparse-mode without a switch-penalty, when the sparse-threshold is exceeded (see http://programmingpraxis.com/2012/03/09/sparse-sets/ for an example). This JIRA aims to implement this functionality in Solr (a proof of concept patch will be provided shortly).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org