Maintain SortedSetDocValuesReaderState in Cache

[Chitra R <ch...@gmail.com>]

Hi,

      We are going to implement faceted search (for both string & numeric)
based on DocValuesField. For better performance, we decided to put the
SortedSetDocValuesReaderState in cache ie

*During first faceted search - new DocValuesReaderState will be opened and
"$facets" values will be loaded from index and then we put that state in
cache using ConcurrentHashMap

*For next faceted search - instead of opening new DocValuesReaderState, we
served the DocValuesReaderState from the ConcurrentHashMap.

we will maintain the DocValuesReaderState in concurrentHashMap till 5
minutes. After 5 minutes, an Old DocValuesReaderState will be removed from
the ConcurrentHashMap.

And for every search calls,

DocValuesReaderState will be taken from cache like

SortedSetDocValuesReaderStatereaderState  oldReaderState = new
DefaultSortedSetDocValuesReaderState(oldIndexSearcher.getIndexReader());

readerStateCacheMap.put(key, oldReaderState);

DocValuesReaderState will be maintained in  readerStateCacheMap till 5
minutes

**** TopDocs will be computed using new IndexSearcher as
FacetsCollector.search(newIndexSearcher, drillDownQuery, boolFilter,
noofdocs, facetsCollector);

****After computing TopDocs, facets will be estimated from using old
SortedSetDocValuesReaderState ad
SortedSetDocValuesFacetCounts facetCount = new
SortedSetDocValuesFacetCounts(oldReaderState, facetsCollector);


1.    So, is there any chance to get conflicts during topDocs search or
facets estimation (because of old indexSearcher in DocValuesReaderState and
newIndexSearcher in topDocs computation)?
2.   How can we maintain SortedSetDocValuesReaderState in cache?


Kindly post your suggestions.

Thanks & Regards,
Chitra

Re: Maintain SortedSetDocValuesReaderState in Cache

[Chitra R <ch...@gmail.com>]

Hi,
      How can I improve the performance of SortedSetDocValuesReaderState
which is responsible for computing '$facets' global ordinals at search time?

Is there any better way to warm up this state instead of putting it in
cache?

Any help is much appreciated.

Thanks & Regards,
Chitra


On Fri, Feb 3, 2017 at 11:32 AM, Chitra R <ch...@gmail.com> wrote:

> Hi,
>
>       We are going to implement faceted search (for both string & numeric)
> based on DocValuesField. For better performance, we decided to put the
> SortedSetDocValuesReaderState in cache ie
>
> *During first faceted search - new DocValuesReaderState will be opened
> and "$facets" values will be loaded from index and then we put that state
> in cache using ConcurrentHashMap
>
> *For next faceted search - instead of opening new DocValuesReaderState,
> we served the DocValuesReaderState from the ConcurrentHashMap.
>
> we will maintain the DocValuesReaderState in concurrentHashMap till 5
> minutes. After 5 minutes, an Old DocValuesReaderState will be removed from
> the ConcurrentHashMap.
>
> And for every search calls,
>
> DocValuesReaderState will be taken from cache like
>
> SortedSetDocValuesReaderStatereaderState  oldReaderState = new
> DefaultSortedSetDocValuesReaderState(oldIndexSearcher.getIndexReader());
>
> readerStateCacheMap.put(key, oldReaderState);
>
> DocValuesReaderState will be maintained in  readerStateCacheMap till 5
> minutes
>
> **** TopDocs will be computed using new IndexSearcher as
> FacetsCollector.search(newIndexSearcher, drillDownQuery, boolFilter,
> noofdocs, facetsCollector);
>
> ****After computing TopDocs, facets will be estimated from using old
> SortedSetDocValuesReaderState ad
> SortedSetDocValuesFacetCounts facetCount = new
> SortedSetDocValuesFacetCounts(oldReaderState, facetsCollector);
>
>
> 1.    So, is there any chance to get conflicts during topDocs search or
> facets estimation (because of old indexSearcher in DocValuesReaderState and
> newIndexSearcher in topDocs computation)?
> 2.   How can we maintain SortedSetDocValuesReaderState in cache?
>
>
> Kindly post your suggestions.
>
> Thanks & Regards,
> Chitra
>
>