distributed faceting: refineFacets()

[Dmitry Kan <dm...@gmail.com>]

Hello list,

In a distributed faceting search scenario, does SOLR frontend (the merger)
expect shard facets to be pre-sorted (by count or by index) ? If so, when
merging the results, is there some smart strategy for combining the shards
results into a final sorted list? Can someone explain what refineFacets()
method does? Removing the call to it, didn't change the SOLR merger
behaviour, but what I'm not yet sure is what was the performance impact.

Our use case is 16 shards with sizes varying from 30G to 130G. When
searching across the entire shard farm, we have noticed that the merger
SOLR takes 10 seconds on avg per request. It could be that the merger is
simply waiting for *fat* shards to answer. What we have found out is that
each shard would answer within 1-4 seconds. That on average leaves about 6
seconds for the merging part.

We also know in advance, that shards respond with non-intersecting hits.
That practically means, that the merger should simply "concatenate" the
shard results into one list (automatically pre-sorted by design).

Can something be improved in the SOLR merger facet logic here? Should we
look at something else as well?

-- 
Thanks,

Dmitry Kan