Re: slow solr facet processing

[Ere Maijala <er...@helsinki.fi>]

Hi Everyone,

This is a followup on the discussion from September 2017. Since then 
I've spent a lot of time gathering a better understanding on docValues 
compared to UIF and other stuff related to Solr performance. Here's a 
summary of the results based on my real-world experience:

1. Making sure Solr needs as little Java heap as possible is crucial.

2. UIF requires a lot of Java heap. With a larger index it becomes 
impractical, since Java GC can't easily keep up with the heaps required.

3. UIF is really fast, but only after serious warmup. DocValues work 
better if the index is updated regularly, since same level of warmup is 
not needed.

4. DocValues, taking advantage of memory-mapped files, don't have the 
above problem, and after moving to all-docValues we have been able to 
reduce the Java heap from 31G to 6G. This is pretty significant, since 
it means we don't have to deal with long GC pauses.

5. Make sure docValues are enabled also for all fields used for sorting. 
This helps avoid spending memory on field cache. Without docValues we 
could easily have 2 GB of field cache entries.

5. It seems that having docValues for the id field is useful too. For 
now stored needs to remain true too (see 
https://issues.apache.org/jira/browse/SOLR-10816).

6. Sharding the index helps faceting with docValues perform more work in 
parallel and results in a lot better performance. This doesn't seem to 
negatively affect the overall performance (at least enough to be 
perceived), and it seems that splitting our index to three shards 
resulted in speedup that's better than previous performance divided by 
three. There is a caveat [1], though.

7. In many cases fields that have docValues enabled can be switched from 
stored="true" to stored="false" since Solr can fetch the contents from 
docValues. A notable exception is multivalued fields where the order of 
the values is important. This means that enabling docValues doesn't add 
to the index size significantly.

8. Different replica types available in Solr 7 are really useful in 
reducing the CPU time spent indexing records. I'd still like to have a 
way to have PULL replicas with NRT replicas so that only the PULL 
replicas handle search queries.

9. Lastly, a lot can be done on the application level. For instance in 
our case many users don't care about facets or only use a couple of 
them, so we fetch them asynchronously as needed and collapse most by 
default without fetching them at all. This lowers the server load 
significantly (I'll work on contributing the option to upstream VuFind).


I hope this helps others make informed choices.

--Ere


[1] Care must be taken to avoid requests that cause Solr to fetch a lot 
of rows at once from each shard, since that blows up the memory usage 
wreaking havoc in Solr. One particular case that, at first sight, 
doesn't look too dangerous, is deep paging without a cursor (Yonik has a 
good explanation of this at http://yonik.com/solr/paging-and-deep-paging/).

Re: slow solr facet processing

[Erick Erickson <er...@gmail.com>]

Ere:

This is an excellent summary, it conforms to what I think I know, it's
always nice to see confirmation!

I'd add two small enhancements. Your point 5 mentions sorting. The same
consideration is true for grouping and faceting as well. What all three
have in common is that they answer the question "for document X, what is
the value(s) of field Y?" which inverted indexes don't handle efficiently.

The second enhancement is an additional caveat for point 7. Not only are
multivalued fields returned in sorted order, if multiple identical values
exist they are collapsed into one (it's a sorted set). So input of
4,5,3,3,3 would return just 3,4,5.

About point 8. I'm thinking lately that a better option would be to use
some of the enhanced metrics being put in place for autoscaling to
intelligently route queries (and sub-queries) to nodes that currently have
the most unused (CPU?) capacity. The problem I have with routing only to
PULL or TLOG replicas is that your NRT replicas can sit idle; it's a crude
hammer. Consider periodic bulk indexing, i.e. your index changes once every
hour for 10 minutes. Your NRT nodes would sit idle for 50 minutes/hour. I'd
rather see effort there than specifying a subset of nodes as search nodes,
WDYT?

Again, thanks for taking the time to write up your summary!
Erick


On Fri, Jan 5, 2018 at 6:15 AM, Ere Maijala <er...@helsinki.fi> wrote:

> Hi Everyone,
>
> This is a followup on the discussion from September 2017. Since then I've
> spent a lot of time gathering a better understanding on docValues compared
> to UIF and other stuff related to Solr performance. Here's a summary of the
> results based on my real-world experience:
>
> 1. Making sure Solr needs as little Java heap as possible is crucial.
>
> 2. UIF requires a lot of Java heap. With a larger index it becomes
> impractical, since Java GC can't easily keep up with the heaps required.
>
> 3. UIF is really fast, but only after serious warmup. DocValues work
> better if the index is updated regularly, since same level of warmup is not
> needed.
>
> 4. DocValues, taking advantage of memory-mapped files, don't have the
> above problem, and after moving to all-docValues we have been able to
> reduce the Java heap from 31G to 6G. This is pretty significant, since it
> means we don't have to deal with long GC pauses.
>
> 5. Make sure docValues are enabled also for all fields used for sorting.
> This helps avoid spending memory on field cache. Without docValues we could
> easily have 2 GB of field cache entries.
>
> 5. It seems that having docValues for the id field is useful too. For now
> stored needs to remain true too (see https://issues.apache.org/jira
> /browse/SOLR-10816).
>
> 6. Sharding the index helps faceting with docValues perform more work in
> parallel and results in a lot better performance. This doesn't seem to
> negatively affect the overall performance (at least enough to be
> perceived), and it seems that splitting our index to three shards resulted
> in speedup that's better than previous performance divided by three. There
> is a caveat [1], though.
>
> 7. In many cases fields that have docValues enabled can be switched from
> stored="true" to stored="false" since Solr can fetch the contents from
> docValues. A notable exception is multivalued fields where the order of the
> values is important. This means that enabling docValues doesn't add to the
> index size significantly.
>
> 8. Different replica types available in Solr 7 are really useful in
> reducing the CPU time spent indexing records. I'd still like to have a way
> to have PULL replicas with NRT replicas so that only the PULL replicas
> handle search queries.
>
> 9. Lastly, a lot can be done on the application level. For instance in our
> case many users don't care about facets or only use a couple of them, so we
> fetch them asynchronously as needed and collapse most by default without
> fetching them at all. This lowers the server load significantly (I'll work
> on contributing the option to upstream VuFind).
>
>
> I hope this helps others make informed choices.
>
> --Ere
>
>
> [1] Care must be taken to avoid requests that cause Solr to fetch a lot of
> rows at once from each shard, since that blows up the memory usage wreaking
> havoc in Solr. One particular case that, at first sight, doesn't look too
> dangerous, is deep paging without a cursor (Yonik has a good explanation of
> this at http://yonik.com/solr/paging-and-deep-paging/).
>