Searchers, threads and performance

[Andy Lester <an...@petdance.com>]

We're getting close to deploying our Solr search solution, and we're doing performance testing, and we've run into some questions and concerns.

Our number one problem: Doing a commit from loading records, which can happen throughout the day, makes all queries stop for 5-7 seconds.  This is a showstopper for deployment.

Here's what we've observed: Upon commit, Solr finishes processing queries in flight, starts up a new searcher, warms it, shuts down the old searcher and puts the new searcher into effect. Does the old searcher stop taking requests before the new searcher is warmed or after? How wide is the window of time wherein Solr is not serving requests?  For us, it's about five seconds and we need to drop that dramatically.  In general, what is the difference between accepting the delay of waiting for warming vs. accepting the delay of running useColdSearcher=true?

Is there any such thing as/any sense in running more than one searcher in our scenario?  What are the benefits of multiple searchers?  Erik Erikson posts in 2012: "Unless you have warming happening, there should only be a single searcher open at any given time." Except: "If your queries run across several commits you'll get multiple searchers open." Not sure if this is a general observation, or specific to the particular poster's situation.

Finally, what do people mean when they blog that they have Solr set up for "n threads"? Is that the same thing as saying that Solr can be processing n requests simultaneously?

Thanks for any insight or even links to relevant pages.  We've been Googling all over and haven't found answers to the above.

Thanks,
xoa

--
Andy Lester => andy@petdance.com => www.petdance.com => AIM:petdance

Re: Searchers, threads and performance

[Mikhail Khludnev <mk...@griddynamics.com>]

Andy,

Solr is supposed to serve requests by old searcher for a while. If the
pause lasts few seconds you can take a thread dump and see clear what it
waits for.
Just a guess: if you have many threads configured in servlet container pool
and push high load then warming can significantly impact your search
latency - try to limit acceptable load by reducing number of concurrent
requests.
What's cpu utilization, and io stats during in your test? do you watch GC
log, for me GC spike is higly probable than a "warming" attack.   Are you
sure that you use MMapDirectory on which OS?
Once again:
- thread dump?
- io/vm-stat dump?
- gclog?
- thread pool size at servlet container config?
- directory impl and os?


On Tue, Nov 13, 2012 at 7:20 PM, Andy Lester <an...@petdance.com> wrote:

> We're getting close to deploying our Solr search solution, and we're doing
> performance testing, and we've run into some questions and concerns.
>
> Our number one problem: Doing a commit from loading records, which can
> happen throughout the day, makes all queries stop for 5-7 seconds.  This is
> a showstopper for deployment.
>
> Here's what we've observed: Upon commit, Solr finishes processing queries
> in flight, starts up a new searcher, warms it, shuts down the old searcher
> and puts the new searcher into effect. Does the old searcher stop taking
> requests before the new searcher is warmed or after? How wide is the window
> of time wherein Solr is not serving requests?  For us, it's about five
> seconds and we need to drop that dramatically.  In general, what is the
> difference between accepting the delay of waiting for warming vs. accepting
> the delay of running useColdSearcher=true?
>
> Is there any such thing as/any sense in running more than one searcher in
> our scenario?  What are the benefits of multiple searchers?  Erik Erikson
> posts in 2012: "Unless you have warming happening, there should only be a
> single searcher open at any given time." Except: "If your queries run
> across several commits you'll get multiple searchers open." Not sure if
> this is a general observation, or specific to the particular poster's
> situation.
>
> Finally, what do people mean when they blog that they have Solr set up for
> "n threads"? Is that the same thing as saying that Solr can be processing n
> requests simultaneously?
>
> Thanks for any insight or even links to relevant pages.  We've been
> Googling all over and haven't found answers to the above.
>
> Thanks,
> xoa
>
> --
> Andy Lester => andy@petdance.com => www.petdance.com => AIM:petdance
>
>


-- 
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics

<http://www.griddynamics.com>
 <mk...@griddynamics.com>