You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Demian Katz <de...@villanova.edu> on 2011/06/03 14:44:33 UTC
Solr performance tuning - disk i/o?
Hello,
I'm trying to move a VuFind installation from an ailing physical server into a virtualized environment, and I'm running into performance problems. VuFind is a Solr 1.4.1-based application with fairly large and complex records (many stored fields, many words per record). My particular installation contains about a million records in the index, with a total index size around 6GB.
The virtual environment has more RAM and better CPUs than the old physical box, and I am satisfied that my Java environment is well-tuned. My index is optimized. Searches that hit the cache respond very well. The problem is that non-cached searches are very slow - the more keywords I add, the slower they get, to the point of taking 6-12 seconds to come back with results on a quiet box and well over a minute under stress testing. (The old box still took a while for equivalent searches, but it was about twice as fast as the new one).
My gut feeling is that disk access reading the index is the bottleneck here, but I know little about the specifics of Solr's internals, so it's entirely possible that my gut is wrong. Outside testing does show that the the virtual environment's disk performance is not as good as the old physical server, especially when multiple processes are trying to access the same file simultaneously.
So, two basic questions:
1.) Would you agree that I'm dealing with a disk bottleneck, or are there some other factors I should be considering? Any good diagnostics I should be looking at?
2.) If the problem is disk access, is there anything I can tune on the Solr side to alleviate the problems?
Thanks,
Demian
Re: Solr performance tuning - disk i/o?
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Demian,
* You can run iostat or vmstat and see if there is disk IO during your slow
queries and compare that to disk IO (if any) with your fast/cached queries
* You can make sure you warm up your index well after the first and any new
searcher, so that OS and Solr caches are warmed up
* You can look at Solr Stats page to make sure your caches are utilized well and
adjust their settings if they are not.
* ...
Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/
----- Original Message ----
> From: Demian Katz <de...@villanova.edu>
> To: "solr-user@lucene.apache.org" <so...@lucene.apache.org>
> Sent: Fri, June 3, 2011 8:44:33 AM
> Subject: Solr performance tuning - disk i/o?
>
> Hello,
>
> I'm trying to move a VuFind installation from an ailing physical server into a
>virtualized environment, and I'm running into performance problems. VuFind is
>a Solr 1.4.1-based application with fairly large and complex records (many
>stored fields, many words per record). My particular installation contains
>about a million records in the index, with a total index size around 6GB.
>
> The virtual environment has more RAM and better CPUs than the old physical
>box, and I am satisfied that my Java environment is well-tuned. My index is
>optimized. Searches that hit the cache respond very well. The problem is that
>non-cached searches are very slow - the more keywords I add, the slower they
>get, to the point of taking 6-12 seconds to come back with results on a quiet
>box and well over a minute under stress testing. (The old box still took a
>while for equivalent searches, but it was about twice as fast as the new one).
>
> My gut feeling is that disk access reading the index is the bottleneck here,
>but I know little about the specifics of Solr's internals, so it's entirely
>possible that my gut is wrong. Outside testing does show that the the virtual
>environment's disk performance is not as good as the old physical server,
>especially when multiple processes are trying to access the same file
>simultaneously.
>
> So, two basic questions:
>
>
> 1.) Would you agree that I'm dealing with a disk bottleneck, or are there
>some other factors I should be considering? Any good diagnostics I should be
>looking at?
>
> 2.) If the problem is disk access, is there anything I can tune on the Solr
>side to alleviate the problems?
>
> Thanks,
> Demian
>
Re: Solr performance tuning - disk i/o?
Posted by Erick Erickson <er...@gmail.com>.
If you're seeing results, things must be OK. It's a little strange,
though, I'm seeing
warmup times of 1 on the trivial reload of the example documents.
But I wouldn't worry too much here. Those are pretty high autowarm counts, you
might have room to reduce them but absent long autowarm times there's not much
reason to mess with them...
Best
Erick
On Mon, Jun 6, 2011 at 1:38 PM, Demian Katz <de...@villanova.edu> wrote:
> All of my cache autowarmCount settings are either 10000 or 50000. maxWarmingSearchers is set to 2. I previously shared the contents of my firstSearcher and newSearcher events -- just a "queries" array surrounded by a standard-looking <listener> tag. The events are definitely firing -- in addition to the measurable performance improvement they give me, I can actually see them happening in the console output during startup. That seems to cover every configuration option in my file that references warming in any way, and it all looks reasonable to me. warmupTime remains consistently 0 in the statistics display. Is there anything else I should be looking at? In any case, I'm not too alarmed by this... it just seems a little strange.
>
> thanks,
> Demian
>
>> -----Original Message-----
>> From: Erick Erickson [mailto:erickerickson@gmail.com]
>> Sent: Monday, June 06, 2011 11:59 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Solr performance tuning - disk i/o?
>>
>> Polling interval was in reference to slaves in a multi-machine
>> master/slave setup. so probably not
>> a concern just at present.
>>
>> Warmup time of 0 is not particularly normal, I'm not quite sure what's
>> going on there but you may
>> want to look at firstsearcher, newsearcher and autowarm parameters in
>> config.xml..
>>
>> Best
>> Erick
>>
>> On Mon, Jun 6, 2011 at 9:08 AM, Demian Katz <de...@villanova.edu>
>> wrote:
>> > Thanks once again for the helpful suggestions!
>> >
>> > Regarding the selection of facet fields, I think publishDate (which
>> is actually just a year) and callnumber-first (which is actually a very
>> broad, high-level category) are okay. authorStr is an interesting
>> problem: it's definitely a useful facet (when a user searches for an
>> author, odds are good that they want the one who published the most
>> books... i.e. a search for dickens will probably show Charles Dickens
>> at the top of the facet list), but it has a long tail since there are
>> many minor authors who have only published one or two books... Is
>> there a possibility that the facet.mincount parameter could be helpful
>> here, or does that have no impact on performance/memory footprint?
>> >
>> > Regarding polling interval for slaves, are you referring to a
>> distributed Solr environment, or is this something to do with Solr's
>> internals? We're currently a single-server environment, so I don't
>> think I have to worry if it's related to a multi-server setup... but
>> if it's something internal, could you point me to the right area of the
>> admin panel to check my stats? I'm not seeing anything about polling
>> on the statistics page. It's also a little strange that all of my
>> warmupTime stats on searchers and caches are showing as 0 -- is that
>> normal?
>> >
>> > thanks,
>> > Demian
>> >
>> >> -----Original Message-----
>> >> From: Erick Erickson [mailto:erickerickson@gmail.com]
>> >> Sent: Friday, June 03, 2011 4:45 PM
>> >> To: solr-user@lucene.apache.org
>> >> Subject: Re: Solr performance tuning - disk i/o?
>> >>
>> >> Quick impressions:
>> >>
>> >> The faceting is usually best done on fields that don't have lots of
>> >> unique
>> >> values for three reasons:
>> >> 1> It's questionable how much use to the user to have a gazillion
>> >> facets.
>> >> In the case of a unique field per document, in fact, it's
>> useless.
>> >> 2> resource requirements go up as a function of the number of unique
>> >> terms. This is true for faceting and sorting.
>> >> 3> warmup times grow the more terms have to be read into memory.
>> >>
>> >>
>> >> Glancing at your warmup stuff, things like publishDate, authorStr
>> and
>> >> maybe
>> >> callnumber-first are questionable. publishDate depends on how coarse
>> >> the
>> >> resolution is. If it's by day, that's not really much use.
>> authorStr..
>> >> How many
>> >> authors have more than one publication? Would this be better served
>> by
>> >> some
>> >> kind of autosuggest rather than facets? callnumber-first... I don't
>> >> really know, but
>> >> if it's unique per document it's probably not something the user
>> would
>> >> find useful
>> >> as a facet.
>> >>
>> >> The admin page will help you determine the number of unique terms
>> per
>> >> field,
>> >> which may guide you whether or not to continue to facet on these
>> >> fields.
>> >>
>> >> As Otis said, doing a sort on the fields during warmup will also
>> help.
>> >>
>> >> Watch your polling interval for any slaves in relation to the warmup
>> >> times.
>> >> If your polling interval is shorter than the warmup times, you run a
>> >> risk of
>> >> "runaway warmups".
>> >>
>> >> As you've figured out, measuring responses to the first few queries
>> >> doesn't
>> >> always measure what you really need <G>..
>> >>
>> >> I don't have the pages handy, but autowarming is a good topic to
>> >> understand,
>> >> so you might spend some time tracking it down.
>> >>
>> >> Best
>> >> Erick
>> >>
>> >> On Fri, Jun 3, 2011 at 11:21 AM, Demian Katz
>> >> <de...@villanova.edu> wrote:
>> >> > Thanks to you and Otis for the suggestions! Some more
>> information:
>> >> >
>> >> > - Based on the Solr stats page, my caches seem to be working
>> pretty
>> >> well (few or no evictions, hit rates in the 75-80% range).
>> >> > - VuFind is actually doing two Solr queries per search (one
>> initial
>> >> search followed by a supplemental spell check search -- I believe
>> this
>> >> is necessary because VuFind has two separate spelling indexes, one
>> for
>> >> shingled terms and one for single words). That is probably
>> >> exaggerating the problem, though based on searches with debugQuery
>> on,
>> >> it looks like it's always the initial search (rather than the
>> >> supplemental spelling search) that's consuming the bulk of the time.
>> >> > - enableLazyFieldLoading is set to true.
>> >> > - I'm retrieving 20 documents per page.
>> >> > - My JVM settings: -server -
>> >> Xloggc:/usr/local/vufind/solr/jetty/logs/gc.log -Xms4096m -Xmx4096m
>> -
>> >> XX:+UseParallelGC -XX:+UseParallelOldGC -XX:NewRatio=5
>> >> >
>> >> > It appears that a large portion of my problem had to do with
>> >> autowarming, a topic that I've never had a strong grasp on, though
>> >> perhaps I'm finally learning (any recommended primer links would be
>> >> welcome!). I did have some autowarming settings in solrconfig.xml
>> (an
>> >> arbitrary search for a bunch of random keywords in the newSearcher
>> and
>> >> firstSearcher events, plus autowarmCount settings on all of my
>> caches).
>> >> However, when I looked at the debugQuery output, I noticed that a
>> huge
>> >> amount of time was being wasted loading facets on the first search
>> >> after restarting Solr, so I changed my newSearcher and firstSearcher
>> >> events to this:
>> >> >
>> >> > <arr name="queries">
>> >> > <lst>
>> >> > <str name="q">*:*</str>
>> >> > <str name="start">0</str>
>> >> > <str name="rows">10</str>
>> >> > <str name="facet">true</str>
>> >> > <str name="facet.mincount">1</str>
>> >> > <str name="facet.field">collection</str>
>> >> > <str name="facet.field">format</str>
>> >> > <str name="facet.field">publishDate</str>
>> >> > <str name="facet.field">callnumber-first</str>
>> >> > <str name="facet.field">topic_facet</str>
>> >> > <str name="facet.field">authorStr</str>
>> >> > <str name="facet.field">language</str>
>> >> > <str name="facet.field">genre_facet</str>
>> >> > <str name="facet.field">era_facet</str>
>> >> > <str name="facet.field">geographic_facet</str>
>> >> > </lst>
>> >> > </arr>
>> >> >
>> >> > Overall performance has now increased dramatically, and now the
>> >> biggest bottleneck in the debug output seems to be the shingle spell
>> >> checking!
>> >> >
>> >> > Any other suggestions are welcome, since I suspect there's still
>> room
>> >> to squeeze more performance out of the system, and I'm still not
>> sure
>> >> I'm making the most of autowarming... but this seems like a big
>> step
>> >> in the right direction. Thanks again for the help!
>> >> >
>> >> > - Demian
>> >> >
>> >> >> -----Original Message-----
>> >> >> From: Erick Erickson [mailto:erickerickson@gmail.com]
>> >> >> Sent: Friday, June 03, 2011 9:41 AM
>> >> >> To: solr-user@lucene.apache.org
>> >> >> Subject: Re: Solr performance tuning - disk i/o?
>> >> >>
>> >> >> This doesn't seem right. Here's a couple of things to try:
>> >> >> 1> attach &debugQuery=on to your long-running queries. The QTime
>> >> >> returned
>> >> >> is the time taken to search, NOT including the time to load
>> the
>> >> >> docs. That'll
>> >> >> help pinpoint whether the problem is the search itself, or
>> >> >> assembling the
>> >> >> documents.
>> >> >> 2> Are you autowarming? If so, be sure it's actually done before
>> >> >> querying.
>> >> >> 3> Measure queries after the first few, particularly if you're
>> >> sorting
>> >> >> or
>> >> >> faceting.
>> >> >> 4> What are your JVM settings? How much memory do you have?
>> >> >> 5> is <enableLazyFieldLoading> set to true in your
>> solrconfig.xml?
>> >> >> 6> How many docs are you returning?
>> >> >>
>> >> >>
>> >> >> There's more, but that'll do for a start.... Let us know if you
>> >> gather
>> >> >> more data
>> >> >> and it's still slow.
>> >> >>
>> >> >> Best
>> >> >> Erick
>> >> >>
>> >> >> On Fri, Jun 3, 2011 at 8:44 AM, Demian Katz
>> >> <de...@villanova.edu>
>> >> >> wrote:
>> >> >> > Hello,
>> >> >> >
>> >> >> > I'm trying to move a VuFind installation from an ailing
>> physical
>> >> >> server into a virtualized environment, and I'm running into
>> >> performance
>> >> >> problems. VuFind is a Solr 1.4.1-based application with fairly
>> >> large
>> >> >> and complex records (many stored fields, many words per record).
>> My
>> >> >> particular installation contains about a million records in the
>> >> index,
>> >> >> with a total index size around 6GB.
>> >> >> >
>> >> >> > The virtual environment has more RAM and better CPUs than the
>> old
>> >> >> physical box, and I am satisfied that my Java environment is
>> well-
>> >> >> tuned. My index is optimized. Searches that hit the cache
>> respond
>> >> >> very well. The problem is that non-cached searches are very slow
>> -
>> >> the
>> >> >> more keywords I add, the slower they get, to the point of taking
>> 6-
>> >> 12
>> >> >> seconds to come back with results on a quiet box and well over a
>> >> minute
>> >> >> under stress testing. (The old box still took a while for
>> >> equivalent
>> >> >> searches, but it was about twice as fast as the new one).
>> >> >> >
>> >> >> > My gut feeling is that disk access reading the index is the
>> >> >> bottleneck here, but I know little about the specifics of Solr's
>> >> >> internals, so it's entirely possible that my gut is wrong.
>> Outside
>> >> >> testing does show that the the virtual environment's disk
>> >> performance
>> >> >> is not as good as the old physical server, especially when
>> multiple
>> >> >> processes are trying to access the same file simultaneously.
>> >> >> >
>> >> >> > So, two basic questions:
>> >> >> >
>> >> >> >
>> >> >> > 1.) Would you agree that I'm dealing with a disk bottleneck,
>> or
>> >> >> are there some other factors I should be considering? Any good
>> >> >> diagnostics I should be looking at?
>> >> >> >
>> >> >> > 2.) If the problem is disk access, is there anything I can
>> tune
>> >> on
>> >> >> the Solr side to alleviate the problems?
>> >> >> >
>> >> >> > Thanks,
>> >> >> > Demian
>> >> >> >
>> >> >
>> >
>
RE: Solr performance tuning - disk i/o?
Posted by Demian Katz <de...@villanova.edu>.
All of my cache autowarmCount settings are either 10000 or 50000. maxWarmingSearchers is set to 2. I previously shared the contents of my firstSearcher and newSearcher events -- just a "queries" array surrounded by a standard-looking <listener> tag. The events are definitely firing -- in addition to the measurable performance improvement they give me, I can actually see them happening in the console output during startup. That seems to cover every configuration option in my file that references warming in any way, and it all looks reasonable to me. warmupTime remains consistently 0 in the statistics display. Is there anything else I should be looking at? In any case, I'm not too alarmed by this... it just seems a little strange.
thanks,
Demian
> -----Original Message-----
> From: Erick Erickson [mailto:erickerickson@gmail.com]
> Sent: Monday, June 06, 2011 11:59 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr performance tuning - disk i/o?
>
> Polling interval was in reference to slaves in a multi-machine
> master/slave setup. so probably not
> a concern just at present.
>
> Warmup time of 0 is not particularly normal, I'm not quite sure what's
> going on there but you may
> want to look at firstsearcher, newsearcher and autowarm parameters in
> config.xml..
>
> Best
> Erick
>
> On Mon, Jun 6, 2011 at 9:08 AM, Demian Katz <de...@villanova.edu>
> wrote:
> > Thanks once again for the helpful suggestions!
> >
> > Regarding the selection of facet fields, I think publishDate (which
> is actually just a year) and callnumber-first (which is actually a very
> broad, high-level category) are okay. authorStr is an interesting
> problem: it's definitely a useful facet (when a user searches for an
> author, odds are good that they want the one who published the most
> books... i.e. a search for dickens will probably show Charles Dickens
> at the top of the facet list), but it has a long tail since there are
> many minor authors who have only published one or two books... Is
> there a possibility that the facet.mincount parameter could be helpful
> here, or does that have no impact on performance/memory footprint?
> >
> > Regarding polling interval for slaves, are you referring to a
> distributed Solr environment, or is this something to do with Solr's
> internals? We're currently a single-server environment, so I don't
> think I have to worry if it's related to a multi-server setup... but
> if it's something internal, could you point me to the right area of the
> admin panel to check my stats? I'm not seeing anything about polling
> on the statistics page. It's also a little strange that all of my
> warmupTime stats on searchers and caches are showing as 0 -- is that
> normal?
> >
> > thanks,
> > Demian
> >
> >> -----Original Message-----
> >> From: Erick Erickson [mailto:erickerickson@gmail.com]
> >> Sent: Friday, June 03, 2011 4:45 PM
> >> To: solr-user@lucene.apache.org
> >> Subject: Re: Solr performance tuning - disk i/o?
> >>
> >> Quick impressions:
> >>
> >> The faceting is usually best done on fields that don't have lots of
> >> unique
> >> values for three reasons:
> >> 1> It's questionable how much use to the user to have a gazillion
> >> facets.
> >> In the case of a unique field per document, in fact, it's
> useless.
> >> 2> resource requirements go up as a function of the number of unique
> >> terms. This is true for faceting and sorting.
> >> 3> warmup times grow the more terms have to be read into memory.
> >>
> >>
> >> Glancing at your warmup stuff, things like publishDate, authorStr
> and
> >> maybe
> >> callnumber-first are questionable. publishDate depends on how coarse
> >> the
> >> resolution is. If it's by day, that's not really much use.
> authorStr..
> >> How many
> >> authors have more than one publication? Would this be better served
> by
> >> some
> >> kind of autosuggest rather than facets? callnumber-first... I don't
> >> really know, but
> >> if it's unique per document it's probably not something the user
> would
> >> find useful
> >> as a facet.
> >>
> >> The admin page will help you determine the number of unique terms
> per
> >> field,
> >> which may guide you whether or not to continue to facet on these
> >> fields.
> >>
> >> As Otis said, doing a sort on the fields during warmup will also
> help.
> >>
> >> Watch your polling interval for any slaves in relation to the warmup
> >> times.
> >> If your polling interval is shorter than the warmup times, you run a
> >> risk of
> >> "runaway warmups".
> >>
> >> As you've figured out, measuring responses to the first few queries
> >> doesn't
> >> always measure what you really need <G>..
> >>
> >> I don't have the pages handy, but autowarming is a good topic to
> >> understand,
> >> so you might spend some time tracking it down.
> >>
> >> Best
> >> Erick
> >>
> >> On Fri, Jun 3, 2011 at 11:21 AM, Demian Katz
> >> <de...@villanova.edu> wrote:
> >> > Thanks to you and Otis for the suggestions! Some more
> information:
> >> >
> >> > - Based on the Solr stats page, my caches seem to be working
> pretty
> >> well (few or no evictions, hit rates in the 75-80% range).
> >> > - VuFind is actually doing two Solr queries per search (one
> initial
> >> search followed by a supplemental spell check search -- I believe
> this
> >> is necessary because VuFind has two separate spelling indexes, one
> for
> >> shingled terms and one for single words). That is probably
> >> exaggerating the problem, though based on searches with debugQuery
> on,
> >> it looks like it's always the initial search (rather than the
> >> supplemental spelling search) that's consuming the bulk of the time.
> >> > - enableLazyFieldLoading is set to true.
> >> > - I'm retrieving 20 documents per page.
> >> > - My JVM settings: -server -
> >> Xloggc:/usr/local/vufind/solr/jetty/logs/gc.log -Xms4096m -Xmx4096m
> -
> >> XX:+UseParallelGC -XX:+UseParallelOldGC -XX:NewRatio=5
> >> >
> >> > It appears that a large portion of my problem had to do with
> >> autowarming, a topic that I've never had a strong grasp on, though
> >> perhaps I'm finally learning (any recommended primer links would be
> >> welcome!). I did have some autowarming settings in solrconfig.xml
> (an
> >> arbitrary search for a bunch of random keywords in the newSearcher
> and
> >> firstSearcher events, plus autowarmCount settings on all of my
> caches).
> >> However, when I looked at the debugQuery output, I noticed that a
> huge
> >> amount of time was being wasted loading facets on the first search
> >> after restarting Solr, so I changed my newSearcher and firstSearcher
> >> events to this:
> >> >
> >> > <arr name="queries">
> >> > <lst>
> >> > <str name="q">*:*</str>
> >> > <str name="start">0</str>
> >> > <str name="rows">10</str>
> >> > <str name="facet">true</str>
> >> > <str name="facet.mincount">1</str>
> >> > <str name="facet.field">collection</str>
> >> > <str name="facet.field">format</str>
> >> > <str name="facet.field">publishDate</str>
> >> > <str name="facet.field">callnumber-first</str>
> >> > <str name="facet.field">topic_facet</str>
> >> > <str name="facet.field">authorStr</str>
> >> > <str name="facet.field">language</str>
> >> > <str name="facet.field">genre_facet</str>
> >> > <str name="facet.field">era_facet</str>
> >> > <str name="facet.field">geographic_facet</str>
> >> > </lst>
> >> > </arr>
> >> >
> >> > Overall performance has now increased dramatically, and now the
> >> biggest bottleneck in the debug output seems to be the shingle spell
> >> checking!
> >> >
> >> > Any other suggestions are welcome, since I suspect there's still
> room
> >> to squeeze more performance out of the system, and I'm still not
> sure
> >> I'm making the most of autowarming... but this seems like a big
> step
> >> in the right direction. Thanks again for the help!
> >> >
> >> > - Demian
> >> >
> >> >> -----Original Message-----
> >> >> From: Erick Erickson [mailto:erickerickson@gmail.com]
> >> >> Sent: Friday, June 03, 2011 9:41 AM
> >> >> To: solr-user@lucene.apache.org
> >> >> Subject: Re: Solr performance tuning - disk i/o?
> >> >>
> >> >> This doesn't seem right. Here's a couple of things to try:
> >> >> 1> attach &debugQuery=on to your long-running queries. The QTime
> >> >> returned
> >> >> is the time taken to search, NOT including the time to load
> the
> >> >> docs. That'll
> >> >> help pinpoint whether the problem is the search itself, or
> >> >> assembling the
> >> >> documents.
> >> >> 2> Are you autowarming? If so, be sure it's actually done before
> >> >> querying.
> >> >> 3> Measure queries after the first few, particularly if you're
> >> sorting
> >> >> or
> >> >> faceting.
> >> >> 4> What are your JVM settings? How much memory do you have?
> >> >> 5> is <enableLazyFieldLoading> set to true in your
> solrconfig.xml?
> >> >> 6> How many docs are you returning?
> >> >>
> >> >>
> >> >> There's more, but that'll do for a start.... Let us know if you
> >> gather
> >> >> more data
> >> >> and it's still slow.
> >> >>
> >> >> Best
> >> >> Erick
> >> >>
> >> >> On Fri, Jun 3, 2011 at 8:44 AM, Demian Katz
> >> <de...@villanova.edu>
> >> >> wrote:
> >> >> > Hello,
> >> >> >
> >> >> > I'm trying to move a VuFind installation from an ailing
> physical
> >> >> server into a virtualized environment, and I'm running into
> >> performance
> >> >> problems. VuFind is a Solr 1.4.1-based application with fairly
> >> large
> >> >> and complex records (many stored fields, many words per record).
> My
> >> >> particular installation contains about a million records in the
> >> index,
> >> >> with a total index size around 6GB.
> >> >> >
> >> >> > The virtual environment has more RAM and better CPUs than the
> old
> >> >> physical box, and I am satisfied that my Java environment is
> well-
> >> >> tuned. My index is optimized. Searches that hit the cache
> respond
> >> >> very well. The problem is that non-cached searches are very slow
> -
> >> the
> >> >> more keywords I add, the slower they get, to the point of taking
> 6-
> >> 12
> >> >> seconds to come back with results on a quiet box and well over a
> >> minute
> >> >> under stress testing. (The old box still took a while for
> >> equivalent
> >> >> searches, but it was about twice as fast as the new one).
> >> >> >
> >> >> > My gut feeling is that disk access reading the index is the
> >> >> bottleneck here, but I know little about the specifics of Solr's
> >> >> internals, so it's entirely possible that my gut is wrong.
> Outside
> >> >> testing does show that the the virtual environment's disk
> >> performance
> >> >> is not as good as the old physical server, especially when
> multiple
> >> >> processes are trying to access the same file simultaneously.
> >> >> >
> >> >> > So, two basic questions:
> >> >> >
> >> >> >
> >> >> > 1.) Would you agree that I'm dealing with a disk bottleneck,
> or
> >> >> are there some other factors I should be considering? Any good
> >> >> diagnostics I should be looking at?
> >> >> >
> >> >> > 2.) If the problem is disk access, is there anything I can
> tune
> >> on
> >> >> the Solr side to alleviate the problems?
> >> >> >
> >> >> > Thanks,
> >> >> > Demian
> >> >> >
> >> >
> >
Re: Solr performance tuning - disk i/o?
Posted by Erick Erickson <er...@gmail.com>.
Polling interval was in reference to slaves in a multi-machine
master/slave setup. so probably not
a concern just at present.
Warmup time of 0 is not particularly normal, I'm not quite sure what's
going on there but you may
want to look at firstsearcher, newsearcher and autowarm parameters in
config.xml..
Best
Erick
On Mon, Jun 6, 2011 at 9:08 AM, Demian Katz <de...@villanova.edu> wrote:
> Thanks once again for the helpful suggestions!
>
> Regarding the selection of facet fields, I think publishDate (which is actually just a year) and callnumber-first (which is actually a very broad, high-level category) are okay. authorStr is an interesting problem: it's definitely a useful facet (when a user searches for an author, odds are good that they want the one who published the most books... i.e. a search for dickens will probably show Charles Dickens at the top of the facet list), but it has a long tail since there are many minor authors who have only published one or two books... Is there a possibility that the facet.mincount parameter could be helpful here, or does that have no impact on performance/memory footprint?
>
> Regarding polling interval for slaves, are you referring to a distributed Solr environment, or is this something to do with Solr's internals? We're currently a single-server environment, so I don't think I have to worry if it's related to a multi-server setup... but if it's something internal, could you point me to the right area of the admin panel to check my stats? I'm not seeing anything about polling on the statistics page. It's also a little strange that all of my warmupTime stats on searchers and caches are showing as 0 -- is that normal?
>
> thanks,
> Demian
>
>> -----Original Message-----
>> From: Erick Erickson [mailto:erickerickson@gmail.com]
>> Sent: Friday, June 03, 2011 4:45 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Solr performance tuning - disk i/o?
>>
>> Quick impressions:
>>
>> The faceting is usually best done on fields that don't have lots of
>> unique
>> values for three reasons:
>> 1> It's questionable how much use to the user to have a gazillion
>> facets.
>> In the case of a unique field per document, in fact, it's useless.
>> 2> resource requirements go up as a function of the number of unique
>> terms. This is true for faceting and sorting.
>> 3> warmup times grow the more terms have to be read into memory.
>>
>>
>> Glancing at your warmup stuff, things like publishDate, authorStr and
>> maybe
>> callnumber-first are questionable. publishDate depends on how coarse
>> the
>> resolution is. If it's by day, that's not really much use. authorStr..
>> How many
>> authors have more than one publication? Would this be better served by
>> some
>> kind of autosuggest rather than facets? callnumber-first... I don't
>> really know, but
>> if it's unique per document it's probably not something the user would
>> find useful
>> as a facet.
>>
>> The admin page will help you determine the number of unique terms per
>> field,
>> which may guide you whether or not to continue to facet on these
>> fields.
>>
>> As Otis said, doing a sort on the fields during warmup will also help.
>>
>> Watch your polling interval for any slaves in relation to the warmup
>> times.
>> If your polling interval is shorter than the warmup times, you run a
>> risk of
>> "runaway warmups".
>>
>> As you've figured out, measuring responses to the first few queries
>> doesn't
>> always measure what you really need <G>..
>>
>> I don't have the pages handy, but autowarming is a good topic to
>> understand,
>> so you might spend some time tracking it down.
>>
>> Best
>> Erick
>>
>> On Fri, Jun 3, 2011 at 11:21 AM, Demian Katz
>> <de...@villanova.edu> wrote:
>> > Thanks to you and Otis for the suggestions! Some more information:
>> >
>> > - Based on the Solr stats page, my caches seem to be working pretty
>> well (few or no evictions, hit rates in the 75-80% range).
>> > - VuFind is actually doing two Solr queries per search (one initial
>> search followed by a supplemental spell check search -- I believe this
>> is necessary because VuFind has two separate spelling indexes, one for
>> shingled terms and one for single words). That is probably
>> exaggerating the problem, though based on searches with debugQuery on,
>> it looks like it's always the initial search (rather than the
>> supplemental spelling search) that's consuming the bulk of the time.
>> > - enableLazyFieldLoading is set to true.
>> > - I'm retrieving 20 documents per page.
>> > - My JVM settings: -server -
>> Xloggc:/usr/local/vufind/solr/jetty/logs/gc.log -Xms4096m -Xmx4096m -
>> XX:+UseParallelGC -XX:+UseParallelOldGC -XX:NewRatio=5
>> >
>> > It appears that a large portion of my problem had to do with
>> autowarming, a topic that I've never had a strong grasp on, though
>> perhaps I'm finally learning (any recommended primer links would be
>> welcome!). I did have some autowarming settings in solrconfig.xml (an
>> arbitrary search for a bunch of random keywords in the newSearcher and
>> firstSearcher events, plus autowarmCount settings on all of my caches).
>> However, when I looked at the debugQuery output, I noticed that a huge
>> amount of time was being wasted loading facets on the first search
>> after restarting Solr, so I changed my newSearcher and firstSearcher
>> events to this:
>> >
>> > <arr name="queries">
>> > <lst>
>> > <str name="q">*:*</str>
>> > <str name="start">0</str>
>> > <str name="rows">10</str>
>> > <str name="facet">true</str>
>> > <str name="facet.mincount">1</str>
>> > <str name="facet.field">collection</str>
>> > <str name="facet.field">format</str>
>> > <str name="facet.field">publishDate</str>
>> > <str name="facet.field">callnumber-first</str>
>> > <str name="facet.field">topic_facet</str>
>> > <str name="facet.field">authorStr</str>
>> > <str name="facet.field">language</str>
>> > <str name="facet.field">genre_facet</str>
>> > <str name="facet.field">era_facet</str>
>> > <str name="facet.field">geographic_facet</str>
>> > </lst>
>> > </arr>
>> >
>> > Overall performance has now increased dramatically, and now the
>> biggest bottleneck in the debug output seems to be the shingle spell
>> checking!
>> >
>> > Any other suggestions are welcome, since I suspect there's still room
>> to squeeze more performance out of the system, and I'm still not sure
>> I'm making the most of autowarming... but this seems like a big step
>> in the right direction. Thanks again for the help!
>> >
>> > - Demian
>> >
>> >> -----Original Message-----
>> >> From: Erick Erickson [mailto:erickerickson@gmail.com]
>> >> Sent: Friday, June 03, 2011 9:41 AM
>> >> To: solr-user@lucene.apache.org
>> >> Subject: Re: Solr performance tuning - disk i/o?
>> >>
>> >> This doesn't seem right. Here's a couple of things to try:
>> >> 1> attach &debugQuery=on to your long-running queries. The QTime
>> >> returned
>> >> is the time taken to search, NOT including the time to load the
>> >> docs. That'll
>> >> help pinpoint whether the problem is the search itself, or
>> >> assembling the
>> >> documents.
>> >> 2> Are you autowarming? If so, be sure it's actually done before
>> >> querying.
>> >> 3> Measure queries after the first few, particularly if you're
>> sorting
>> >> or
>> >> faceting.
>> >> 4> What are your JVM settings? How much memory do you have?
>> >> 5> is <enableLazyFieldLoading> set to true in your solrconfig.xml?
>> >> 6> How many docs are you returning?
>> >>
>> >>
>> >> There's more, but that'll do for a start.... Let us know if you
>> gather
>> >> more data
>> >> and it's still slow.
>> >>
>> >> Best
>> >> Erick
>> >>
>> >> On Fri, Jun 3, 2011 at 8:44 AM, Demian Katz
>> <de...@villanova.edu>
>> >> wrote:
>> >> > Hello,
>> >> >
>> >> > I'm trying to move a VuFind installation from an ailing physical
>> >> server into a virtualized environment, and I'm running into
>> performance
>> >> problems. VuFind is a Solr 1.4.1-based application with fairly
>> large
>> >> and complex records (many stored fields, many words per record). My
>> >> particular installation contains about a million records in the
>> index,
>> >> with a total index size around 6GB.
>> >> >
>> >> > The virtual environment has more RAM and better CPUs than the old
>> >> physical box, and I am satisfied that my Java environment is well-
>> >> tuned. My index is optimized. Searches that hit the cache respond
>> >> very well. The problem is that non-cached searches are very slow -
>> the
>> >> more keywords I add, the slower they get, to the point of taking 6-
>> 12
>> >> seconds to come back with results on a quiet box and well over a
>> minute
>> >> under stress testing. (The old box still took a while for
>> equivalent
>> >> searches, but it was about twice as fast as the new one).
>> >> >
>> >> > My gut feeling is that disk access reading the index is the
>> >> bottleneck here, but I know little about the specifics of Solr's
>> >> internals, so it's entirely possible that my gut is wrong. Outside
>> >> testing does show that the the virtual environment's disk
>> performance
>> >> is not as good as the old physical server, especially when multiple
>> >> processes are trying to access the same file simultaneously.
>> >> >
>> >> > So, two basic questions:
>> >> >
>> >> >
>> >> > 1.) Would you agree that I'm dealing with a disk bottleneck, or
>> >> are there some other factors I should be considering? Any good
>> >> diagnostics I should be looking at?
>> >> >
>> >> > 2.) If the problem is disk access, is there anything I can tune
>> on
>> >> the Solr side to alleviate the problems?
>> >> >
>> >> > Thanks,
>> >> > Demian
>> >> >
>> >
>
RE: Solr performance tuning - disk i/o?
Posted by Demian Katz <de...@villanova.edu>.
Thanks once again for the helpful suggestions!
Regarding the selection of facet fields, I think publishDate (which is actually just a year) and callnumber-first (which is actually a very broad, high-level category) are okay. authorStr is an interesting problem: it's definitely a useful facet (when a user searches for an author, odds are good that they want the one who published the most books... i.e. a search for dickens will probably show Charles Dickens at the top of the facet list), but it has a long tail since there are many minor authors who have only published one or two books... Is there a possibility that the facet.mincount parameter could be helpful here, or does that have no impact on performance/memory footprint?
Regarding polling interval for slaves, are you referring to a distributed Solr environment, or is this something to do with Solr's internals? We're currently a single-server environment, so I don't think I have to worry if it's related to a multi-server setup... but if it's something internal, could you point me to the right area of the admin panel to check my stats? I'm not seeing anything about polling on the statistics page. It's also a little strange that all of my warmupTime stats on searchers and caches are showing as 0 -- is that normal?
thanks,
Demian
> -----Original Message-----
> From: Erick Erickson [mailto:erickerickson@gmail.com]
> Sent: Friday, June 03, 2011 4:45 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr performance tuning - disk i/o?
>
> Quick impressions:
>
> The faceting is usually best done on fields that don't have lots of
> unique
> values for three reasons:
> 1> It's questionable how much use to the user to have a gazillion
> facets.
> In the case of a unique field per document, in fact, it's useless.
> 2> resource requirements go up as a function of the number of unique
> terms. This is true for faceting and sorting.
> 3> warmup times grow the more terms have to be read into memory.
>
>
> Glancing at your warmup stuff, things like publishDate, authorStr and
> maybe
> callnumber-first are questionable. publishDate depends on how coarse
> the
> resolution is. If it's by day, that's not really much use. authorStr..
> How many
> authors have more than one publication? Would this be better served by
> some
> kind of autosuggest rather than facets? callnumber-first... I don't
> really know, but
> if it's unique per document it's probably not something the user would
> find useful
> as a facet.
>
> The admin page will help you determine the number of unique terms per
> field,
> which may guide you whether or not to continue to facet on these
> fields.
>
> As Otis said, doing a sort on the fields during warmup will also help.
>
> Watch your polling interval for any slaves in relation to the warmup
> times.
> If your polling interval is shorter than the warmup times, you run a
> risk of
> "runaway warmups".
>
> As you've figured out, measuring responses to the first few queries
> doesn't
> always measure what you really need <G>..
>
> I don't have the pages handy, but autowarming is a good topic to
> understand,
> so you might spend some time tracking it down.
>
> Best
> Erick
>
> On Fri, Jun 3, 2011 at 11:21 AM, Demian Katz
> <de...@villanova.edu> wrote:
> > Thanks to you and Otis for the suggestions! Some more information:
> >
> > - Based on the Solr stats page, my caches seem to be working pretty
> well (few or no evictions, hit rates in the 75-80% range).
> > - VuFind is actually doing two Solr queries per search (one initial
> search followed by a supplemental spell check search -- I believe this
> is necessary because VuFind has two separate spelling indexes, one for
> shingled terms and one for single words). That is probably
> exaggerating the problem, though based on searches with debugQuery on,
> it looks like it's always the initial search (rather than the
> supplemental spelling search) that's consuming the bulk of the time.
> > - enableLazyFieldLoading is set to true.
> > - I'm retrieving 20 documents per page.
> > - My JVM settings: -server -
> Xloggc:/usr/local/vufind/solr/jetty/logs/gc.log -Xms4096m -Xmx4096m -
> XX:+UseParallelGC -XX:+UseParallelOldGC -XX:NewRatio=5
> >
> > It appears that a large portion of my problem had to do with
> autowarming, a topic that I've never had a strong grasp on, though
> perhaps I'm finally learning (any recommended primer links would be
> welcome!). I did have some autowarming settings in solrconfig.xml (an
> arbitrary search for a bunch of random keywords in the newSearcher and
> firstSearcher events, plus autowarmCount settings on all of my caches).
> However, when I looked at the debugQuery output, I noticed that a huge
> amount of time was being wasted loading facets on the first search
> after restarting Solr, so I changed my newSearcher and firstSearcher
> events to this:
> >
> > <arr name="queries">
> > <lst>
> > <str name="q">*:*</str>
> > <str name="start">0</str>
> > <str name="rows">10</str>
> > <str name="facet">true</str>
> > <str name="facet.mincount">1</str>
> > <str name="facet.field">collection</str>
> > <str name="facet.field">format</str>
> > <str name="facet.field">publishDate</str>
> > <str name="facet.field">callnumber-first</str>
> > <str name="facet.field">topic_facet</str>
> > <str name="facet.field">authorStr</str>
> > <str name="facet.field">language</str>
> > <str name="facet.field">genre_facet</str>
> > <str name="facet.field">era_facet</str>
> > <str name="facet.field">geographic_facet</str>
> > </lst>
> > </arr>
> >
> > Overall performance has now increased dramatically, and now the
> biggest bottleneck in the debug output seems to be the shingle spell
> checking!
> >
> > Any other suggestions are welcome, since I suspect there's still room
> to squeeze more performance out of the system, and I'm still not sure
> I'm making the most of autowarming... but this seems like a big step
> in the right direction. Thanks again for the help!
> >
> > - Demian
> >
> >> -----Original Message-----
> >> From: Erick Erickson [mailto:erickerickson@gmail.com]
> >> Sent: Friday, June 03, 2011 9:41 AM
> >> To: solr-user@lucene.apache.org
> >> Subject: Re: Solr performance tuning - disk i/o?
> >>
> >> This doesn't seem right. Here's a couple of things to try:
> >> 1> attach &debugQuery=on to your long-running queries. The QTime
> >> returned
> >> is the time taken to search, NOT including the time to load the
> >> docs. That'll
> >> help pinpoint whether the problem is the search itself, or
> >> assembling the
> >> documents.
> >> 2> Are you autowarming? If so, be sure it's actually done before
> >> querying.
> >> 3> Measure queries after the first few, particularly if you're
> sorting
> >> or
> >> faceting.
> >> 4> What are your JVM settings? How much memory do you have?
> >> 5> is <enableLazyFieldLoading> set to true in your solrconfig.xml?
> >> 6> How many docs are you returning?
> >>
> >>
> >> There's more, but that'll do for a start.... Let us know if you
> gather
> >> more data
> >> and it's still slow.
> >>
> >> Best
> >> Erick
> >>
> >> On Fri, Jun 3, 2011 at 8:44 AM, Demian Katz
> <de...@villanova.edu>
> >> wrote:
> >> > Hello,
> >> >
> >> > I'm trying to move a VuFind installation from an ailing physical
> >> server into a virtualized environment, and I'm running into
> performance
> >> problems. VuFind is a Solr 1.4.1-based application with fairly
> large
> >> and complex records (many stored fields, many words per record). My
> >> particular installation contains about a million records in the
> index,
> >> with a total index size around 6GB.
> >> >
> >> > The virtual environment has more RAM and better CPUs than the old
> >> physical box, and I am satisfied that my Java environment is well-
> >> tuned. My index is optimized. Searches that hit the cache respond
> >> very well. The problem is that non-cached searches are very slow -
> the
> >> more keywords I add, the slower they get, to the point of taking 6-
> 12
> >> seconds to come back with results on a quiet box and well over a
> minute
> >> under stress testing. (The old box still took a while for
> equivalent
> >> searches, but it was about twice as fast as the new one).
> >> >
> >> > My gut feeling is that disk access reading the index is the
> >> bottleneck here, but I know little about the specifics of Solr's
> >> internals, so it's entirely possible that my gut is wrong. Outside
> >> testing does show that the the virtual environment's disk
> performance
> >> is not as good as the old physical server, especially when multiple
> >> processes are trying to access the same file simultaneously.
> >> >
> >> > So, two basic questions:
> >> >
> >> >
> >> > 1.) Would you agree that I'm dealing with a disk bottleneck, or
> >> are there some other factors I should be considering? Any good
> >> diagnostics I should be looking at?
> >> >
> >> > 2.) If the problem is disk access, is there anything I can tune
> on
> >> the Solr side to alleviate the problems?
> >> >
> >> > Thanks,
> >> > Demian
> >> >
> >
Re: Solr performance tuning - disk i/o?
Posted by Erick Erickson <er...@gmail.com>.
Quick impressions:
The faceting is usually best done on fields that don't have lots of unique
values for three reasons:
1> It's questionable how much use to the user to have a gazillion facets.
In the case of a unique field per document, in fact, it's useless.
2> resource requirements go up as a function of the number of unique
terms. This is true for faceting and sorting.
3> warmup times grow the more terms have to be read into memory.
Glancing at your warmup stuff, things like publishDate, authorStr and maybe
callnumber-first are questionable. publishDate depends on how coarse the
resolution is. If it's by day, that's not really much use. authorStr.. How many
authors have more than one publication? Would this be better served by some
kind of autosuggest rather than facets? callnumber-first... I don't
really know, but
if it's unique per document it's probably not something the user would
find useful
as a facet.
The admin page will help you determine the number of unique terms per field,
which may guide you whether or not to continue to facet on these fields.
As Otis said, doing a sort on the fields during warmup will also help.
Watch your polling interval for any slaves in relation to the warmup times.
If your polling interval is shorter than the warmup times, you run a risk of
"runaway warmups".
As you've figured out, measuring responses to the first few queries doesn't
always measure what you really need <G>..
I don't have the pages handy, but autowarming is a good topic to understand,
so you might spend some time tracking it down.
Best
Erick
On Fri, Jun 3, 2011 at 11:21 AM, Demian Katz <de...@villanova.edu> wrote:
> Thanks to you and Otis for the suggestions! Some more information:
>
> - Based on the Solr stats page, my caches seem to be working pretty well (few or no evictions, hit rates in the 75-80% range).
> - VuFind is actually doing two Solr queries per search (one initial search followed by a supplemental spell check search -- I believe this is necessary because VuFind has two separate spelling indexes, one for shingled terms and one for single words). That is probably exaggerating the problem, though based on searches with debugQuery on, it looks like it's always the initial search (rather than the supplemental spelling search) that's consuming the bulk of the time.
> - enableLazyFieldLoading is set to true.
> - I'm retrieving 20 documents per page.
> - My JVM settings: -server -Xloggc:/usr/local/vufind/solr/jetty/logs/gc.log -Xms4096m -Xmx4096m -XX:+UseParallelGC -XX:+UseParallelOldGC -XX:NewRatio=5
>
> It appears that a large portion of my problem had to do with autowarming, a topic that I've never had a strong grasp on, though perhaps I'm finally learning (any recommended primer links would be welcome!). I did have some autowarming settings in solrconfig.xml (an arbitrary search for a bunch of random keywords in the newSearcher and firstSearcher events, plus autowarmCount settings on all of my caches). However, when I looked at the debugQuery output, I noticed that a huge amount of time was being wasted loading facets on the first search after restarting Solr, so I changed my newSearcher and firstSearcher events to this:
>
> <arr name="queries">
> <lst>
> <str name="q">*:*</str>
> <str name="start">0</str>
> <str name="rows">10</str>
> <str name="facet">true</str>
> <str name="facet.mincount">1</str>
> <str name="facet.field">collection</str>
> <str name="facet.field">format</str>
> <str name="facet.field">publishDate</str>
> <str name="facet.field">callnumber-first</str>
> <str name="facet.field">topic_facet</str>
> <str name="facet.field">authorStr</str>
> <str name="facet.field">language</str>
> <str name="facet.field">genre_facet</str>
> <str name="facet.field">era_facet</str>
> <str name="facet.field">geographic_facet</str>
> </lst>
> </arr>
>
> Overall performance has now increased dramatically, and now the biggest bottleneck in the debug output seems to be the shingle spell checking!
>
> Any other suggestions are welcome, since I suspect there's still room to squeeze more performance out of the system, and I'm still not sure I'm making the most of autowarming... but this seems like a big step in the right direction. Thanks again for the help!
>
> - Demian
>
>> -----Original Message-----
>> From: Erick Erickson [mailto:erickerickson@gmail.com]
>> Sent: Friday, June 03, 2011 9:41 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Solr performance tuning - disk i/o?
>>
>> This doesn't seem right. Here's a couple of things to try:
>> 1> attach &debugQuery=on to your long-running queries. The QTime
>> returned
>> is the time taken to search, NOT including the time to load the
>> docs. That'll
>> help pinpoint whether the problem is the search itself, or
>> assembling the
>> documents.
>> 2> Are you autowarming? If so, be sure it's actually done before
>> querying.
>> 3> Measure queries after the first few, particularly if you're sorting
>> or
>> faceting.
>> 4> What are your JVM settings? How much memory do you have?
>> 5> is <enableLazyFieldLoading> set to true in your solrconfig.xml?
>> 6> How many docs are you returning?
>>
>>
>> There's more, but that'll do for a start.... Let us know if you gather
>> more data
>> and it's still slow.
>>
>> Best
>> Erick
>>
>> On Fri, Jun 3, 2011 at 8:44 AM, Demian Katz <de...@villanova.edu>
>> wrote:
>> > Hello,
>> >
>> > I'm trying to move a VuFind installation from an ailing physical
>> server into a virtualized environment, and I'm running into performance
>> problems. VuFind is a Solr 1.4.1-based application with fairly large
>> and complex records (many stored fields, many words per record). My
>> particular installation contains about a million records in the index,
>> with a total index size around 6GB.
>> >
>> > The virtual environment has more RAM and better CPUs than the old
>> physical box, and I am satisfied that my Java environment is well-
>> tuned. My index is optimized. Searches that hit the cache respond
>> very well. The problem is that non-cached searches are very slow - the
>> more keywords I add, the slower they get, to the point of taking 6-12
>> seconds to come back with results on a quiet box and well over a minute
>> under stress testing. (The old box still took a while for equivalent
>> searches, but it was about twice as fast as the new one).
>> >
>> > My gut feeling is that disk access reading the index is the
>> bottleneck here, but I know little about the specifics of Solr's
>> internals, so it's entirely possible that my gut is wrong. Outside
>> testing does show that the the virtual environment's disk performance
>> is not as good as the old physical server, especially when multiple
>> processes are trying to access the same file simultaneously.
>> >
>> > So, two basic questions:
>> >
>> >
>> > 1.) Would you agree that I'm dealing with a disk bottleneck, or
>> are there some other factors I should be considering? Any good
>> diagnostics I should be looking at?
>> >
>> > 2.) If the problem is disk access, is there anything I can tune on
>> the Solr side to alleviate the problems?
>> >
>> > Thanks,
>> > Demian
>> >
>
Re: Solr performance tuning - disk i/o?
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Right, if you facet results, then your warmup queries should include those
facets. The same with sorting. If you sort on fields A and B, then include
warmup queries that sort on A and B.
Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/
----- Original Message ----
> From: Demian Katz <de...@villanova.edu>
> To: "solr-user@lucene.apache.org" <so...@lucene.apache.org>
> Sent: Fri, June 3, 2011 11:21:52 AM
> Subject: RE: Solr performance tuning - disk i/o?
>
> Thanks to you and Otis for the suggestions! Some more information:
>
> - Based on the Solr stats page, my caches seem to be working pretty well (few
>or no evictions, hit rates in the 75-80% range).
> - VuFind is actually doing two Solr queries per search (one initial search
>followed by a supplemental spell check search -- I believe this is necessary
>because VuFind has two separate spelling indexes, one for shingled terms and
>one for single words). That is probably exaggerating the problem, though based
>on searches with debugQuery on, it looks like it's always the initial search
>(rather than the supplemental spelling search) that's consuming the bulk of the
>time.
> - enableLazyFieldLoading is set to true.
> - I'm retrieving 20 documents per page.
> - My JVM settings: -server -Xloggc:/usr/local/vufind/solr/jetty/logs/gc.log
>-Xms4096m -Xmx4096m -XX:+UseParallelGC -XX:+UseParallelOldGC -XX:NewRatio=5
>
> It appears that a large portion of my problem had to do with autowarming, a
>topic that I've never had a strong grasp on, though perhaps I'm finally
>learning (any recommended primer links would be welcome!). I did have some
>autowarming settings in solrconfig.xml (an arbitrary search for a bunch of
>random keywords in the newSearcher and firstSearcher events, plus autowarmCount
>settings on all of my caches). However, when I looked at the debugQuery
>output, I noticed that a huge amount of time was being wasted loading facets on
>the first search after restarting Solr, so I changed my newSearcher and
>firstSearcher events to this:
>
> <arr name="queries">
> <lst>
> <str name="q">*:*</str>
> <str name="start">0</str>
> <str name="rows">10</str>
> <str name="facet">true</str>
> <str name="facet.mincount">1</str>
> <str name="facet.field">collection</str>
> <str name="facet.field">format</str>
> <str name="facet.field">publishDate</str>
> <str name="facet.field">callnumber-first</str>
> <str name="facet.field">topic_facet</str>
> <str name="facet.field">authorStr</str>
> <str name="facet.field">language</str>
> <str name="facet.field">genre_facet</str>
> <str name="facet.field">era_facet</str>
> <str name="facet.field">geographic_facet</str>
> </lst>
> </arr>
>
> Overall performance has now increased dramatically, and now the biggest
>bottleneck in the debug output seems to be the shingle spell checking!
>
> Any other suggestions are welcome, since I suspect there's still room to
>squeeze more performance out of the system, and I'm still not sure I'm making
>the most of autowarming... but this seems like a big step in the right
>direction. Thanks again for the help!
>
> - Demian
>
> > -----Original Message-----
> > From: Erick Erickson [mailto:erickerickson@gmail.com]
> > Sent: Friday, June 03, 2011 9:41 AM
> > To: solr-user@lucene.apache.org
> > Subject: Re: Solr performance tuning - disk i/o?
> >
> > This doesn't seem right. Here's a couple of things to try:
> > 1> attach &debugQuery=on to your long-running queries. The QTime
> > returned
> > is the time taken to search, NOT including the time to load the
> > docs. That'll
> > help pinpoint whether the problem is the search itself, or
> > assembling the
> > documents.
> > 2> Are you autowarming? If so, be sure it's actually done before
> > querying.
> > 3> Measure queries after the first few, particularly if you're sorting
> > or
> > faceting.
> > 4> What are your JVM settings? How much memory do you have?
> > 5> is <enableLazyFieldLoading> set to true in your solrconfig.xml?
> > 6> How many docs are you returning?
> >
> >
> > There's more, but that'll do for a start.... Let us know if you gather
> > more data
> > and it's still slow.
> >
> > Best
> > Erick
> >
> > On Fri, Jun 3, 2011 at 8:44 AM, Demian Katz <de...@villanova.edu>
> > wrote:
> > > Hello,
> > >
> > > I'm trying to move a VuFind installation from an ailing physical
> > server into a virtualized environment, and I'm running into performance
> > problems. VuFind is a Solr 1.4.1-based application with fairly large
> > and complex records (many stored fields, many words per record). My
> > particular installation contains about a million records in the index,
> > with a total index size around 6GB.
> > >
> > > The virtual environment has more RAM and better CPUs than the old
> > physical box, and I am satisfied that my Java environment is well-
> > tuned. My index is optimized. Searches that hit the cache respond
> > very well. The problem is that non-cached searches are very slow - the
> > more keywords I add, the slower they get, to the point of taking 6-12
> > seconds to come back with results on a quiet box and well over a minute
> > under stress testing. (The old box still took a while for equivalent
> > searches, but it was about twice as fast as the new one).
> > >
> > > My gut feeling is that disk access reading the index is the
> > bottleneck here, but I know little about the specifics of Solr's
> > internals, so it's entirely possible that my gut is wrong. Outside
> > testing does show that the the virtual environment's disk performance
> > is not as good as the old physical server, especially when multiple
> > processes are trying to access the same file simultaneously.
> > >
> > > So, two basic questions:
> > >
> > >
> > > 1.) Would you agree that I'm dealing with a disk bottleneck, or
> > are there some other factors I should be considering? Any good
> > diagnostics I should be looking at?
> > >
> > > 2.) If the problem is disk access, is there anything I can tune on
> > the Solr side to alleviate the problems?
> > >
> > > Thanks,
> > > Demian
> > >
>
RE: Solr performance tuning - disk i/o?
Posted by Demian Katz <de...@villanova.edu>.
Thanks to you and Otis for the suggestions! Some more information:
- Based on the Solr stats page, my caches seem to be working pretty well (few or no evictions, hit rates in the 75-80% range).
- VuFind is actually doing two Solr queries per search (one initial search followed by a supplemental spell check search -- I believe this is necessary because VuFind has two separate spelling indexes, one for shingled terms and one for single words). That is probably exaggerating the problem, though based on searches with debugQuery on, it looks like it's always the initial search (rather than the supplemental spelling search) that's consuming the bulk of the time.
- enableLazyFieldLoading is set to true.
- I'm retrieving 20 documents per page.
- My JVM settings: -server -Xloggc:/usr/local/vufind/solr/jetty/logs/gc.log -Xms4096m -Xmx4096m -XX:+UseParallelGC -XX:+UseParallelOldGC -XX:NewRatio=5
It appears that a large portion of my problem had to do with autowarming, a topic that I've never had a strong grasp on, though perhaps I'm finally learning (any recommended primer links would be welcome!). I did have some autowarming settings in solrconfig.xml (an arbitrary search for a bunch of random keywords in the newSearcher and firstSearcher events, plus autowarmCount settings on all of my caches). However, when I looked at the debugQuery output, I noticed that a huge amount of time was being wasted loading facets on the first search after restarting Solr, so I changed my newSearcher and firstSearcher events to this:
<arr name="queries">
<lst>
<str name="q">*:*</str>
<str name="start">0</str>
<str name="rows">10</str>
<str name="facet">true</str>
<str name="facet.mincount">1</str>
<str name="facet.field">collection</str>
<str name="facet.field">format</str>
<str name="facet.field">publishDate</str>
<str name="facet.field">callnumber-first</str>
<str name="facet.field">topic_facet</str>
<str name="facet.field">authorStr</str>
<str name="facet.field">language</str>
<str name="facet.field">genre_facet</str>
<str name="facet.field">era_facet</str>
<str name="facet.field">geographic_facet</str>
</lst>
</arr>
Overall performance has now increased dramatically, and now the biggest bottleneck in the debug output seems to be the shingle spell checking!
Any other suggestions are welcome, since I suspect there's still room to squeeze more performance out of the system, and I'm still not sure I'm making the most of autowarming... but this seems like a big step in the right direction. Thanks again for the help!
- Demian
> -----Original Message-----
> From: Erick Erickson [mailto:erickerickson@gmail.com]
> Sent: Friday, June 03, 2011 9:41 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr performance tuning - disk i/o?
>
> This doesn't seem right. Here's a couple of things to try:
> 1> attach &debugQuery=on to your long-running queries. The QTime
> returned
> is the time taken to search, NOT including the time to load the
> docs. That'll
> help pinpoint whether the problem is the search itself, or
> assembling the
> documents.
> 2> Are you autowarming? If so, be sure it's actually done before
> querying.
> 3> Measure queries after the first few, particularly if you're sorting
> or
> faceting.
> 4> What are your JVM settings? How much memory do you have?
> 5> is <enableLazyFieldLoading> set to true in your solrconfig.xml?
> 6> How many docs are you returning?
>
>
> There's more, but that'll do for a start.... Let us know if you gather
> more data
> and it's still slow.
>
> Best
> Erick
>
> On Fri, Jun 3, 2011 at 8:44 AM, Demian Katz <de...@villanova.edu>
> wrote:
> > Hello,
> >
> > I'm trying to move a VuFind installation from an ailing physical
> server into a virtualized environment, and I'm running into performance
> problems. VuFind is a Solr 1.4.1-based application with fairly large
> and complex records (many stored fields, many words per record). My
> particular installation contains about a million records in the index,
> with a total index size around 6GB.
> >
> > The virtual environment has more RAM and better CPUs than the old
> physical box, and I am satisfied that my Java environment is well-
> tuned. My index is optimized. Searches that hit the cache respond
> very well. The problem is that non-cached searches are very slow - the
> more keywords I add, the slower they get, to the point of taking 6-12
> seconds to come back with results on a quiet box and well over a minute
> under stress testing. (The old box still took a while for equivalent
> searches, but it was about twice as fast as the new one).
> >
> > My gut feeling is that disk access reading the index is the
> bottleneck here, but I know little about the specifics of Solr's
> internals, so it's entirely possible that my gut is wrong. Outside
> testing does show that the the virtual environment's disk performance
> is not as good as the old physical server, especially when multiple
> processes are trying to access the same file simultaneously.
> >
> > So, two basic questions:
> >
> >
> > 1.) Would you agree that I'm dealing with a disk bottleneck, or
> are there some other factors I should be considering? Any good
> diagnostics I should be looking at?
> >
> > 2.) If the problem is disk access, is there anything I can tune on
> the Solr side to alleviate the problems?
> >
> > Thanks,
> > Demian
> >
Re: Solr performance tuning - disk i/o?
Posted by Erick Erickson <er...@gmail.com>.
This doesn't seem right. Here's a couple of things to try:
1> attach &debugQuery=on to your long-running queries. The QTime returned
is the time taken to search, NOT including the time to load the
docs. That'll
help pinpoint whether the problem is the search itself, or assembling the
documents.
2> Are you autowarming? If so, be sure it's actually done before querying.
3> Measure queries after the first few, particularly if you're sorting or
faceting.
4> What are your JVM settings? How much memory do you have?
5> is <enableLazyFieldLoading> set to true in your solrconfig.xml?
6> How many docs are you returning?
There's more, but that'll do for a start.... Let us know if you gather more data
and it's still slow.
Best
Erick
On Fri, Jun 3, 2011 at 8:44 AM, Demian Katz <de...@villanova.edu> wrote:
> Hello,
>
> I'm trying to move a VuFind installation from an ailing physical server into a virtualized environment, and I'm running into performance problems. VuFind is a Solr 1.4.1-based application with fairly large and complex records (many stored fields, many words per record). My particular installation contains about a million records in the index, with a total index size around 6GB.
>
> The virtual environment has more RAM and better CPUs than the old physical box, and I am satisfied that my Java environment is well-tuned. My index is optimized. Searches that hit the cache respond very well. The problem is that non-cached searches are very slow - the more keywords I add, the slower they get, to the point of taking 6-12 seconds to come back with results on a quiet box and well over a minute under stress testing. (The old box still took a while for equivalent searches, but it was about twice as fast as the new one).
>
> My gut feeling is that disk access reading the index is the bottleneck here, but I know little about the specifics of Solr's internals, so it's entirely possible that my gut is wrong. Outside testing does show that the the virtual environment's disk performance is not as good as the old physical server, especially when multiple processes are trying to access the same file simultaneously.
>
> So, two basic questions:
>
>
> 1.) Would you agree that I'm dealing with a disk bottleneck, or are there some other factors I should be considering? Any good diagnostics I should be looking at?
>
> 2.) If the problem is disk access, is there anything I can tune on the Solr side to alleviate the problems?
>
> Thanks,
> Demian
>