You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Peter Karich <pe...@yahoo.de> on 2010/11/15 21:31:17 UTC
Re: Tuning Solr caches with high commit rates (NRT)
Just in case someone is interested:
I put the emails of Peter Sturge with some minor edits in the wiki:
http://wiki.apache.org/solr/NearRealtimeSearchTuning
I found myself search the thread again and again ;-)
Feel free to add and edit content!
Regards,
Peter.
> Hi Erik,
>
> I thought this would be good for the wiki, but I've not submitted to
> the wiki before, so I thought I'd put this info out there first, then
> add it if it was deemed useful.
> If you could let me know the procedure for submitting, it probably
> would be worth getting it into the wiki (couldn't do it straightaway,
> as I have a lot of projects on at the moment). If you're able/willing
> to put it on there for me, that would be very kind of you!
>
> Thanks!
> Peter
>
>
> On Sun, Sep 12, 2010 at 5:43 PM, Erick Erickson<er...@gmail.com> wrote:
>> Peter:
>>
>> This kind of information is extremely useful to document, thanks! Do you
>> have the time/energy to put it up on the Wiki? Anyone can edit it by
>> creating
>> a logon. If you don't, would it be OK if someone else did it (with
>> attribution,
>> of course)? I guess that by bringing it up I'm volunteering :)...
>>
>> Best
>> Erick
>>
>> On Sun, Sep 12, 2010 at 12:26 PM, Peter Sturge<pe...@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> Below are some notes regarding Solr cache tuning that should prove
>>> useful for anyone who uses Solr with frequent commits (e.g.<5min).
>>>
>>> Environment:
>>> Solr 1.4.1 or branch_3x trunk.
>>> Note the 4.x trunk has lots of neat new features, so the notes here
>>> are likely less relevant to the 4.x environment.
>>>
>>> Overview:
>>> Our Solr environment makes extensive use of faceting, we perform
>>> commits every 30secs, and the indexes tend be on the large-ish side
>>> (>20million docs).
>>> Note: For our data, when we commit, we are always adding new data,
>>> never changing existing data.
>>> This type of environment can be tricky to tune, as Solr is more geared
>>> toward fast reads than frequent writes.
>>>
>>> Symptoms:
>>> If anyone has used faceting in searches where you are also performing
>>> frequent commits, you've likely encountered the dreaded OutOfMemory or
>>> GC Overhead Exeeded errors.
>>> In high commit rate environments, this is almost always due to
>>> multiple 'onDeck' searchers and autowarming - i.e. new searchers don't
>>> finish autowarming their caches before the next commit()
>>> comes along and invalidates them.
>>> Once this starts happening on a regular basis, it is likely your
>>> Solr's JVM will run out of memory eventually, as the number of
>>> searchers (and their cache arrays) will keep growing until the JVM
>>> dies of thirst.
>>> To check if your Solr environment is suffering from this, turn on INFO
>>> level logging, and look for: 'PERFORMANCE WARNING: Overlapping
>>> onDeckSearchers=x'.
>>>
>>> In tests, we've only ever seen this problem when using faceting, and
>>> facet.method=fc.
>>>
>>> Some solutions to this are:
>>> Reduce the commit rate to allow searchers to fully warm before the
>>> next commit
>>> Reduce or eliminate the autowarming in caches
>>> Both of the above
>>>
>>> The trouble is, if you're doing NRT commits, you likely have a good
>>> reason for it, and reducing/elimintating autowarming will very
>>> significantly impact search performance in high commit rate
>>> environments.
>>>
>>> Solution:
>>> Here are some setup steps we've used that allow lots of faceting (we
>>> typically search with at least 20-35 different facet fields, and date
>>> faceting/sorting) on large indexes, and still keep decent search
>>> performance:
>>>
>>> 1. Firstly, you should consider using the enum method for facet
>>> searches (facet.method=enum) unless you've got A LOT of memory on your
>>> machine. In our tests, this method uses a lot less memory and
>>> autowarms more quickly than fc. (Note, I've not tried the new
>>> segement-based 'fcs' option, as I can't find support for it in
>>> branch_3x - looks nice for 4.x though)
>>> Admittedly, for our data, enum is not quite as fast for searching as
>>> fc, but short of purchsing a Thaiwanese RAM factory, it's a worthwhile
>>> tradeoff.
>>> If you do have access to LOTS of memory, AND you can guarantee that
>>> the index won't grow beyond the memory capacity (i.e. you have some
>>> sort of deletion policy in place), fc can be a lot faster than enum
>>> when searching with lots of facets across many terms.
>>>
>>> 2. Secondly, we've found that LRUCache is faster at autowarming than
>>> FastLRUCache - in our tests, about 20% faster. Maybe this is just our
>>> environment - your mileage may vary.
>>>
>>> So, our filterCache section in solrconfig.xml looks like this:
>>> <filterCache
>>> class="solr.LRUCache"
>>> size="3600"
>>> initialSize="1400"
>>> autowarmCount="3600"/>
>>>
>>> For a 28GB index, running in a quad-core x64 VMWare instance, 30
>>> warmed facet fields, Solr is running at ~4GB. Stats filterCache size
>>> shows usually in the region of ~2400.
>>>
>>> 3. It's also a good idea to have some sort of
>>> firstSearcher/newSearcher event listener queries to allow new data to
>>> populate the caches.
>>> Of course, what you put in these is dependent on the facets you need/use.
>>> We've found a good combination is a firstSearcher with as many facets
>>> in the search as your environment can handle, then a subset of the
>>> most common facets for the newSearcher.
>>>
>>> 4. We also set:
>>> <useColdSearcher>true</useColdSearcher>
>>> just in case.
>>>
>>> 5. Another key area for search performance with high commits is to use
>>> 2 Solr instances - one for the high commit rate indexing, and one for
>>> searching.
>>> The read-only searching instance can be a remote replica, or a local
>>> read-only instance that reads the same core as the indexing instance
>>> (for the latter, you'll need something that periodically refreshes -
>>> i.e. runs commit()).
>>> This way, you can tune the indexing instance for writing performance
>>> and the searching instance as above for max read performance.
>>>
>>> Using the setup above, we get fantastic searching speed for small
>>> facet sets (well under 1sec), and really good searching for large
>>> facet sets (a couple of secs depending on index size, number of
>>> facets, unique terms etc. etc.),
>>> even when searching against largeish indexes (>20million docs).
>>> We have yet to see any OOM or GC errors using the techniques above,
>>> even in low memory conditions.
>>>
>>> I hope there are people that find this useful. I know I've spent a lot
>>> of time looking for stuff like this, so hopefullly, this will save
>>> someone some time.
>>>
>>>
>>> Peter
>>>
--
http://jetwick.com twitter search prototype
Re: Tuning Solr caches with high commit rates (NRT)
Posted by Peter Sturge <pe...@gmail.com>.
In order for the 'read-only' instance to see any new/updated
documents, it needs to do a commit (since it's read-only, it is a
commit of 0 documents).
You can do this via a client service that issues periodic commits, or
use autorefresh from within solrconfig.xml. Be careful that you don't
do anything in the read-only instance that will change the underlying
index - like optimize.
Peter
On Thu, Dec 2, 2010 at 12:51 PM, stockii <st...@shopgate.com> wrote:
>
> great thread and exactly my problems :D
>
> i set up two solr-instances, one for update the index and another for
> searching.
>
> When i perform an update. the search-instance dont get the new documents.
> when i start a commit on searcher he found it. how can i say the searcher
> that he alwas look not only the "old" index. automatic refresh ? XD
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Tuning-Solr-caches-with-high-commit-rates-NRT-tp1461275p2005738.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Tuning Solr caches with high commit rates (NRT)
Posted by stockii <st...@shopgate.com>.
great thread and exactly my problems :D
i set up two solr-instances, one for update the index and another for
searching.
When i perform an update. the search-instance dont get the new documents.
when i start a commit on searcher he found it. how can i say the searcher
that he alwas look not only the "old" index. automatic refresh ? XD
--
View this message in context: http://lucene.472066.n3.nabble.com/Tuning-Solr-caches-with-high-commit-rates-NRT-tp1461275p2005738.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Tuning Solr caches with high commit rates (NRT)
Posted by Peter Sturge <pe...@gmail.com>.
Many thanks, Peter K. for posting up on the wiki - great!
Yes, fc = field cache. Field Collapsing is something very nice indeed,
but is entirely different.
As Erik mentions in the wiki post, using per-segment faceting can be a
huge boon to performance. It does require the latest Solr trunk build
and new Lucene, though (last time I checked, this isn't in the Solr 3x
branch).
enum vs fc? This will depend a lot on what your data looks like - e.g.
lots of unique terms vs lots of the same terms.
In all the tests we've done here with >20m doc indexes (using 3x
branch), enum has always used less memory than fc (sometimes much
less), but fc is faster for searches. Again, your data experience may
vary.
The main point in this thread for NRT and faceting is to warm caches
as quickly as possible - this generally means judicious facet
selection, and for us at least, using LRUCache a.o.t. FastLRUCache for
filter caches.
On Mon, Nov 15, 2010 at 11:56 PM, Koji Sekiguchi <ko...@r.email.ne.jp> wrote:
> (10/11/16 8:36), Jonathan Rochkind wrote:
>>
>> In Solr 1.4, facet.method=enum DOES work on multi-valued fields, I'm
>> pretty certain.
>
> Correct, and I didn't say that facet.method=enum doesn't work for
> multiValued/tokenized field in my previous mail.
>
>> I think Koji's explanation is based on before Solr 1.4
>
> No, as facet.method had been introduced in 1.4.
>
> Koji
> --
> http://www.rondhuit.com/en/
>
Re: Tuning Solr caches with high commit rates (NRT)
Posted by Koji Sekiguchi <ko...@r.email.ne.jp>.
(10/11/16 8:36), Jonathan Rochkind wrote:
> In Solr 1.4, facet.method=enum DOES work on multi-valued fields, I'm pretty certain.
Correct, and I didn't say that facet.method=enum doesn't work for multiValued/tokenized field in my
previous mail.
> I think Koji's explanation is based on before Solr 1.4
No, as facet.method had been introduced in 1.4.
Koji
--
http://www.rondhuit.com/en/
Re: Tuning Solr caches with high commit rates (NRT)
Posted by Jonathan Rochkind <ro...@jhu.edu>.
Koji Sekiguchi wrote:
>
> Usually, you do not need to set facet.method because Solr
> automatically uses most appropriate facet method for
> each field type:
>
> boolean: TermEnum
> multiValued/tokenized: UnInvertedField
> other than those above: FieldCache
>
As I understand it, in Solr 1.4, (and I may NOT understand it, it is
confusing), it would be more clear to say that Solr will by default use
facet.method=enum for boolean, and facet.method=fc for everything else.
facet.method=fc is a strategy that will use one of several different
actual methods depending on field qualities. Looking at the code it
appears to be _more_ than two branches to me of choices, not sure there
are only two methods and it only depends on whether the field is
multiValued/tokenized. But if Koji knows and is sure, maybe he knows
more than me.
It has been suggested on the list and in the wiki, that in Solr 1.4+, if
you have a facet field with "few" unique values (I have not seen better
guidance for exactly what qualifies as 'few', with regard to total
number of documents), it may be profitable to use facet.method=enum.
In Solr 1.4, facet.method=enum DOES work on multi-valued fields, I'm
pretty certain. I think Koji is wrong about that, if I understand Koji
right to say that you can't use facet.method=enum with multi-valued
fields, this is not in fact true.
I think it is somewhat more complicated in 1.4+ than Koji suggests,
although I don't understand it well enough to explain it completely. I
think Koji's explanation is based on before Solr 1.4 made improvements
to the faceting algorithms.
Re: Tuning Solr caches with high commit rates (NRT)
Posted by Koji Sekiguchi <ko...@r.email.ne.jp>.
(10/11/16 6:43), Dennis Gearon wrote:
> fc='field collapsing'?
fc of facet.method=fc stands for Lucene's FieldCache.
enum of facet.method=enum stands for Lucene's TermEnum.
Usually, you do not need to set facet.method because Solr
automatically uses most appropriate facet method for
each field type:
boolean: TermEnum
multiValued/tokenized: UnInvertedField
other than those above: FieldCache
If you prefer Solr to use TermEnum, you can set facet.method=enum.
If you prefer Solr to use FieldCache, you can set facet.method=fc
(but Solr uses UnInvertedField for multiValued/tokenized fields).
Koji
--
http://www.rondhuit.com/en/
Re: Tuning Solr caches with high commit rates (NRT)
Posted by Peter Karich <pe...@yahoo.de>.
I think it stands for field cache (according to
http://wiki.apache.org/solr/SimpleFacetParameters this could be true ;-))
> fc='field collapsing'?
>
> Dennis Gearon
>
>
> Signature Warning
> ----------------
> It is always a good idea to learn from your own mistakes. It is usually a better
> idea to learn from others’ mistakes, so you do not have to make them yourself.
> from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
>
>
> EARTH has a Right To Life,
> otherwise we all die.
>
>
>
> ----- Original Message ----
> From: Peter Karich<pe...@yahoo.de>
> To: solr-user@lucene.apache.org
> Sent: Mon, November 15, 2010 1:37:00 PM
> Subject: Re: Tuning Solr caches with high commit rates (NRT)
>
> Hi Jonathan,
>
> I am too using fc because it simply was faster. Not sure if this can be applied
> in general.
> I will add this info to the wiki.
>
> Regards,
> Peter.
>
>> Awesome. I'm not sure his point 1 about facet.method=enum is still valid in
>> Solr 1.4+. The "fc" facet.method was changed significantly in 1.4, and
>> generally no longer takes a lot of memory -- for facets with "many" unique
>> values, method fc in fact should take less than enum, I think?
>>
>> Peter Karich wrote:
>>> Just in case someone is interested:
>>>
>>> I put the emails of Peter Sturge with some minor edits in the wiki:
>>>
>>> http://wiki.apache.org/solr/NearRealtimeSearchTuning
>>>
>>> I found myself search the thread again and again ;-)
>>>
>>> Feel free to add and edit content!
>>>
>>> Regards,
>>> Peter.
>>>
>>>> Hi Erik,
>>>>
>>>> I thought this would be good for the wiki, but I've not submitted to
>>>> the wiki before, so I thought I'd put this info out there first, then
>>>> add it if it was deemed useful.
>>>> If you could let me know the procedure for submitting, it probably
>>>> would be worth getting it into the wiki (couldn't do it straightaway,
>>>> as I have a lot of projects on at the moment). If you're able/willing
>>>> to put it on there for me, that would be very kind of you!
>>>>
>>>> Thanks!
>>>> Peter
>>>>
>>>>
>>>> On Sun, Sep 12, 2010 at 5:43 PM, Erick Erickson<er...@gmail.com>
>>>> wrote:
>>>>> Peter:
>>>>>
>>>>> This kind of information is extremely useful to document, thanks! Do you
>>>>> have the time/energy to put it up on the Wiki? Anyone can edit it by
>>>>> creating
>>>>> a logon. If you don't, would it be OK if someone else did it (with
>>>>> attribution,
>>>>> of course)? I guess that by bringing it up I'm volunteering :)...
>>>>>
>>>>> Best
>>>>> Erick
>>>>>
>>>>> On Sun, Sep 12, 2010 at 12:26 PM, Peter
> Sturge<pe...@gmail.com>wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Below are some notes regarding Solr cache tuning that should prove
>>>>>> useful for anyone who uses Solr with frequent commits (e.g.<5min).
>>>>>>
>>>>>> Environment:
>>>>>> Solr 1.4.1 or branch_3x trunk.
>>>>>> Note the 4.x trunk has lots of neat new features, so the notes here
>>>>>> are likely less relevant to the 4.x environment.
>>>>>>
>>>>>> Overview:
>>>>>> Our Solr environment makes extensive use of faceting, we perform
>>>>>> commits every 30secs, and the indexes tend be on the large-ish side
>>>>>> (>20million docs).
>>>>>> Note: For our data, when we commit, we are always adding new data,
>>>>>> never changing existing data.
>>>>>> This type of environment can be tricky to tune, as Solr is more geared
>>>>>> toward fast reads than frequent writes.
>>>>>>
>>>>>> Symptoms:
>>>>>> If anyone has used faceting in searches where you are also performing
>>>>>> frequent commits, you've likely encountered the dreaded OutOfMemory or
>>>>>> GC Overhead Exeeded errors.
>>>>>> In high commit rate environments, this is almost always due to
>>>>>> multiple 'onDeck' searchers and autowarming - i.e. new searchers don't
>>>>>> finish autowarming their caches before the next commit()
>>>>>> comes along and invalidates them.
>>>>>> Once this starts happening on a regular basis, it is likely your
>>>>>> Solr's JVM will run out of memory eventually, as the number of
>>>>>> searchers (and their cache arrays) will keep growing until the JVM
>>>>>> dies of thirst.
>>>>>> To check if your Solr environment is suffering from this, turn on INFO
>>>>>> level logging, and look for: 'PERFORMANCE WARNING: Overlapping
>>>>>> onDeckSearchers=x'.
>>>>>>
>>>>>> In tests, we've only ever seen this problem when using faceting, and
>>>>>> facet.method=fc.
>>>>>>
>>>>>> Some solutions to this are:
>>>>>> Reduce the commit rate to allow searchers to fully warm before the
>>>>>> next commit
>>>>>> Reduce or eliminate the autowarming in caches
>>>>>> Both of the above
>>>>>>
>>>>>> The trouble is, if you're doing NRT commits, you likely have a good
>>>>>> reason for it, and reducing/elimintating autowarming will very
>>>>>> significantly impact search performance in high commit rate
>>>>>> environments.
>>>>>>
>>>>>> Solution:
>>>>>> Here are some setup steps we've used that allow lots of faceting (we
>>>>>> typically search with at least 20-35 different facet fields, and date
>>>>>> faceting/sorting) on large indexes, and still keep decent search
>>>>>> performance:
>>>>>>
>>>>>> 1. Firstly, you should consider using the enum method for facet
>>>>>> searches (facet.method=enum) unless you've got A LOT of memory on your
>>>>>> machine. In our tests, this method uses a lot less memory and
>>>>>> autowarms more quickly than fc. (Note, I've not tried the new
>>>>>> segement-based 'fcs' option, as I can't find support for it in
>>>>>> branch_3x - looks nice for 4.x though)
>>>>>> Admittedly, for our data, enum is not quite as fast for searching as
>>>>>> fc, but short of purchsing a Thaiwanese RAM factory, it's a worthwhile
>>>>>> tradeoff.
>>>>>> If you do have access to LOTS of memory, AND you can guarantee that
>>>>>> the index won't grow beyond the memory capacity (i.e. you have some
>>>>>> sort of deletion policy in place), fc can be a lot faster than enum
>>>>>> when searching with lots of facets across many terms.
>>>>>>
>>>>>> 2. Secondly, we've found that LRUCache is faster at autowarming than
>>>>>> FastLRUCache - in our tests, about 20% faster. Maybe this is just our
>>>>>> environment - your mileage may vary.
>>>>>>
>>>>>> So, our filterCache section in solrconfig.xml looks like this:
>>>>>> <filterCache
>>>>>> class="solr.LRUCache"
>>>>>> size="3600"
>>>>>> initialSize="1400"
>>>>>> autowarmCount="3600"/>
>>>>>>
>>>>>> For a 28GB index, running in a quad-core x64 VMWare instance, 30
>>>>>> warmed facet fields, Solr is running at ~4GB. Stats filterCache size
>>>>>> shows usually in the region of ~2400.
>>>>>>
>>>>>> 3. It's also a good idea to have some sort of
>>>>>> firstSearcher/newSearcher event listener queries to allow new data to
>>>>>> populate the caches.
>>>>>> Of course, what you put in these is dependent on the facets you need/use.
>>>>>> We've found a good combination is a firstSearcher with as many facets
>>>>>> in the search as your environment can handle, then a subset of the
>>>>>> most common facets for the newSearcher.
>>>>>>
>>>>>> 4. We also set:
>>>>>> <useColdSearcher>true</useColdSearcher>
>>>>>> just in case.
>>>>>>
>>>>>> 5. Another key area for search performance with high commits is to use
>>>>>> 2 Solr instances - one for the high commit rate indexing, and one for
>>>>>> searching.
>>>>>> The read-only searching instance can be a remote replica, or a local
>>>>>> read-only instance that reads the same core as the indexing instance
>>>>>> (for the latter, you'll need something that periodically refreshes -
>>>>>> i.e. runs commit()).
>>>>>> This way, you can tune the indexing instance for writing performance
>>>>>> and the searching instance as above for max read performance.
>>>>>>
>>>>>> Using the setup above, we get fantastic searching speed for small
>>>>>> facet sets (well under 1sec), and really good searching for large
>>>>>> facet sets (a couple of secs depending on index size, number of
>>>>>> facets, unique terms etc. etc.),
>>>>>> even when searching against largeish indexes (>20million docs).
>>>>>> We have yet to see any OOM or GC errors using the techniques above,
>>>>>> even in low memory conditions.
>>>>>>
>>>>>> I hope there are people that find this useful. I know I've spent a lot
>>>>>> of time looking for stuff like this, so hopefullly, this will save
>>>>>> someone some time.
>>>>>>
>>>>>>
>>>>>> Peter
>>>>>>
--
http://jetwick.com twitter search prototype
Re: Tuning Solr caches with high commit rates (NRT)
Posted by Jonathan Rochkind <ro...@jhu.edu>.
Don't know, don't care. It may have began standing for that, I dont'
know. It's now more of a 'strategy' than a method, it uses different
algorithms depending on the nature of your facets, including whether
they are multi-term or not. I don't entirely understand it. I've looked
at the source a bit.
Dennis Gearon wrote:
> fc='field collapsing'?
>
> Dennis Gearon
>
>
> Signature Warning
> ----------------
> It is always a good idea to learn from your own mistakes. It is usually a better
> idea to learn from others’ mistakes, so you do not have to make them yourself.
> from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
>
>
> EARTH has a Right To Life,
> otherwise we all die.
>
>
>
> ----- Original Message ----
> From: Peter Karich <pe...@yahoo.de>
> To: solr-user@lucene.apache.org
> Sent: Mon, November 15, 2010 1:37:00 PM
> Subject: Re: Tuning Solr caches with high commit rates (NRT)
>
> Hi Jonathan,
>
> I am too using fc because it simply was faster. Not sure if this can be applied
> in general.
> I will add this info to the wiki.
>
> Regards,
> Peter.
>
>
>> Awesome. I'm not sure his point 1 about facet.method=enum is still valid in
>> Solr 1.4+. The "fc" facet.method was changed significantly in 1.4, and
>> generally no longer takes a lot of memory -- for facets with "many" unique
>> values, method fc in fact should take less than enum, I think?
>>
>> Peter Karich wrote:
>>
>>> Just in case someone is interested:
>>>
>>> I put the emails of Peter Sturge with some minor edits in the wiki:
>>>
>>> http://wiki.apache.org/solr/NearRealtimeSearchTuning
>>>
>>> I found myself search the thread again and again ;-)
>>>
>>> Feel free to add and edit content!
>>>
>>> Regards,
>>> Peter.
>>>
>>>
>>>> Hi Erik,
>>>>
>>>> I thought this would be good for the wiki, but I've not submitted to
>>>> the wiki before, so I thought I'd put this info out there first, then
>>>> add it if it was deemed useful.
>>>> If you could let me know the procedure for submitting, it probably
>>>> would be worth getting it into the wiki (couldn't do it straightaway,
>>>> as I have a lot of projects on at the moment). If you're able/willing
>>>> to put it on there for me, that would be very kind of you!
>>>>
>>>> Thanks!
>>>> Peter
>>>>
>>>>
>>>> On Sun, Sep 12, 2010 at 5:43 PM, Erick Erickson<er...@gmail.com>
>>>> wrote:
>>>>
>>>>> Peter:
>>>>>
>>>>> This kind of information is extremely useful to document, thanks! Do you
>>>>> have the time/energy to put it up on the Wiki? Anyone can edit it by
>>>>> creating
>>>>> a logon. If you don't, would it be OK if someone else did it (with
>>>>> attribution,
>>>>> of course)? I guess that by bringing it up I'm volunteering :)...
>>>>>
>>>>> Best
>>>>> Erick
>>>>>
>>>>> On Sun, Sep 12, 2010 at 12:26 PM, Peter
>>>>>
> Sturge<pe...@gmail.com>wrote:
>
>>>>>> Hi,
>>>>>>
>>>>>> Below are some notes regarding Solr cache tuning that should prove
>>>>>> useful for anyone who uses Solr with frequent commits (e.g.<5min).
>>>>>>
>>>>>> Environment:
>>>>>> Solr 1.4.1 or branch_3x trunk.
>>>>>> Note the 4.x trunk has lots of neat new features, so the notes here
>>>>>> are likely less relevant to the 4.x environment.
>>>>>>
>>>>>> Overview:
>>>>>> Our Solr environment makes extensive use of faceting, we perform
>>>>>> commits every 30secs, and the indexes tend be on the large-ish side
>>>>>> (>20million docs).
>>>>>> Note: For our data, when we commit, we are always adding new data,
>>>>>> never changing existing data.
>>>>>> This type of environment can be tricky to tune, as Solr is more geared
>>>>>> toward fast reads than frequent writes.
>>>>>>
>>>>>> Symptoms:
>>>>>> If anyone has used faceting in searches where you are also performing
>>>>>> frequent commits, you've likely encountered the dreaded OutOfMemory or
>>>>>> GC Overhead Exeeded errors.
>>>>>> In high commit rate environments, this is almost always due to
>>>>>> multiple 'onDeck' searchers and autowarming - i.e. new searchers don't
>>>>>> finish autowarming their caches before the next commit()
>>>>>> comes along and invalidates them.
>>>>>> Once this starts happening on a regular basis, it is likely your
>>>>>> Solr's JVM will run out of memory eventually, as the number of
>>>>>> searchers (and their cache arrays) will keep growing until the JVM
>>>>>> dies of thirst.
>>>>>> To check if your Solr environment is suffering from this, turn on INFO
>>>>>> level logging, and look for: 'PERFORMANCE WARNING: Overlapping
>>>>>> onDeckSearchers=x'.
>>>>>>
>>>>>> In tests, we've only ever seen this problem when using faceting, and
>>>>>> facet.method=fc.
>>>>>>
>>>>>> Some solutions to this are:
>>>>>> Reduce the commit rate to allow searchers to fully warm before the
>>>>>> next commit
>>>>>> Reduce or eliminate the autowarming in caches
>>>>>> Both of the above
>>>>>>
>>>>>> The trouble is, if you're doing NRT commits, you likely have a good
>>>>>> reason for it, and reducing/elimintating autowarming will very
>>>>>> significantly impact search performance in high commit rate
>>>>>> environments.
>>>>>>
>>>>>> Solution:
>>>>>> Here are some setup steps we've used that allow lots of faceting (we
>>>>>> typically search with at least 20-35 different facet fields, and date
>>>>>> faceting/sorting) on large indexes, and still keep decent search
>>>>>> performance:
>>>>>>
>>>>>> 1. Firstly, you should consider using the enum method for facet
>>>>>> searches (facet.method=enum) unless you've got A LOT of memory on your
>>>>>> machine. In our tests, this method uses a lot less memory and
>>>>>> autowarms more quickly than fc. (Note, I've not tried the new
>>>>>> segement-based 'fcs' option, as I can't find support for it in
>>>>>> branch_3x - looks nice for 4.x though)
>>>>>> Admittedly, for our data, enum is not quite as fast for searching as
>>>>>> fc, but short of purchsing a Thaiwanese RAM factory, it's a worthwhile
>>>>>> tradeoff.
>>>>>> If you do have access to LOTS of memory, AND you can guarantee that
>>>>>> the index won't grow beyond the memory capacity (i.e. you have some
>>>>>> sort of deletion policy in place), fc can be a lot faster than enum
>>>>>> when searching with lots of facets across many terms.
>>>>>>
>>>>>> 2. Secondly, we've found that LRUCache is faster at autowarming than
>>>>>> FastLRUCache - in our tests, about 20% faster. Maybe this is just our
>>>>>> environment - your mileage may vary.
>>>>>>
>>>>>> So, our filterCache section in solrconfig.xml looks like this:
>>>>>> <filterCache
>>>>>> class="solr.LRUCache"
>>>>>> size="3600"
>>>>>> initialSize="1400"
>>>>>> autowarmCount="3600"/>
>>>>>>
>>>>>> For a 28GB index, running in a quad-core x64 VMWare instance, 30
>>>>>> warmed facet fields, Solr is running at ~4GB. Stats filterCache size
>>>>>> shows usually in the region of ~2400.
>>>>>>
>>>>>> 3. It's also a good idea to have some sort of
>>>>>> firstSearcher/newSearcher event listener queries to allow new data to
>>>>>> populate the caches.
>>>>>> Of course, what you put in these is dependent on the facets you need/use.
>>>>>> We've found a good combination is a firstSearcher with as many facets
>>>>>> in the search as your environment can handle, then a subset of the
>>>>>> most common facets for the newSearcher.
>>>>>>
>>>>>> 4. We also set:
>>>>>> <useColdSearcher>true</useColdSearcher>
>>>>>> just in case.
>>>>>>
>>>>>> 5. Another key area for search performance with high commits is to use
>>>>>> 2 Solr instances - one for the high commit rate indexing, and one for
>>>>>> searching.
>>>>>> The read-only searching instance can be a remote replica, or a local
>>>>>> read-only instance that reads the same core as the indexing instance
>>>>>> (for the latter, you'll need something that periodically refreshes -
>>>>>> i.e. runs commit()).
>>>>>> This way, you can tune the indexing instance for writing performance
>>>>>> and the searching instance as above for max read performance.
>>>>>>
>>>>>> Using the setup above, we get fantastic searching speed for small
>>>>>> facet sets (well under 1sec), and really good searching for large
>>>>>> facet sets (a couple of secs depending on index size, number of
>>>>>> facets, unique terms etc. etc.),
>>>>>> even when searching against largeish indexes (>20million docs).
>>>>>> We have yet to see any OOM or GC errors using the techniques above,
>>>>>> even in low memory conditions.
>>>>>>
>>>>>> I hope there are people that find this useful. I know I've spent a lot
>>>>>> of time looking for stuff like this, so hopefullly, this will save
>>>>>> someone some time.
>>>>>>
>>>>>>
>>>>>> Peter
>>>>>>
>>>>>>
>>>
>
>
> -- http://jetwick.com twitter search prototype
>
Re: Tuning Solr caches with high commit rates (NRT)
Posted by Dennis Gearon <ge...@sbcglobal.net>.
fc='field collapsing'?
Dennis Gearon
Signature Warning
----------------
It is always a good idea to learn from your own mistakes. It is usually a better
idea to learn from others’ mistakes, so you do not have to make them yourself.
from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
EARTH has a Right To Life,
otherwise we all die.
----- Original Message ----
From: Peter Karich <pe...@yahoo.de>
To: solr-user@lucene.apache.org
Sent: Mon, November 15, 2010 1:37:00 PM
Subject: Re: Tuning Solr caches with high commit rates (NRT)
Hi Jonathan,
I am too using fc because it simply was faster. Not sure if this can be applied
in general.
I will add this info to the wiki.
Regards,
Peter.
> Awesome. I'm not sure his point 1 about facet.method=enum is still valid in
>Solr 1.4+. The "fc" facet.method was changed significantly in 1.4, and
>generally no longer takes a lot of memory -- for facets with "many" unique
>values, method fc in fact should take less than enum, I think?
>
> Peter Karich wrote:
>> Just in case someone is interested:
>>
>> I put the emails of Peter Sturge with some minor edits in the wiki:
>>
>> http://wiki.apache.org/solr/NearRealtimeSearchTuning
>>
>> I found myself search the thread again and again ;-)
>>
>> Feel free to add and edit content!
>>
>> Regards,
>> Peter.
>>
>>> Hi Erik,
>>>
>>> I thought this would be good for the wiki, but I've not submitted to
>>> the wiki before, so I thought I'd put this info out there first, then
>>> add it if it was deemed useful.
>>> If you could let me know the procedure for submitting, it probably
>>> would be worth getting it into the wiki (couldn't do it straightaway,
>>> as I have a lot of projects on at the moment). If you're able/willing
>>> to put it on there for me, that would be very kind of you!
>>>
>>> Thanks!
>>> Peter
>>>
>>>
>>> On Sun, Sep 12, 2010 at 5:43 PM, Erick Erickson<er...@gmail.com>
>>>wrote:
>>>> Peter:
>>>>
>>>> This kind of information is extremely useful to document, thanks! Do you
>>>> have the time/energy to put it up on the Wiki? Anyone can edit it by
>>>> creating
>>>> a logon. If you don't, would it be OK if someone else did it (with
>>>> attribution,
>>>> of course)? I guess that by bringing it up I'm volunteering :)...
>>>>
>>>> Best
>>>> Erick
>>>>
>>>> On Sun, Sep 12, 2010 at 12:26 PM, Peter
Sturge<pe...@gmail.com>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Below are some notes regarding Solr cache tuning that should prove
>>>>> useful for anyone who uses Solr with frequent commits (e.g.<5min).
>>>>>
>>>>> Environment:
>>>>> Solr 1.4.1 or branch_3x trunk.
>>>>> Note the 4.x trunk has lots of neat new features, so the notes here
>>>>> are likely less relevant to the 4.x environment.
>>>>>
>>>>> Overview:
>>>>> Our Solr environment makes extensive use of faceting, we perform
>>>>> commits every 30secs, and the indexes tend be on the large-ish side
>>>>> (>20million docs).
>>>>> Note: For our data, when we commit, we are always adding new data,
>>>>> never changing existing data.
>>>>> This type of environment can be tricky to tune, as Solr is more geared
>>>>> toward fast reads than frequent writes.
>>>>>
>>>>> Symptoms:
>>>>> If anyone has used faceting in searches where you are also performing
>>>>> frequent commits, you've likely encountered the dreaded OutOfMemory or
>>>>> GC Overhead Exeeded errors.
>>>>> In high commit rate environments, this is almost always due to
>>>>> multiple 'onDeck' searchers and autowarming - i.e. new searchers don't
>>>>> finish autowarming their caches before the next commit()
>>>>> comes along and invalidates them.
>>>>> Once this starts happening on a regular basis, it is likely your
>>>>> Solr's JVM will run out of memory eventually, as the number of
>>>>> searchers (and their cache arrays) will keep growing until the JVM
>>>>> dies of thirst.
>>>>> To check if your Solr environment is suffering from this, turn on INFO
>>>>> level logging, and look for: 'PERFORMANCE WARNING: Overlapping
>>>>> onDeckSearchers=x'.
>>>>>
>>>>> In tests, we've only ever seen this problem when using faceting, and
>>>>> facet.method=fc.
>>>>>
>>>>> Some solutions to this are:
>>>>> Reduce the commit rate to allow searchers to fully warm before the
>>>>> next commit
>>>>> Reduce or eliminate the autowarming in caches
>>>>> Both of the above
>>>>>
>>>>> The trouble is, if you're doing NRT commits, you likely have a good
>>>>> reason for it, and reducing/elimintating autowarming will very
>>>>> significantly impact search performance in high commit rate
>>>>> environments.
>>>>>
>>>>> Solution:
>>>>> Here are some setup steps we've used that allow lots of faceting (we
>>>>> typically search with at least 20-35 different facet fields, and date
>>>>> faceting/sorting) on large indexes, and still keep decent search
>>>>> performance:
>>>>>
>>>>> 1. Firstly, you should consider using the enum method for facet
>>>>> searches (facet.method=enum) unless you've got A LOT of memory on your
>>>>> machine. In our tests, this method uses a lot less memory and
>>>>> autowarms more quickly than fc. (Note, I've not tried the new
>>>>> segement-based 'fcs' option, as I can't find support for it in
>>>>> branch_3x - looks nice for 4.x though)
>>>>> Admittedly, for our data, enum is not quite as fast for searching as
>>>>> fc, but short of purchsing a Thaiwanese RAM factory, it's a worthwhile
>>>>> tradeoff.
>>>>> If you do have access to LOTS of memory, AND you can guarantee that
>>>>> the index won't grow beyond the memory capacity (i.e. you have some
>>>>> sort of deletion policy in place), fc can be a lot faster than enum
>>>>> when searching with lots of facets across many terms.
>>>>>
>>>>> 2. Secondly, we've found that LRUCache is faster at autowarming than
>>>>> FastLRUCache - in our tests, about 20% faster. Maybe this is just our
>>>>> environment - your mileage may vary.
>>>>>
>>>>> So, our filterCache section in solrconfig.xml looks like this:
>>>>> <filterCache
>>>>> class="solr.LRUCache"
>>>>> size="3600"
>>>>> initialSize="1400"
>>>>> autowarmCount="3600"/>
>>>>>
>>>>> For a 28GB index, running in a quad-core x64 VMWare instance, 30
>>>>> warmed facet fields, Solr is running at ~4GB. Stats filterCache size
>>>>> shows usually in the region of ~2400.
>>>>>
>>>>> 3. It's also a good idea to have some sort of
>>>>> firstSearcher/newSearcher event listener queries to allow new data to
>>>>> populate the caches.
>>>>> Of course, what you put in these is dependent on the facets you need/use.
>>>>> We've found a good combination is a firstSearcher with as many facets
>>>>> in the search as your environment can handle, then a subset of the
>>>>> most common facets for the newSearcher.
>>>>>
>>>>> 4. We also set:
>>>>> <useColdSearcher>true</useColdSearcher>
>>>>> just in case.
>>>>>
>>>>> 5. Another key area for search performance with high commits is to use
>>>>> 2 Solr instances - one for the high commit rate indexing, and one for
>>>>> searching.
>>>>> The read-only searching instance can be a remote replica, or a local
>>>>> read-only instance that reads the same core as the indexing instance
>>>>> (for the latter, you'll need something that periodically refreshes -
>>>>> i.e. runs commit()).
>>>>> This way, you can tune the indexing instance for writing performance
>>>>> and the searching instance as above for max read performance.
>>>>>
>>>>> Using the setup above, we get fantastic searching speed for small
>>>>> facet sets (well under 1sec), and really good searching for large
>>>>> facet sets (a couple of secs depending on index size, number of
>>>>> facets, unique terms etc. etc.),
>>>>> even when searching against largeish indexes (>20million docs).
>>>>> We have yet to see any OOM or GC errors using the techniques above,
>>>>> even in low memory conditions.
>>>>>
>>>>> I hope there are people that find this useful. I know I've spent a lot
>>>>> of time looking for stuff like this, so hopefullly, this will save
>>>>> someone some time.
>>>>>
>>>>>
>>>>> Peter
>>>>>
>>
>>
>
-- http://jetwick.com twitter search prototype
Re: Tuning Solr caches with high commit rates (NRT)
Posted by Peter Karich <pe...@yahoo.de>.
Hi Jonathan,
I am too using fc because it simply was faster. Not sure if this can be
applied in general.
I will add this info to the wiki.
Regards,
Peter.
> Awesome. I'm not sure his point 1 about facet.method=enum is still
> valid in Solr 1.4+. The "fc" facet.method was changed significantly
> in 1.4, and generally no longer takes a lot of memory -- for facets
> with "many" unique values, method fc in fact should take less than
> enum, I think?
>
> Peter Karich wrote:
>> Just in case someone is interested:
>>
>> I put the emails of Peter Sturge with some minor edits in the wiki:
>>
>> http://wiki.apache.org/solr/NearRealtimeSearchTuning
>>
>> I found myself search the thread again and again ;-)
>>
>> Feel free to add and edit content!
>>
>> Regards,
>> Peter.
>>
>>> Hi Erik,
>>>
>>> I thought this would be good for the wiki, but I've not submitted to
>>> the wiki before, so I thought I'd put this info out there first, then
>>> add it if it was deemed useful.
>>> If you could let me know the procedure for submitting, it probably
>>> would be worth getting it into the wiki (couldn't do it straightaway,
>>> as I have a lot of projects on at the moment). If you're able/willing
>>> to put it on there for me, that would be very kind of you!
>>>
>>> Thanks!
>>> Peter
>>>
>>>
>>> On Sun, Sep 12, 2010 at 5:43 PM, Erick
>>> Erickson<er...@gmail.com> wrote:
>>>> Peter:
>>>>
>>>> This kind of information is extremely useful to document, thanks!
>>>> Do you
>>>> have the time/energy to put it up on the Wiki? Anyone can edit it by
>>>> creating
>>>> a logon. If you don't, would it be OK if someone else did it (with
>>>> attribution,
>>>> of course)? I guess that by bringing it up I'm volunteering :)...
>>>>
>>>> Best
>>>> Erick
>>>>
>>>> On Sun, Sep 12, 2010 at 12:26 PM, Peter
>>>> Sturge<pe...@gmail.com>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Below are some notes regarding Solr cache tuning that should prove
>>>>> useful for anyone who uses Solr with frequent commits (e.g.<5min).
>>>>>
>>>>> Environment:
>>>>> Solr 1.4.1 or branch_3x trunk.
>>>>> Note the 4.x trunk has lots of neat new features, so the notes here
>>>>> are likely less relevant to the 4.x environment.
>>>>>
>>>>> Overview:
>>>>> Our Solr environment makes extensive use of faceting, we perform
>>>>> commits every 30secs, and the indexes tend be on the large-ish side
>>>>> (>20million docs).
>>>>> Note: For our data, when we commit, we are always adding new data,
>>>>> never changing existing data.
>>>>> This type of environment can be tricky to tune, as Solr is more
>>>>> geared
>>>>> toward fast reads than frequent writes.
>>>>>
>>>>> Symptoms:
>>>>> If anyone has used faceting in searches where you are also performing
>>>>> frequent commits, you've likely encountered the dreaded
>>>>> OutOfMemory or
>>>>> GC Overhead Exeeded errors.
>>>>> In high commit rate environments, this is almost always due to
>>>>> multiple 'onDeck' searchers and autowarming - i.e. new searchers
>>>>> don't
>>>>> finish autowarming their caches before the next commit()
>>>>> comes along and invalidates them.
>>>>> Once this starts happening on a regular basis, it is likely your
>>>>> Solr's JVM will run out of memory eventually, as the number of
>>>>> searchers (and their cache arrays) will keep growing until the JVM
>>>>> dies of thirst.
>>>>> To check if your Solr environment is suffering from this, turn on
>>>>> INFO
>>>>> level logging, and look for: 'PERFORMANCE WARNING: Overlapping
>>>>> onDeckSearchers=x'.
>>>>>
>>>>> In tests, we've only ever seen this problem when using faceting, and
>>>>> facet.method=fc.
>>>>>
>>>>> Some solutions to this are:
>>>>> Reduce the commit rate to allow searchers to fully warm before
>>>>> the
>>>>> next commit
>>>>> Reduce or eliminate the autowarming in caches
>>>>> Both of the above
>>>>>
>>>>> The trouble is, if you're doing NRT commits, you likely have a good
>>>>> reason for it, and reducing/elimintating autowarming will very
>>>>> significantly impact search performance in high commit rate
>>>>> environments.
>>>>>
>>>>> Solution:
>>>>> Here are some setup steps we've used that allow lots of faceting (we
>>>>> typically search with at least 20-35 different facet fields, and date
>>>>> faceting/sorting) on large indexes, and still keep decent search
>>>>> performance:
>>>>>
>>>>> 1. Firstly, you should consider using the enum method for facet
>>>>> searches (facet.method=enum) unless you've got A LOT of memory on
>>>>> your
>>>>> machine. In our tests, this method uses a lot less memory and
>>>>> autowarms more quickly than fc. (Note, I've not tried the new
>>>>> segement-based 'fcs' option, as I can't find support for it in
>>>>> branch_3x - looks nice for 4.x though)
>>>>> Admittedly, for our data, enum is not quite as fast for searching as
>>>>> fc, but short of purchsing a Thaiwanese RAM factory, it's a
>>>>> worthwhile
>>>>> tradeoff.
>>>>> If you do have access to LOTS of memory, AND you can guarantee that
>>>>> the index won't grow beyond the memory capacity (i.e. you have some
>>>>> sort of deletion policy in place), fc can be a lot faster than enum
>>>>> when searching with lots of facets across many terms.
>>>>>
>>>>> 2. Secondly, we've found that LRUCache is faster at autowarming than
>>>>> FastLRUCache - in our tests, about 20% faster. Maybe this is just our
>>>>> environment - your mileage may vary.
>>>>>
>>>>> So, our filterCache section in solrconfig.xml looks like this:
>>>>> <filterCache
>>>>> class="solr.LRUCache"
>>>>> size="3600"
>>>>> initialSize="1400"
>>>>> autowarmCount="3600"/>
>>>>>
>>>>> For a 28GB index, running in a quad-core x64 VMWare instance, 30
>>>>> warmed facet fields, Solr is running at ~4GB. Stats filterCache size
>>>>> shows usually in the region of ~2400.
>>>>>
>>>>> 3. It's also a good idea to have some sort of
>>>>> firstSearcher/newSearcher event listener queries to allow new data to
>>>>> populate the caches.
>>>>> Of course, what you put in these is dependent on the facets you
>>>>> need/use.
>>>>> We've found a good combination is a firstSearcher with as many facets
>>>>> in the search as your environment can handle, then a subset of the
>>>>> most common facets for the newSearcher.
>>>>>
>>>>> 4. We also set:
>>>>> <useColdSearcher>true</useColdSearcher>
>>>>> just in case.
>>>>>
>>>>> 5. Another key area for search performance with high commits is to
>>>>> use
>>>>> 2 Solr instances - one for the high commit rate indexing, and one for
>>>>> searching.
>>>>> The read-only searching instance can be a remote replica, or a local
>>>>> read-only instance that reads the same core as the indexing instance
>>>>> (for the latter, you'll need something that periodically refreshes -
>>>>> i.e. runs commit()).
>>>>> This way, you can tune the indexing instance for writing performance
>>>>> and the searching instance as above for max read performance.
>>>>>
>>>>> Using the setup above, we get fantastic searching speed for small
>>>>> facet sets (well under 1sec), and really good searching for large
>>>>> facet sets (a couple of secs depending on index size, number of
>>>>> facets, unique terms etc. etc.),
>>>>> even when searching against largeish indexes (>20million docs).
>>>>> We have yet to see any OOM or GC errors using the techniques above,
>>>>> even in low memory conditions.
>>>>>
>>>>> I hope there are people that find this useful. I know I've spent a
>>>>> lot
>>>>> of time looking for stuff like this, so hopefullly, this will save
>>>>> someone some time.
>>>>>
>>>>>
>>>>> Peter
>>>>>
>>
>>
>
--
http://jetwick.com twitter search prototype
Re: Tuning Solr caches with high commit rates (NRT)
Posted by Jonathan Rochkind <ro...@jhu.edu>.
Awesome. I'm not sure his point 1 about facet.method=enum is still valid
in Solr 1.4+. The "fc" facet.method was changed significantly in 1.4,
and generally no longer takes a lot of memory -- for facets with "many"
unique values, method fc in fact should take less than enum, I think?
Peter Karich wrote:
> Just in case someone is interested:
>
> I put the emails of Peter Sturge with some minor edits in the wiki:
>
> http://wiki.apache.org/solr/NearRealtimeSearchTuning
>
> I found myself search the thread again and again ;-)
>
> Feel free to add and edit content!
>
> Regards,
> Peter.
>
>
>> Hi Erik,
>>
>> I thought this would be good for the wiki, but I've not submitted to
>> the wiki before, so I thought I'd put this info out there first, then
>> add it if it was deemed useful.
>> If you could let me know the procedure for submitting, it probably
>> would be worth getting it into the wiki (couldn't do it straightaway,
>> as I have a lot of projects on at the moment). If you're able/willing
>> to put it on there for me, that would be very kind of you!
>>
>> Thanks!
>> Peter
>>
>>
>> On Sun, Sep 12, 2010 at 5:43 PM, Erick Erickson<er...@gmail.com> wrote:
>>
>>> Peter:
>>>
>>> This kind of information is extremely useful to document, thanks! Do you
>>> have the time/energy to put it up on the Wiki? Anyone can edit it by
>>> creating
>>> a logon. If you don't, would it be OK if someone else did it (with
>>> attribution,
>>> of course)? I guess that by bringing it up I'm volunteering :)...
>>>
>>> Best
>>> Erick
>>>
>>> On Sun, Sep 12, 2010 at 12:26 PM, Peter Sturge<pe...@gmail.com>wrote:
>>>
>>>
>>>> Hi,
>>>>
>>>> Below are some notes regarding Solr cache tuning that should prove
>>>> useful for anyone who uses Solr with frequent commits (e.g.<5min).
>>>>
>>>> Environment:
>>>> Solr 1.4.1 or branch_3x trunk.
>>>> Note the 4.x trunk has lots of neat new features, so the notes here
>>>> are likely less relevant to the 4.x environment.
>>>>
>>>> Overview:
>>>> Our Solr environment makes extensive use of faceting, we perform
>>>> commits every 30secs, and the indexes tend be on the large-ish side
>>>> (>20million docs).
>>>> Note: For our data, when we commit, we are always adding new data,
>>>> never changing existing data.
>>>> This type of environment can be tricky to tune, as Solr is more geared
>>>> toward fast reads than frequent writes.
>>>>
>>>> Symptoms:
>>>> If anyone has used faceting in searches where you are also performing
>>>> frequent commits, you've likely encountered the dreaded OutOfMemory or
>>>> GC Overhead Exeeded errors.
>>>> In high commit rate environments, this is almost always due to
>>>> multiple 'onDeck' searchers and autowarming - i.e. new searchers don't
>>>> finish autowarming their caches before the next commit()
>>>> comes along and invalidates them.
>>>> Once this starts happening on a regular basis, it is likely your
>>>> Solr's JVM will run out of memory eventually, as the number of
>>>> searchers (and their cache arrays) will keep growing until the JVM
>>>> dies of thirst.
>>>> To check if your Solr environment is suffering from this, turn on INFO
>>>> level logging, and look for: 'PERFORMANCE WARNING: Overlapping
>>>> onDeckSearchers=x'.
>>>>
>>>> In tests, we've only ever seen this problem when using faceting, and
>>>> facet.method=fc.
>>>>
>>>> Some solutions to this are:
>>>> Reduce the commit rate to allow searchers to fully warm before the
>>>> next commit
>>>> Reduce or eliminate the autowarming in caches
>>>> Both of the above
>>>>
>>>> The trouble is, if you're doing NRT commits, you likely have a good
>>>> reason for it, and reducing/elimintating autowarming will very
>>>> significantly impact search performance in high commit rate
>>>> environments.
>>>>
>>>> Solution:
>>>> Here are some setup steps we've used that allow lots of faceting (we
>>>> typically search with at least 20-35 different facet fields, and date
>>>> faceting/sorting) on large indexes, and still keep decent search
>>>> performance:
>>>>
>>>> 1. Firstly, you should consider using the enum method for facet
>>>> searches (facet.method=enum) unless you've got A LOT of memory on your
>>>> machine. In our tests, this method uses a lot less memory and
>>>> autowarms more quickly than fc. (Note, I've not tried the new
>>>> segement-based 'fcs' option, as I can't find support for it in
>>>> branch_3x - looks nice for 4.x though)
>>>> Admittedly, for our data, enum is not quite as fast for searching as
>>>> fc, but short of purchsing a Thaiwanese RAM factory, it's a worthwhile
>>>> tradeoff.
>>>> If you do have access to LOTS of memory, AND you can guarantee that
>>>> the index won't grow beyond the memory capacity (i.e. you have some
>>>> sort of deletion policy in place), fc can be a lot faster than enum
>>>> when searching with lots of facets across many terms.
>>>>
>>>> 2. Secondly, we've found that LRUCache is faster at autowarming than
>>>> FastLRUCache - in our tests, about 20% faster. Maybe this is just our
>>>> environment - your mileage may vary.
>>>>
>>>> So, our filterCache section in solrconfig.xml looks like this:
>>>> <filterCache
>>>> class="solr.LRUCache"
>>>> size="3600"
>>>> initialSize="1400"
>>>> autowarmCount="3600"/>
>>>>
>>>> For a 28GB index, running in a quad-core x64 VMWare instance, 30
>>>> warmed facet fields, Solr is running at ~4GB. Stats filterCache size
>>>> shows usually in the region of ~2400.
>>>>
>>>> 3. It's also a good idea to have some sort of
>>>> firstSearcher/newSearcher event listener queries to allow new data to
>>>> populate the caches.
>>>> Of course, what you put in these is dependent on the facets you need/use.
>>>> We've found a good combination is a firstSearcher with as many facets
>>>> in the search as your environment can handle, then a subset of the
>>>> most common facets for the newSearcher.
>>>>
>>>> 4. We also set:
>>>> <useColdSearcher>true</useColdSearcher>
>>>> just in case.
>>>>
>>>> 5. Another key area for search performance with high commits is to use
>>>> 2 Solr instances - one for the high commit rate indexing, and one for
>>>> searching.
>>>> The read-only searching instance can be a remote replica, or a local
>>>> read-only instance that reads the same core as the indexing instance
>>>> (for the latter, you'll need something that periodically refreshes -
>>>> i.e. runs commit()).
>>>> This way, you can tune the indexing instance for writing performance
>>>> and the searching instance as above for max read performance.
>>>>
>>>> Using the setup above, we get fantastic searching speed for small
>>>> facet sets (well under 1sec), and really good searching for large
>>>> facet sets (a couple of secs depending on index size, number of
>>>> facets, unique terms etc. etc.),
>>>> even when searching against largeish indexes (>20million docs).
>>>> We have yet to see any OOM or GC errors using the techniques above,
>>>> even in low memory conditions.
>>>>
>>>> I hope there are people that find this useful. I know I've spent a lot
>>>> of time looking for stuff like this, so hopefullly, this will save
>>>> someone some time.
>>>>
>>>>
>>>> Peter
>>>>
>>>>
>
>
>