You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Rallavagu <ra...@gmail.com> on 2016/08/25 23:56:32 UTC
Re: solr.NRTCachingDirectoryFactory
Follow up update ...
Set autowarm count to zero for caches for NRT and I could negotiate
latency from 2 min to 5 min :)
However, still seeing high QTimes and wondering where else can I look?
Should I debug the code or run some tools to isolate bottlenecks (disk
I/O, CPU or Query itself). Looking for some tuning advice. Thanks.
On 7/26/16 9:42 AM, Erick Erickson wrote:
> And, I might add, you should look through your old logs
> and see how long it takes to open a searcher. Let's
> say Shawn's lower bound is what you see, i.e.
> it takes a minute each to execute all the autowarming
> in filterCache and queryResultCache... So you're current
> latency is _at least_ 2 minutes between the time something
> is indexed and it's available for search just for autowarming.
>
> Plus up to another 2 minutes for your soft commit interval
> to expire.
>
> So if your business people haven't noticed a 4 minute
> latency yet, tell them they don't know what they're talking
> about when they insist on the NRT interval being a few
> seconds ;).
>
> Best,
> Erick
>
> On Tue, Jul 26, 2016 at 7:20 AM, Rallavagu <ra...@gmail.com> wrote:
>>
>>
>> On 7/26/16 5:46 AM, Shawn Heisey wrote:
>>>
>>> On 7/22/2016 10:15 AM, Rallavagu wrote:
>>>>
>>>> <filterCache class="solr.FastLRUCache"
>>>> size="5000"
>>>> initialSize="5000"
>>>> autowarmCount="500"/>
>>>>
>>>
>>>> <queryResultCache class="solr.LRUCache"
>>>> size="20000"
>>>> initialSize="20000"
>>>> autowarmCount="500"/>
>>>
>>>
>>> As Erick indicated, these settings are incompatible with Near Real Time
>>> updates.
>>>
>>> With those settings, every time you commit and create a new searcher,
>>> Solr will execute up to 1000 queries (potentially 500 for each of the
>>> caches above) before that new searcher will begin returning new results.
>>>
>>> I do not know how fast your filter queries execute when they aren't
>>> cached... but even if they only take 100 milliseconds each, that's could
>>> take up to a minute for filterCache warming. If each one takes two
>>> seconds and there are 500 entries in the cache, then autowarming the
>>> filterCache would take nearly 17 minutes. You would also need to wait
>>> for the warming queries on queryResultCache.
>>>
>>> The autowarmCount on my filterCache is 4, and warming that cache *still*
>>> sometimes takes ten or more seconds to complete.
>>>
>>> If you want true NRT, you need to set all your autowarmCount values to
>>> zero. The tradeoff with NRT is that your caches are ineffective
>>> immediately after a new searcher is created.
>>
>> Will look into this and make changes as suggested.
>>
>>>
>>> Looking at the "top" screenshot ... you have plenty of memory to cache
>>> the entire index. Unless your queries are extreme, this is usually
>>> enough for good performance.
>>>
>>> One possible problem is that cache warming is taking far longer than
>>> your autoSoftCommit interval, and the server is constantly busy making
>>> thousands of warming queries. Reducing autowarmCount, possibly to zero,
>>> *might* fix that. I would expect higher CPU load than what your
>>> screenshot shows if this were happening, but it still might be the
>>> problem.
>>
>> Great point. Thanks for the help.
>>
>>>
>>> Thanks,
>>> Shawn
>>>
>>
Re: solr.NRTCachingDirectoryFactory
Posted by Rallavagu <ra...@gmail.com>.
Thanks Michail.
I am unable to locate bottleneck so far. Will try jstack and other tools.
On 8/25/16 11:40 PM, Mikhail Khludnev wrote:
> Rough sampling under load makes sense as usual. JMC is one of the suitable
> tools for this.
> Sometimes even just jstack <PID> or looking at SolrAdmin/Threads is enough.
> If the only small ratio of documents is updated and a bottleneck is
> filterCache you can experiment with segmened filters which suite more for
> NRT.
> http://blog-archive.griddynamics.com/2014/01/segmented-filter-cache-in-solr.html
>
>
> On Fri, Aug 26, 2016 at 2:56 AM, Rallavagu <ra...@gmail.com> wrote:
>
>> Follow up update ...
>>
>> Set autowarm count to zero for caches for NRT and I could negotiate
>> latency from 2 min to 5 min :)
>>
>> However, still seeing high QTimes and wondering where else can I look?
>> Should I debug the code or run some tools to isolate bottlenecks (disk I/O,
>> CPU or Query itself). Looking for some tuning advice. Thanks.
>>
>>
>> On 7/26/16 9:42 AM, Erick Erickson wrote:
>>
>>> And, I might add, you should look through your old logs
>>> and see how long it takes to open a searcher. Let's
>>> say Shawn's lower bound is what you see, i.e.
>>> it takes a minute each to execute all the autowarming
>>> in filterCache and queryResultCache... So you're current
>>> latency is _at least_ 2 minutes between the time something
>>> is indexed and it's available for search just for autowarming.
>>>
>>> Plus up to another 2 minutes for your soft commit interval
>>> to expire.
>>>
>>> So if your business people haven't noticed a 4 minute
>>> latency yet, tell them they don't know what they're talking
>>> about when they insist on the NRT interval being a few
>>> seconds ;).
>>>
>>> Best,
>>> Erick
>>>
>>> On Tue, Jul 26, 2016 at 7:20 AM, Rallavagu <ra...@gmail.com> wrote:
>>>
>>>>
>>>>
>>>> On 7/26/16 5:46 AM, Shawn Heisey wrote:
>>>>
>>>>>
>>>>> On 7/22/2016 10:15 AM, Rallavagu wrote:
>>>>>
>>>>>>
>>>>>> <filterCache class="solr.FastLRUCache"
>>>>>> size="5000"
>>>>>> initialSize="5000"
>>>>>> autowarmCount="500"/>
>>>>>>
>>>>>>
>>>>> <queryResultCache class="solr.LRUCache"
>>>>>> size="20000"
>>>>>> initialSize="20000"
>>>>>> autowarmCount="500"/>
>>>>>>
>>>>>
>>>>>
>>>>> As Erick indicated, these settings are incompatible with Near Real Time
>>>>> updates.
>>>>>
>>>>> With those settings, every time you commit and create a new searcher,
>>>>> Solr will execute up to 1000 queries (potentially 500 for each of the
>>>>> caches above) before that new searcher will begin returning new results.
>>>>>
>>>>> I do not know how fast your filter queries execute when they aren't
>>>>> cached... but even if they only take 100 milliseconds each, that's could
>>>>> take up to a minute for filterCache warming. If each one takes two
>>>>> seconds and there are 500 entries in the cache, then autowarming the
>>>>> filterCache would take nearly 17 minutes. You would also need to wait
>>>>> for the warming queries on queryResultCache.
>>>>>
>>>>> The autowarmCount on my filterCache is 4, and warming that cache *still*
>>>>> sometimes takes ten or more seconds to complete.
>>>>>
>>>>> If you want true NRT, you need to set all your autowarmCount values to
>>>>> zero. The tradeoff with NRT is that your caches are ineffective
>>>>> immediately after a new searcher is created.
>>>>>
>>>>
>>>> Will look into this and make changes as suggested.
>>>>
>>>>
>>>>> Looking at the "top" screenshot ... you have plenty of memory to cache
>>>>> the entire index. Unless your queries are extreme, this is usually
>>>>> enough for good performance.
>>>>>
>>>>> One possible problem is that cache warming is taking far longer than
>>>>> your autoSoftCommit interval, and the server is constantly busy making
>>>>> thousands of warming queries. Reducing autowarmCount, possibly to zero,
>>>>> *might* fix that. I would expect higher CPU load than what your
>>>>> screenshot shows if this were happening, but it still might be the
>>>>> problem.
>>>>>
>>>>
>>>> Great point. Thanks for the help.
>>>>
>>>>
>>>>> Thanks,
>>>>> Shawn
>>>>>
>>>>>
>>>>
>
>
Re: solr.NRTCachingDirectoryFactory
Posted by Mikhail Khludnev <mk...@apache.org>.
Rough sampling under load makes sense as usual. JMC is one of the suitable
tools for this.
Sometimes even just jstack <PID> or looking at SolrAdmin/Threads is enough.
If the only small ratio of documents is updated and a bottleneck is
filterCache you can experiment with segmened filters which suite more for
NRT.
http://blog-archive.griddynamics.com/2014/01/segmented-filter-cache-in-solr.html
On Fri, Aug 26, 2016 at 2:56 AM, Rallavagu <ra...@gmail.com> wrote:
> Follow up update ...
>
> Set autowarm count to zero for caches for NRT and I could negotiate
> latency from 2 min to 5 min :)
>
> However, still seeing high QTimes and wondering where else can I look?
> Should I debug the code or run some tools to isolate bottlenecks (disk I/O,
> CPU or Query itself). Looking for some tuning advice. Thanks.
>
>
> On 7/26/16 9:42 AM, Erick Erickson wrote:
>
>> And, I might add, you should look through your old logs
>> and see how long it takes to open a searcher. Let's
>> say Shawn's lower bound is what you see, i.e.
>> it takes a minute each to execute all the autowarming
>> in filterCache and queryResultCache... So you're current
>> latency is _at least_ 2 minutes between the time something
>> is indexed and it's available for search just for autowarming.
>>
>> Plus up to another 2 minutes for your soft commit interval
>> to expire.
>>
>> So if your business people haven't noticed a 4 minute
>> latency yet, tell them they don't know what they're talking
>> about when they insist on the NRT interval being a few
>> seconds ;).
>>
>> Best,
>> Erick
>>
>> On Tue, Jul 26, 2016 at 7:20 AM, Rallavagu <ra...@gmail.com> wrote:
>>
>>>
>>>
>>> On 7/26/16 5:46 AM, Shawn Heisey wrote:
>>>
>>>>
>>>> On 7/22/2016 10:15 AM, Rallavagu wrote:
>>>>
>>>>>
>>>>> <filterCache class="solr.FastLRUCache"
>>>>> size="5000"
>>>>> initialSize="5000"
>>>>> autowarmCount="500"/>
>>>>>
>>>>>
>>>> <queryResultCache class="solr.LRUCache"
>>>>> size="20000"
>>>>> initialSize="20000"
>>>>> autowarmCount="500"/>
>>>>>
>>>>
>>>>
>>>> As Erick indicated, these settings are incompatible with Near Real Time
>>>> updates.
>>>>
>>>> With those settings, every time you commit and create a new searcher,
>>>> Solr will execute up to 1000 queries (potentially 500 for each of the
>>>> caches above) before that new searcher will begin returning new results.
>>>>
>>>> I do not know how fast your filter queries execute when they aren't
>>>> cached... but even if they only take 100 milliseconds each, that's could
>>>> take up to a minute for filterCache warming. If each one takes two
>>>> seconds and there are 500 entries in the cache, then autowarming the
>>>> filterCache would take nearly 17 minutes. You would also need to wait
>>>> for the warming queries on queryResultCache.
>>>>
>>>> The autowarmCount on my filterCache is 4, and warming that cache *still*
>>>> sometimes takes ten or more seconds to complete.
>>>>
>>>> If you want true NRT, you need to set all your autowarmCount values to
>>>> zero. The tradeoff with NRT is that your caches are ineffective
>>>> immediately after a new searcher is created.
>>>>
>>>
>>> Will look into this and make changes as suggested.
>>>
>>>
>>>> Looking at the "top" screenshot ... you have plenty of memory to cache
>>>> the entire index. Unless your queries are extreme, this is usually
>>>> enough for good performance.
>>>>
>>>> One possible problem is that cache warming is taking far longer than
>>>> your autoSoftCommit interval, and the server is constantly busy making
>>>> thousands of warming queries. Reducing autowarmCount, possibly to zero,
>>>> *might* fix that. I would expect higher CPU load than what your
>>>> screenshot shows if this were happening, but it still might be the
>>>> problem.
>>>>
>>>
>>> Great point. Thanks for the help.
>>>
>>>
>>>> Thanks,
>>>> Shawn
>>>>
>>>>
>>>
--
Sincerely yours
Mikhail Khludnev