Re: Possible Memory Leaks / Upgrading to a Later Version of Solr or Lucene

[Markus Jelsma <ma...@openindex.io>]

Heap usage can spike after a commit. Existing caches are still in use and new 
caches are being generated and/or auto warmed. Can you confirm this is the 
case?

On Friday 28 January 2011 00:34:42 Simon Wistow wrote:
> On Tue, Jan 25, 2011 at 01:28:16PM +0100, Markus Jelsma said:
> > Are you sure you need CMS incremental mode? It's only adviced when
> > running on a machine with one or two processors. If you have more you
> > should consider disabling the incremental flags.
> 
> I'll test agin but we added those to get better performance - not much
> but there did seem to be an improvement.
> 
> The problem seems to not be in average use but that occasionally there's
> huge spike in load (there doesn't seem to be a particular "killer
> query") and Solr just never recovers.
> 
> Thanks,
> 
> Simon

-- 
Markus Jelsma - CTO - Openindex
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

Re: Possible Memory Leaks / Upgrading to a Later Version of Solr or Lucene

[Simon Wistow <si...@thegestalt.org>]

On Wed, Apr 06, 2011 at 12:05:57AM +0200, Jan Høydahl said:
> Just curious, was there any resolution to this?

Not really.

We tuned the GC pretty aggressively - we use these options

-server 
-Xmx20G -Xms20G -Xss10M
-XX:+UseConcMarkSweepGC 
-XX:+UseParNewGC 
-XX:+CMSIncrementalMode 
-XX:+CMSIncrementalPacing
-XX:SoftRefLRUPolicyMSPerMB=10

and we've played a little with CompressOops and AggressiveOpts.

We also backported the MMapDirectory factory to 1.4.1 and that helped a 
lot.

We do still gets spikes of long (5s-20s queries) a few times an hour 
which don't appear to be caused by any kind of "Query of Death". 
Occasionally (once every few days) one of the slaves will experience a 
period of sustained slowness but recovers by itself in less than a 
minute.

According to our GC logs we haven't had a full GC for a long time. 

Currently the state of play is that we commit on our master every 5000ms 
and replicate from the slaves every 2 minutes. Our reponse times for 
searches on the slaves are about 180-270ms but if we turn off 
replication then we get 60-90ms. So something is clearly "up" with that.

Having talked to the good people at Lucid we're going to try playing 
around with commit intervals, upping our mergeFactor from 10 to 25 and 
maybe using the BalancedSegmentMergePolicy. 

The system seems to be stable at the moment which is good but obviously 
we'd like to lower our query times if possible.

Hopefully this might be of some use to somebody out there, sometime.

Simon

Re: Possible Memory Leaks / Upgrading to a Later Version of Solr or Lucene

[Jan Høydahl <ja...@cominvent.com>]

Hi,

Just curious, was there any resolution to this?

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

On 8. feb. 2011, at 03.40, Markus Jelsma wrote:

> Do you have GC logging enabled? Tail -f the log file and you'll see what CMS is 
> telling you. Tuning the occupation fraction of the tenured generation to a 
> lower value than default and telling the JVM to only use your value to 
> initiate a collection can help a lot. The same goes for sizing the young 
> generation and sometimes the survivor ratio.
> 
> Consult the HotSpot CMS settings and young generation (or new) sizes. They are 
> very important.
> 
> If you have multiple slaves under the same load you can easily try different 
> configurations. Keeping an eye on the nodes with a tool like JConsole and at 
> the same time tailing the GC log will help a lot. Don't forget to send updates 
> and frequent commits or you won't be able to replay. I've never seen a Solr 
> instance go down under heavy load and without commits but they tend to behave 
> badly when commits occur while under heavy load with long cache warming times 
> (and heap consumption).
> 
> You might also be suffering from memory fragmentation, this is bad and can lead 
> to failure. You can configure the JVM to fore a compaction before a GC, that's 
> nice but it does consume CPU time.
> 
> A query of death can, in theory, also happen when you sort on a very large 
> dataset that isn't optimized, in this case the maxDoc value is too high.
> 
> Anyway, try some settings and monitor the nodes and please report your 
> findings.
> 
>> On Mon, Feb 07, 2011 at 02:06:00PM +0100, Markus Jelsma said:
>>> Heap usage can spike after a commit. Existing caches are still in use and
>>> new caches are being generated and/or auto warmed. Can you confirm this
>>> is the case?
>> 
>> We see spikes after replication which I suspect is, as you say, because
>> of the ensuing commit.
>> 
>> What we seem to have found is that when we weren't using the Concurrent
>> GC stop-the-world gc runs would kill the app. Now that we're using CMS
>> we occasionally find ourselves in situations where the app still has
>> memory "left over" but the load on the machine spikes, the GC duty cycle
>> goes to 100 and the app never recovers.> 
>> Restarting usually helps but sometimes we have to take the machine out
>> of the laod balancer, wait for a number of minutes and then out it back
>> in.
>> 
>> We're working on two hypotheses
>> 
>> Firstly - we're CPU bound somehow and that at some point we cross some
>> threshhold and GC or something else is just unable to to keep up. So
>> whilst it looks like instantaneous death of the app it's actually
>> gradual resource exhaustion where the definition of 'gradual' is 'a very
>> short period of time' (as opposed to some cataclysmic infinite loop bug
>> somewhere).
>> 
>> Either that or ... Secondly - there's some sort of Query Of Death that
>> kills machines. We just haven't found it yet, even when replaying logs.
>> 
>> Or some combination of both. Or other things. It's maddeningly
>> frustrating.
>> 
>> We're also got to try deploying a custom solr.war and try using the
>> MMapDirectory to see if that helps with anything.

Re: Possible Memory Leaks / Upgrading to a Later Version of Solr or Lucene

[Markus Jelsma <ma...@openindex.io>]

Do you have GC logging enabled? Tail -f the log file and you'll see what CMS is 
telling you. Tuning the occupation fraction of the tenured generation to a 
lower value than default and telling the JVM to only use your value to 
initiate a collection can help a lot. The same goes for sizing the young 
generation and sometimes the survivor ratio.

Consult the HotSpot CMS settings and young generation (or new) sizes. They are 
very important.

If you have multiple slaves under the same load you can easily try different 
configurations. Keeping an eye on the nodes with a tool like JConsole and at 
the same time tailing the GC log will help a lot. Don't forget to send updates 
and frequent commits or you won't be able to replay. I've never seen a Solr 
instance go down under heavy load and without commits but they tend to behave 
badly when commits occur while under heavy load with long cache warming times 
(and heap consumption).

You might also be suffering from memory fragmentation, this is bad and can lead 
to failure. You can configure the JVM to fore a compaction before a GC, that's 
nice but it does consume CPU time.

A query of death can, in theory, also happen when you sort on a very large 
dataset that isn't optimized, in this case the maxDoc value is too high.

Anyway, try some settings and monitor the nodes and please report your 
findings.

> On Mon, Feb 07, 2011 at 02:06:00PM +0100, Markus Jelsma said:
> > Heap usage can spike after a commit. Existing caches are still in use and
> > new caches are being generated and/or auto warmed. Can you confirm this
> > is the case?
> 
> We see spikes after replication which I suspect is, as you say, because
> of the ensuing commit.
> 
> What we seem to have found is that when we weren't using the Concurrent
> GC stop-the-world gc runs would kill the app. Now that we're using CMS
> we occasionally find ourselves in situations where the app still has
> memory "left over" but the load on the machine spikes, the GC duty cycle
> goes to 100 and the app never recovers.> 
> Restarting usually helps but sometimes we have to take the machine out
> of the laod balancer, wait for a number of minutes and then out it back
> in.
> 
> We're working on two hypotheses
> 
> Firstly - we're CPU bound somehow and that at some point we cross some
> threshhold and GC or something else is just unable to to keep up. So
> whilst it looks like instantaneous death of the app it's actually
> gradual resource exhaustion where the definition of 'gradual' is 'a very
> short period of time' (as opposed to some cataclysmic infinite loop bug
> somewhere).
> 
> Either that or ... Secondly - there's some sort of Query Of Death that
> kills machines. We just haven't found it yet, even when replaying logs.
> 
> Or some combination of both. Or other things. It's maddeningly
> frustrating.
> 
> We're also got to try deploying a custom solr.war and try using the
> MMapDirectory to see if that helps with anything.

Re: Possible Memory Leaks / Upgrading to a Later Version of Solr or Lucene

[Simon Wistow <si...@thegestalt.org>]

On Mon, Feb 07, 2011 at 02:06:00PM +0100, Markus Jelsma said:
> Heap usage can spike after a commit. Existing caches are still in use and new 
> caches are being generated and/or auto warmed. Can you confirm this is the 
> case?

We see spikes after replication which I suspect is, as you say, because 
of the ensuing commit.

What we seem to have found is that when we weren't using the Concurrent 
GC stop-the-world gc runs would kill the app. Now that we're using CMS 
we occasionally find ourselves in situations where the app still has 
memory "left over" but the load on the machine spikes, the GC duty cycle 
goes to 100 and the app never recovers.

Restarting usually helps but sometimes we have to take the machine out 
of the laod balancer, wait for a number of minutes and then out it back 
in.

We're working on two hypotheses 

Firstly - we're CPU bound somehow and that at some point we cross some 
threshhold and GC or something else is just unable to to keep up. So 
whilst it looks like instantaneous death of the app it's actually 
gradual resource exhaustion where the definition of 'gradual' is 'a very 
short period of time' (as opposed to some cataclysmic infinite loop bug 
somewhere).

Either that or ... Secondly - there's some sort of Query Of Death that 
kills machines. We just haven't found it yet, even when replaying logs. 

Or some combination of both. Or other things. It's maddeningly 
frustrating.

We're also got to try deploying a custom solr.war and try using the 
MMapDirectory to see if that helps with anything.