swap and GC

[Bernd Fehling <be...@uni-bielefeld.de>]

Something interesting I have noticed today,
after running my huge single index (49 mio. records / 137 GB index) for
about a week and replicating today I recognized that the heap usage after
replication did not go down as expected. Expected means if solr is started
I have a heap size between 4 to 5 GB and during the week under heavy load
it might go up to 10 GB. But after replication in offline mode it recovers
to between 5 to 6 GB. But today it was not going under 8 GB, even with
forced GC from jvisualvm.
So I first dropped the caches and tried again, no success.
Next I turned off swap which took quite a while and turned it back on.
This forced all content from swap back into memory. After calling
Perform GC from jvisualvm the heap dropped below 5 GB. Bingo!

This leads me to the conclusion that java GC is not "seeing" or "reaching"
objects which are located in swap.

Anyone else seen this?

As I am not short on memory or have any other problems I don't need any solution,
but if there are some users having memory problems with old objects in swap
I would suggest a cronjob after replication with swapoff/swapon and GC afterwards.

Bernd

Re: swap and GC

[Bernd Fehling <be...@uni-bielefeld.de>]

Am 29.07.2013 14:46, schrieb Michael Ryan:
> This is interesting... How are you measuring the heap size?

This is displayed in jvisualvm and also logged with munin via JMX.

Bernd

> 
> -Michael
> 
> -----Original Message-----
> From: Bernd Fehling [mailto:bernd.fehling@uni-bielefeld.de] 
> Sent: Monday, July 29, 2013 5:34 AM
> To: solr-user@lucene.apache.org
> Subject: swap and GC
> 
> Something interesting I have noticed today, after running my huge single index (49 mio. records / 137 GB index) for about a week and replicating today I recognized that the heap usage after replication did not go down as expected. Expected means if solr is started I have a heap size between 4 to 5 GB and during the week under heavy load it might go up to 10 GB. But after replication in offline mode it recovers to between 5 to 6 GB. But today it was not going under 8 GB, even with forced GC from jvisualvm.
> So I first dropped the caches and tried again, no success.
> Next I turned off swap which took quite a while and turned it back on.
> This forced all content from swap back into memory. After calling Perform GC from jvisualvm the heap dropped below 5 GB. Bingo!
> 
> This leads me to the conclusion that java GC is not "seeing" or "reaching"
> objects which are located in swap.
> 
> Anyone else seen this?
> 
> As I am not short on memory or have any other problems I don't need any solution, but if there are some users having memory problems with old objects in swap I would suggest a cronjob after replication with swapoff/swapon and GC afterwards.
> 
> Bernd
> 

RE: swap and GC

[Michael Ryan <mr...@moreover.com>]

This is interesting... How are you measuring the heap size?

-Michael

-----Original Message-----
From: Bernd Fehling [mailto:bernd.fehling@uni-bielefeld.de] 
Sent: Monday, July 29, 2013 5:34 AM
To: solr-user@lucene.apache.org
Subject: swap and GC

Something interesting I have noticed today, after running my huge single index (49 mio. records / 137 GB index) for about a week and replicating today I recognized that the heap usage after replication did not go down as expected. Expected means if solr is started I have a heap size between 4 to 5 GB and during the week under heavy load it might go up to 10 GB. But after replication in offline mode it recovers to between 5 to 6 GB. But today it was not going under 8 GB, even with forced GC from jvisualvm.
So I first dropped the caches and tried again, no success.
Next I turned off swap which took quite a while and turned it back on.
This forced all content from swap back into memory. After calling Perform GC from jvisualvm the heap dropped below 5 GB. Bingo!

This leads me to the conclusion that java GC is not "seeing" or "reaching"
objects which are located in swap.

Anyone else seen this?

As I am not short on memory or have any other problems I don't need any solution, but if there are some users having memory problems with old objects in swap I would suggest a cronjob after replication with swapoff/swapon and GC afterwards.

Bernd