Major GC does not reduce the old gen size

[neoman <ha...@gmail.com>]

Hello everyone,
We are using solr 4.4 version production with 4 shards. This is our memory
settings.
-d64 -server -Xms8192m -Xmx12288m -XX:MaxPermSize=256m \
-XX:NewRatio=1 -XX:SurvivorRatio=6 \
-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
-XX:CMSIncrementalDutyCycleMin=0 \
-XX:CMSIncrementalDutyCycle=10 -XX:+CMSIncrementalPacing \
-XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
-XX:+PrintHeapAtGC \
-XX:+CMSClassUnloadingEnabled -XX:+DisableExplicitGC \
-XX:+UseLargePages \
-XX:+UseParNewGC \
-XX:ConcGCThreads=10 \
-XX:ParallelGCThreads=10 \
-XX:MaxGCPauseMillis=30000 \
I notice in production that, the old generation becomes full and no amount
of garbage collection will free up the memory
This is similar to the issue discussed in this link. 
http://grokbase.com/t/lucene/solr-user/12bwydq5jr/permanently-full-old-generation
Did anyone have this problem? Can you please point anything wrong with the
GC configuration?



--
View this message in context: http://lucene.472066.n3.nabble.com/Major-GC-does-not-reduce-the-old-gen-size-tp4096880.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Major GC does not reduce the old gen size

[pvar <pv...@paypal.com>]

psoting on behalf of neoman,

alt_schema.xml
<http://lucene.472066.n3.nabble.com/file/n4103405/alt_schema.xml>  

details about index

Number of documents: 350 million
number of shards: 4
number of nodes:8
replicatorFactor: 1(default/no additional replication).
Total RAM on each server node: 16 GB
number of documents per shard: close to 80 million.

documents gets added to index every 15 mins in a batch job and 
yes, as you mentioned updates to documents is also happening



--
View this message in context: http://lucene.472066.n3.nabble.com/Major-GC-does-not-reduce-the-old-gen-size-tp4096880p4103405.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Major GC does not reduce the old gen size

[Shawn Heisey <so...@elyograg.org>]

On 10/21/2013 2:11 PM, neoman wrote:
> We are using solr 4.4 version production with 4 shards. This is our memory
> settings.
> -d64 -server -Xms8192m -Xmx12288m -XX:MaxPermSize=256m \
> -XX:NewRatio=1 -XX:SurvivorRatio=6 \
> -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
> -XX:CMSIncrementalDutyCycleMin=0 \
> -XX:CMSIncrementalDutyCycle=10 -XX:+CMSIncrementalPacing \
> -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
> -XX:+PrintHeapAtGC \
> -XX:+CMSClassUnloadingEnabled -XX:+DisableExplicitGC \
> -XX:+UseLargePages \
> -XX:+UseParNewGC \
> -XX:ConcGCThreads=10 \
> -XX:ParallelGCThreads=10 \
> -XX:MaxGCPauseMillis=30000 \
> I notice in production that, the old generation becomes full and no amount
> of garbage collection will free up the memory
> This is similar to the issue discussed in this link. 
> http://grokbase.com/t/lucene/solr-user/12bwydq5jr/permanently-full-old-generation
> Did anyone have this problem? Can you please point anything wrong with the
> GC configuration?

Your GC configuration is very likely fine.  There are two possibilities
that seem likely to me, and it might be a combination of both:

1) Your index size, configuration, and/or query characteristics require
lots of memory.

2) You are experiencing the SolrCloud update deadlocks that have been
become fairly commonplace, and the blocked threads have resulted in too
much memory being used.

For the first problem, you can either increase your max heap or figure
out how to reduce memory requirements.  Can you share your
solrconfig.xml and details about your index, like number of documents,
total index size, index size per shard replica, number of servers,
replication factor, total RAM installed per server, etc?

http://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap

For the second problem, upgrading to a newer version may be the only
option.  Version 4.5.0 has made the situation better, but some users are
still having problems.  Version 4.5.1 is due out soon, but some of the
fixes might require running a dev 4.6 snapshot.  I think there might be
some deadlock issues that don't have fixes committed yet.

Thanks,
Shawn

Re: Major GC does not reduce the old gen size

[neoman <ha...@gmail.com>]

help please



--
View this message in context: http://lucene.472066.n3.nabble.com/Major-GC-does-not-reduce-the-old-gen-size-tp4096880p4097429.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Major GC does not reduce the old gen size

[neoman <ha...@gmail.com>]

can anyone please reply to this. i appreciate your help.



--
View this message in context: http://lucene.472066.n3.nabble.com/Major-GC-does-not-reduce-the-old-gen-size-tp4096880p4097036.html
Sent from the Solr - User mailing list archive at Nabble.com.