JVM Heap Memory Increase (SOLR CLOUD)

[Doss <it...@gmail.com>]

We have SOLR(7.0.1) cloud 3 VM Linux instances wit 4 CPU, 90 GB RAM with
zookeeper (3.4.11) ensemble running on the same machines. We have 130 cores of
overall size of 45GB. No Sharding, almost all VMs has the same copy of data.
These nodes are under LB.

http://lucene.472066.n3.nabble.com/SOLR-Cloud-1500-threads-are-in-TIMED-WAITING-status-td4383636.html
- Refer this for Merge and commit configs

JVM Heap Size : 15GB

Optimize : Daily Once

GC Config: Default

-XX:+PrintGCApplicationStoppedTime
-XX:+PrintGCDateStamps
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintHeapAtGC
-XX:+PrintTenuringDistribution
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseConcMarkSweepGC
-XX:+UseGCLogFileRotation
-XX:+UseParNewGC
-XX:-OmitStackTraceInFastThrow
-XX:CMSInitiatingOccupancyFraction=50
-XX:CMSMaxAbortablePrecleanTime=6000
-XX:ConcGCThreads=4
-XX:GCLogFileSize=20M
-XX:MaxTenuringThreshold=8
-XX:NewRatio=3
-XX:NumberOfGCLogFiles=9
-XX:OnOutOfMemoryError=/home/solr/bin/oom_solr.sh 8980
/data/solr/server/logs
-XX:ParallelGCThreads=4

ISSUE: After running 8 to 9 hours the JVM heap memory keeps on increasing,
at that time if we did optimize then I am seeing 3 to 3.5 gb reduction, but
doing optimize at day time will be a problem, on the other side if the heap
got full then OOM exception is happening and the cloud crashes.

I read somewhere that G1GC will give better results, but SOLR experts
doesn't encourage to use it

what else we can do to resolve this issue?

Thanks,
Doss.

Re: JVM Heap Memory Increase (SOLR CLOUD)

[Erick Erickson <er...@gmail.com>]

G1GC will not solve OOM problems, it just _may_ give better GC performance.

I'm assuming you're actively adding documents to this index, and
further assuming that you're replacing some existing documents.

If those assumptions are true, here's what I think is happening:
- You optimize, which creates one very large segment.
- You're adding docs over the course of the day. you'll note that the
"maxdoc" number will keep increasing.
- Because you have such a large segment, the deleted docs won't be
merged away for quite a long time, so over the course of the day you
keep accumulating deleted docs.
- Certain structures in Solr are sensitive to "maxDoc" and increase as
maxDoc increases
- You're running close to your limit of Java heap so get OOMs

So while the optimize may let you run, it's also the source of much of
your instability.

Your choices are:
1> get more memory. Having 45G of indexes and running in 15G of RAM
indicates you're probably running close to your heap limits in the
best of circumstances.
2> re-index from scratch and do _not_ optimize. Tune your system to
remove the need to optimize
3> insure you're using the most heap-friendly, particularly any field
used for grouping, faceting or sorting should have docValues set to
true.

Best,
Erick
background info about the above:
https://lucidworks.com/2017/10/13/segment-merging-deleted-documents-optimize-may-bad/

On Fri, Apr 20, 2018 at 10:34 PM, Doss <it...@gmail.com> wrote:
> We have SOLR(7.0.1) cloud 3 VM Linux instances wit 4 CPU, 90 GB RAM with
> zookeeper (3.4.11) ensemble running on the same machines. We have 130 cores of
> overall size of 45GB. No Sharding, almost all VMs has the same copy of data.
> These nodes are under LB.
>
> http://lucene.472066.n3.nabble.com/SOLR-Cloud-1500-threads-are-in-TIMED-WAITING-status-td4383636.html
> - Refer this for Merge and commit configs
>
> JVM Heap Size : 15GB
>
> Optimize : Daily Once
>
> GC Config: Default
>
> -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintGCDateStamps
> -XX:+PrintGCDetails
> -XX:+PrintGCTimeStamps
> -XX:+PrintHeapAtGC
> -XX:+PrintTenuringDistribution
> -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseConcMarkSweepGC
> -XX:+UseGCLogFileRotation
> -XX:+UseParNewGC
> -XX:-OmitStackTraceInFastThrow
> -XX:CMSInitiatingOccupancyFraction=50
> -XX:CMSMaxAbortablePrecleanTime=6000
> -XX:ConcGCThreads=4
> -XX:GCLogFileSize=20M
> -XX:MaxTenuringThreshold=8
> -XX:NewRatio=3
> -XX:NumberOfGCLogFiles=9
> -XX:OnOutOfMemoryError=/home/solr/bin/oom_solr.sh 8980
> /data/solr/server/logs
> -XX:ParallelGCThreads=4
>
> ISSUE: After running 8 to 9 hours the JVM heap memory keeps on increasing,
> at that time if we did optimize then I am seeing 3 to 3.5 gb reduction, but
> doing optimize at day time will be a problem, on the other side if the heap
> got full then OOM exception is happening and the cloud crashes.
>
> I read somewhere that G1GC will give better results, but SOLR experts
> doesn't encourage to use it
>
> what else we can do to resolve this issue?
>
> Thanks,
> Doss.