You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Thiru M <m....@gmail.com> on 2016/08/27 15:08:53 UTC
Spike in SOLR Process and Frequent GC
Dear Folks,
We are using Solr 5.4.0 - "stand-alone" mode in our production boxes hosted
in Red Hat Enterprise Linux (RHEL) OS.
Each box have number of different cores. Have attached the screenshot shot
with the Solr core & system details.
1. Earlier indexing was performed every 30 minutes in both production
servers,
2. In linux-a server 30 (stand-alone) cores created on same day and content
indexed into it,
3. we then spotted unusual GC performing every 2 to 7 seconds in linux-a
server and the Solr process spiked,
4. Then we removed the indexing in linux-a server for a week, monitored the
both Solr process and GC.(No indexing performed during this time),
5. No one uses the system during night time which we ensured it from our
end. But both Solr process and GC were in its peak, even "during non
business hours",
6. Have restated Solr instance in linux-a server. GC started again after
Solr instance brought up,
7. In linux-b server no spike in Solr process and no issues with GC,
We couldn't figure out why Solr process is hight and GC performing
frequently in linux-a server.
Have anyone encountered similar issue?
Can anyone help to suggest on ways to spot the issue please ?
Regards,
Thirukumaran M
Re: Spike in SOLR Process and Frequent GC
Posted by Shawn Heisey <ap...@elyograg.org>.
On 8/30/2016 10:28 PM, Thiru M wrote:
> What part of the information you obtained represents a problem in your mind?
>
> � Information obtained from top and the SOLR GC log in the *linux-a*
> server
>
> � Allocated max heap size is 2 GB. Based on the jConsole monitor h*eap
> usage is < 600 MB in linux-a server but the CPU utilization by SOLR process
> is on top (in figures it consumes 0.5 to 5 %)*.
I'm not seeing a problem here. The CPU usage is very low, and only a
third of the heap is consumed. Neither of these seems like a reason to
be concerned.
What are you seeing that prompted you to post your initial question?
The subject mentions frequent GC, but you have not mentioned anything
here about how often GC is happening. Extremely frequent garbage
collection is usually caused by having a heap that's too small.
On server A, you said there are 49 cores, 6 million docs, and 9.9GB. Is
that 6 million docs *per core*, or 6 million docs total? Is it 9.9GB
total, or 9.9GB per core?
If these are totals, then 2GB of heap may not be enough, and I would try
increasing it to 4GB. As Emir said, I would set both Xms and Xmx to 4GB.
If those numbers are per-core, 2GB is *definitely* not enough, and you
should probably start with 16GB.
Thanks,
Shawn
Re: Spike in SOLR Process and Frequent GC
Posted by Emir Arnautovic <em...@sematext.com>.
Hi Thiru,
Two Solrs with different data and usage patterns should be tuned
separately and comparing one to another does not give much value.
Like Shawn suggested, first thing that you can try is increase heap
size. Having different Xms and Xmx is bad practice so make sure it is
set to the same value. Even you have enough RAM on your machines, start
with smaller heap (e.g. 4GB) and monitor JVM. Heap should be just about
large to handle your top load.
You could also use more advanced monitoring tool that will allow you to
correlate Solr activities with JVM/OS metrics. One such tool is our's
SPM: http://sematext.com/spm.
Regards,
Emir
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On 31.08.2016 06:28, Thiru M wrote:
> Thanks for your response *Shawn*.
>
> Have responded your queries and shared the SOLR details here.
>
> What precisely were you measuring during the night with no activity ?
>
> � Earlier we have enabled delta content indexing (content which got
> updated on day to day basis) script which triggers every 30 minutes. We
> monitored the load on SOLR when the delta indexing is performed.
>
> � CPU utilization by SOLR process,
>
> � Heap memory usage.
>
> what precise methods were you using to measure it?
>
> � We used the \u201c*top*\u201d command to monitor the solr process,
>
> � We monitored the free space in the server during the delta change
> indexing process,
>
> � We also enabled JMX remote monitoring in SOLR and used *jConsole*
> & *jVisualVM* to monitor CPU, Heap & Thread utilizations.
>
> What part of the information you obtained represents a problem in your mind?
>
> � Information obtained from top and the SOLR GC log in the *linux-a*
> server
>
> � Allocated max heap size is 2 GB. Based on the jConsole monitor h*eap
> usage is < 600 MB in linux-a server but the CPU utilization by SOLR process
> is on top (in figures it consumes 0.5 to 5 %)*.
>
>
> There is one solr instance per server i.e in linux-a one solr instance and
> in linux-b one solr instance (stand alone) is available.
>
>
>
> linux-a
>
> linux-b
>
> Number of Cores
>
> 49
>
> 14
>
> Number of Documents
>
> 6145495 (~6M)
>
> 1923181 (~1M)
>
> Overall index directory size
>
> 9.9 GB
>
> 14 GB
>
> Min Heap Memory
>
> 256 MB
>
> Max Heap Memory
>
> 2048 MB
>
> Total # System Processors
>
> 40
>
> Overall RAM size
>
> 125 GB
>
> Java Version
>
> 1.8.0_65 64-Bit Server VM
>
> SOLR Parameters
>
> -Xms512m
> -Xmx2048m
> -XX:NewRatio=3
> -XX:SurvivorRatio=4
> -XX:TargetSurvivorRatio=90
> -XX:MaxTenuringThreshold=8
> -XX:+UseConcMarkSweepGC
> -XX:+UseParNewGC
> -XX:ConcGCThreads=4
> -XX:ParallelGCThreads=4
> -XX:+CMSScavengeBeforeRemark
> -XX:PretenureSizeThreshold=64m
> -XX:+UseCMSInitiatingOccupancyOnly
> -XX:CMSInitiatingOccupancyFraction=50
> -XX:CMSMaxAbortablePrecleanTime=6000
> -XX:+CMSParallelRemarkEnabled
> -XX:+ParallelRefProcEnabled
> -verbose:gc
> -XX:+PrintHeapAtGC
> -XX:+PrintGCDetails
> -XX:+PrintGCDateStamps
> -XX:+PrintGCTimeStamps
> -XX:+PrintTenuringDistribution
> -XX:+PrintGCApplicationStoppedTime
> -Xloggc:/app/solr/logs/solr_gc.log
> -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.local.only=false
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.port=17010
> -Dcom.sun.management.jmxremote.rmi.port=17010
> -Djetty.port=7010
> -DSTOP.PORT=6010
> -DSTOP.KEY=solrrocks
> -Duser.timezone=UTC
> -Djetty.home=/opt/solr/server
> -Dsolr.solr.home=/local/solr/default/data
> -Dsolr.install.dir=/opt/solr
> -Dlog4j.configuration=file:/app/solr/log4j.properties
> -Xss256k
>
>
> Regards,
>
> Thirukumaran M
>
>
> On Sun, Aug 28, 2016 at 2:01 AM, Shawn Heisey <ap...@elyograg.org> wrote:
>
>> On 8/27/2016 9:08 AM, Thiru M wrote:
>>> We are using Solr 5.4.0 - "stand-alone" mode in our production boxes
>>> hosted in Red Hat Enterprise Linux (RHEL) OS.
>>>
>>> Each box have number of different cores. Have attached the screenshot
>>> shot with the Solr core & system details.
>>>
>>> 1. Earlier indexing was performed every 30 minutes in both production
>>> servers,
>>>
>>> 2. In linux-a server 30 (stand-alone) cores created on same day and
>>> content indexed into it,
>>>
>>> 3. we then spotted unusual GC performing every 2 to 7 seconds in
>>> linux-a server and the Solr process spiked,
>>>
>>> 4. Then we removed the indexing in linux-a server for a week,
>>> monitored the both Solr process and GC.(No indexing performed during
>>> this time),
>>>
>>> 5. No one uses the system during night time which we ensured it from
>>> our end. But both Solr process and GC were in its peak, even "during
>>> non business hours",
>>>
>>> 6. Have restated Solr instance in linux-a server. GC started again
>>> after Solr instance brought up,
>>>
>>> 7. In linux-b server no spike in Solr process and no issues with GC,
>>>
>> Indexing creates a LOT of garbage. Queries also create garbage, but not
>> nearly as fast as indexing. Solr has some background processes, and
>> these will create garbage too. Java uses a garbage collection memory
>> model, so this is completely normal for java applications.
>>
>> What precisely were you measuring during the night with no activity, and
>> what precise methods were you using to measure it? What part of the
>> information you obtained represents a problem in your mind?
>>
>> We'll also need some details about these servers:
>>
>> * Total index size of all Solr cores on the server.
>> * Total amount of memory installed in the server.
>> * Total number of documents contained in all Solr cores.
>> * How many Solr instances per server? What is the max heap size of each
>> instance?
>>
>> Attachments rarely make it to the list. The screenshot you mentioned
>> did not make it. You'll need to put it somewhere on the Internet an
>> provide a URL. Sharing sites like drobox or imgur are good choices for
>> image data.
>>
>> Since I don't have a clear idea of what the exact issue is here, I don't
>> have any immediate suggestions, aside from possibly increasing your max
>> heap size ... but depending on the answers to the questions above, that
>> might make things worse.
>>
>> Thanks,
>> Shawn
>>
>>
>
Re: Spike in SOLR Process and Frequent GC
Posted by Thiru M <m....@gmail.com>.
Thanks for your response *Shawn*.
Have responded your queries and shared the SOLR details here.
What precisely were you measuring during the night with no activity ?
· Earlier we have enabled delta content indexing (content which got
updated on day to day basis) script which triggers every 30 minutes. We
monitored the load on SOLR when the delta indexing is performed.
· CPU utilization by SOLR process,
· Heap memory usage.
what precise methods were you using to measure it?
· We used the “*top*” command to monitor the solr process,
· We monitored the free space in the server during the delta change
indexing process,
· We also enabled JMX remote monitoring in SOLR and used *jConsole*
& *jVisualVM* to monitor CPU, Heap & Thread utilizations.
What part of the information you obtained represents a problem in your mind?
· Information obtained from top and the SOLR GC log in the *linux-a*
server
· Allocated max heap size is 2 GB. Based on the jConsole monitor h*eap
usage is < 600 MB in linux-a server but the CPU utilization by SOLR process
is on top (in figures it consumes 0.5 to 5 %)*.
There is one solr instance per server i.e in linux-a one solr instance and
in linux-b one solr instance (stand alone) is available.
linux-a
linux-b
Number of Cores
49
14
Number of Documents
6145495 (~6M)
1923181 (~1M)
Overall index directory size
9.9 GB
14 GB
Min Heap Memory
256 MB
Max Heap Memory
2048 MB
Total # System Processors
40
Overall RAM size
125 GB
Java Version
1.8.0_65 64-Bit Server VM
SOLR Parameters
-Xms512m
-Xmx2048m
-XX:NewRatio=3
-XX:SurvivorRatio=4
-XX:TargetSurvivorRatio=90
-XX:MaxTenuringThreshold=8
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC
-XX:ConcGCThreads=4
-XX:ParallelGCThreads=4
-XX:+CMSScavengeBeforeRemark
-XX:PretenureSizeThreshold=64m
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
-XX:CMSMaxAbortablePrecleanTime=6000
-XX:+CMSParallelRemarkEnabled
-XX:+ParallelRefProcEnabled
-verbose:gc
-XX:+PrintHeapAtGC
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:+PrintGCTimeStamps
-XX:+PrintTenuringDistribution
-XX:+PrintGCApplicationStoppedTime
-Xloggc:/app/solr/logs/solr_gc.log
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=17010
-Dcom.sun.management.jmxremote.rmi.port=17010
-Djetty.port=7010
-DSTOP.PORT=6010
-DSTOP.KEY=solrrocks
-Duser.timezone=UTC
-Djetty.home=/opt/solr/server
-Dsolr.solr.home=/local/solr/default/data
-Dsolr.install.dir=/opt/solr
-Dlog4j.configuration=file:/app/solr/log4j.properties
-Xss256k
Regards,
Thirukumaran M
On Sun, Aug 28, 2016 at 2:01 AM, Shawn Heisey <ap...@elyograg.org> wrote:
> On 8/27/2016 9:08 AM, Thiru M wrote:
> > We are using Solr 5.4.0 - "stand-alone" mode in our production boxes
> > hosted in Red Hat Enterprise Linux (RHEL) OS.
> >
> > Each box have number of different cores. Have attached the screenshot
> > shot with the Solr core & system details.
> >
> > 1. Earlier indexing was performed every 30 minutes in both production
> > servers,
> >
> > 2. In linux-a server 30 (stand-alone) cores created on same day and
> > content indexed into it,
> >
> > 3. we then spotted unusual GC performing every 2 to 7 seconds in
> > linux-a server and the Solr process spiked,
> >
> > 4. Then we removed the indexing in linux-a server for a week,
> > monitored the both Solr process and GC.(No indexing performed during
> > this time),
> >
> > 5. No one uses the system during night time which we ensured it from
> > our end. But both Solr process and GC were in its peak, even "during
> > non business hours",
> >
> > 6. Have restated Solr instance in linux-a server. GC started again
> > after Solr instance brought up,
> >
> > 7. In linux-b server no spike in Solr process and no issues with GC,
> >
>
> Indexing creates a LOT of garbage. Queries also create garbage, but not
> nearly as fast as indexing. Solr has some background processes, and
> these will create garbage too. Java uses a garbage collection memory
> model, so this is completely normal for java applications.
>
> What precisely were you measuring during the night with no activity, and
> what precise methods were you using to measure it? What part of the
> information you obtained represents a problem in your mind?
>
> We'll also need some details about these servers:
>
> * Total index size of all Solr cores on the server.
> * Total amount of memory installed in the server.
> * Total number of documents contained in all Solr cores.
> * How many Solr instances per server? What is the max heap size of each
> instance?
>
> Attachments rarely make it to the list. The screenshot you mentioned
> did not make it. You'll need to put it somewhere on the Internet an
> provide a URL. Sharing sites like drobox or imgur are good choices for
> image data.
>
> Since I don't have a clear idea of what the exact issue is here, I don't
> have any immediate suggestions, aside from possibly increasing your max
> heap size ... but depending on the answers to the questions above, that
> might make things worse.
>
> Thanks,
> Shawn
>
>
--
By
M.Thirukumaran
Re: Spike in SOLR Process and Frequent GC
Posted by Shawn Heisey <ap...@elyograg.org>.
On 8/27/2016 9:08 AM, Thiru M wrote:
> We are using Solr 5.4.0 - "stand-alone" mode in our production boxes
> hosted in Red Hat Enterprise Linux (RHEL) OS.
>
> Each box have number of different cores. Have attached the screenshot
> shot with the Solr core & system details.
>
> 1. Earlier indexing was performed every 30 minutes in both production
> servers,
>
> 2. In linux-a server 30 (stand-alone) cores created on same day and
> content indexed into it,
>
> 3. we then spotted unusual GC performing every 2 to 7 seconds in
> linux-a server and the Solr process spiked,
>
> 4. Then we removed the indexing in linux-a server for a week,
> monitored the both Solr process and GC.(No indexing performed during
> this time),
>
> 5. No one uses the system during night time which we ensured it from
> our end. But both Solr process and GC were in its peak, even "during
> non business hours",
>
> 6. Have restated Solr instance in linux-a server. GC started again
> after Solr instance brought up,
>
> 7. In linux-b server no spike in Solr process and no issues with GC,
>
Indexing creates a LOT of garbage. Queries also create garbage, but not
nearly as fast as indexing. Solr has some background processes, and
these will create garbage too. Java uses a garbage collection memory
model, so this is completely normal for java applications.
What precisely were you measuring during the night with no activity, and
what precise methods were you using to measure it? What part of the
information you obtained represents a problem in your mind?
We'll also need some details about these servers:
* Total index size of all Solr cores on the server.
* Total amount of memory installed in the server.
* Total number of documents contained in all Solr cores.
* How many Solr instances per server? What is the max heap size of each
instance?
Attachments rarely make it to the list. The screenshot you mentioned
did not make it. You'll need to put it somewhere on the Internet an
provide a URL. Sharing sites like drobox or imgur are good choices for
image data.
Since I don't have a clear idea of what the exact issue is here, I don't
have any immediate suggestions, aside from possibly increasing your max
heap size ... but depending on the answers to the questions above, that
might make things worse.
Thanks,
Shawn