You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Shrestha, Biva" <sh...@ornl.gov> on 2012/11/20 16:35:57 UTC
SOLR USING 100% percent CPU and not responding after a while
Hello all,
We have solr 3.6 in a tomcat server with 16 GB memory allocated to it in a Linux server. It is a multicore solr and one of the core has over 100 million records. We have a web application that is fed by this solr. We don't use sharding, different cores are for different purposes. There are all kind of queries involved: simple queries, queries with filters, queries with facets, queries with pagination ( deep pagination as well) and a combination of them . When there are 100 concurrent users and quite some queries thrown to the solr the CPU utilization by the solr exceeds 100% and it becomes unresponsive.
Has someone from this group have had this problem and found a solution. We did look at the queries from the logs and there is some optimization to be done there but is there anything we can do in the configuration. I know that there is a parameter called MergeFactor which can be set low in the solrconfig.xml for enabling faster searching but I don't see anything about CPU utilization related to that settings. So my question in a jest is that is there any CPU utilization related configuration settings for SOLR.
Any help will be appreciated a lot
Thank you a bunch
Biva
Re: SOLR USING 100% percent CPU and not responding after a while
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,
Show us more graphs. Is the GC working hard? Any of the JVM mem pools at
or near 100%? SPM for Solr is your friend for long term
monitoring/alerting/trends, jconsole and visualvm for a quick look.
Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Tue, Jan 28, 2014 at 2:11 PM, heaven <ah...@gmail.com> wrote:
> I have the same problem, please look at the image:
> <http://lucene.472066.n3.nabble.com/file/n4114026/Screenshot_733.png>
>
> And this is on idle. Index size is about 90Gb. Solr 4.4.0. Memory is not an
> issue, there's a lot. RAID 10 (15000RPM rapid hdd).
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/SOLR-USING-100-percent-CPU-and-not-responding-after-a-while-tp4021359p4114026.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
RE: SOLR USING 100% percent CPU and not responding after a while
Posted by heaven <ah...@gmail.com>.
I have the same problem, please look at the image:
<http://lucene.472066.n3.nabble.com/file/n4114026/Screenshot_733.png>
And this is on idle. Index size is about 90Gb. Solr 4.4.0. Memory is not an
issue, there's a lot. RAID 10 (15000RPM rapid hdd).
--
View this message in context: http://lucene.472066.n3.nabble.com/SOLR-USING-100-percent-CPU-and-not-responding-after-a-while-tp4021359p4114026.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: SOLR USING 100% percent CPU and not responding after a while
Posted by nitin4php <ni...@gmail.com>.
Hi Biva,
Any luck on this?
Even we are facing same issue with exactly same configuration and setup.
Any inputs will help a lot.
--
View this message in context: http://lucene.472066.n3.nabble.com/SOLR-USING-100-percent-CPU-and-not-responding-after-a-while-tp4021359p4083234.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: SOLR USING 100% percent CPU and not responding after a while
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,
I looked at the spreadsheet and the graph and it looks like that's memory
for the whole OS. What you want to look at is the JVM heap and GC counts
and timings there, esp. when you say performance sinks. At that time also
look at your query, filter, and document caches and see if evictions go up
and cache hit rate down. Or maybe queries are so random that there is lots
of disk IO happening? Have a look at the URL in my signature, you may find
that helpful with this sort of troubleshooting.
Otis
--
Solr Performance Monitoring - http://sematext.com/spm/index.html
Search Analytics - http://sematext.com/search-analytics/index.html
On Tue, Nov 20, 2012 at 11:49 AM, Shrestha, Biva <sh...@ornl.gov> wrote:
> Hello,
> We were doing fine with memory, comparitively. This was in a one hour span
> of load testing we did. And no out of memory exceptions seen in the log. I
> have copied the memory chart here. It shows that most of the memory was
> being used for cache.
> Thank you
>
>
> -----Original Message-----
> From: Rafał Kuć [mailto:r.kuc@solr.pl]
> Sent: Tuesday, November 20, 2012 11:25 AM
> To: solr-user@lucene.apache.org
> Subject: Re: SOLR USING 100% percent CPU and not responding after a while
>
> Hello!
>
> The first thing one should ask is what is the cause of the 100% CPU
> utilization. It can be an issue with the memory or your queries may be
> quite resource intensive. Can you provide some more information about your
> issues ? Do you see any OutOfMemory exceptions in your logs ? How are JVM
> garbage collector behaving when you are experiencing Solr being
> unresponsive ?
>
> --
> Regards,
> Rafał Kuć
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - ElasticSearch
>
> > Hello all,
> > We have solr 3.6 in a tomcat server with 16 GB memory allocated to it
> > in a Linux server. It is a multicore solr and one of the core has over
> > 100 million records. We have a web application that is fed by this
> > solr. We don't use sharding, different cores are for different
> > purposes. There are all kind of queries involved: simple queries,
> > queries with filters, queries with facets, queries with pagination (
> > deep pagination as well) and a combination of them . When there are
> > 100 concurrent users and quite some queries thrown to the solr the CPU
> > utilization by the solr exceeds 100% and it becomes unresponsive.
> > Has someone from this group have had this problem and found a
> > solution. We did look at the queries from the logs and there is some
> > optimization to be done there but is there anything we can do in the
> > configuration. I know that there is a parameter called MergeFactor
> > which can be set low in the solrconfig.xml for enabling faster
> > searching but I don't see anything about CPU utilization related to
> > that settings. So my question in a jest is that is there any CPU
> > utilization related configuration settings for SOLR.
> > Any help will be appreciated a lot
> > Thank you a bunch
> > Biva
>
>
>
>
Re: SOLR USING 100% percent CPU and not responding after a while
Posted by Rafał Kuć <r....@solr.pl>.
Hello!
How about some information how Garbage Collector works during the time
when you have Solr unresponsive ?
--
Regards,
Rafał Kuć
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - ElasticSearch
> Hello,
> We were doing fine with memory, comparitively. This was in a one
> hour span of load testing we did. And no out of memory exceptions
> seen in the log. I have copied the memory chart here. It shows that
> most of the memory was being used for cache.
> Thank you
> -----Original Message-----
> From: Rafał Kuć [mailto:r.kuc@solr.pl]
> Sent: Tuesday, November 20, 2012 11:25 AM
> To: solr-user@lucene.apache.org
> Subject: Re: SOLR USING 100% percent CPU and not responding after a while
> Hello!
> The first thing one should ask is what is the cause of the 100% CPU
> utilization. It can be an issue with the memory or your queries may
> be quite resource intensive. Can you provide some more information
> about your issues ? Do you see any OutOfMemory exceptions in your
> logs ? How are JVM garbage collector behaving when you are
> experiencing Solr being unresponsive ?
> --
> Regards,
> Rafał Kuć
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - ElasticSearch
>> Hello all,
>> We have solr 3.6 in a tomcat server with 16 GB memory allocated to it
>> in a Linux server. It is a multicore solr and one of the core has over
>> 100 million records. We have a web application that is fed by this
>> solr. We don't use sharding, different cores are for different
>> purposes. There are all kind of queries involved: simple queries,
>> queries with filters, queries with facets, queries with pagination (
>> deep pagination as well) and a combination of them . When there are
>> 100 concurrent users and quite some queries thrown to the solr the CPU
>> utilization by the solr exceeds 100% and it becomes unresponsive.
>> Has someone from this group have had this problem and found a
>> solution. We did look at the queries from the logs and there is some
>> optimization to be done there but is there anything we can do in the
>> configuration. I know that there is a parameter called MergeFactor
>> which can be set low in the solrconfig.xml for enabling faster
>> searching but I don't see anything about CPU utilization related to
>> that settings. So my question in a jest is that is there any CPU
>> utilization related configuration settings for SOLR.
>> Any help will be appreciated a lot
>> Thank you a bunch
>> Biva
RE: SOLR USING 100% percent CPU and not responding after a while
Posted by "Shrestha, Biva" <sh...@ornl.gov>.
Hello,
We were doing fine with memory, comparitively. This was in a one hour span of load testing we did. And no out of memory exceptions seen in the log. I have copied the memory chart here. It shows that most of the memory was being used for cache.
Thank you
-----Original Message-----
From: Rafał Kuć [mailto:r.kuc@solr.pl]
Sent: Tuesday, November 20, 2012 11:25 AM
To: solr-user@lucene.apache.org
Subject: Re: SOLR USING 100% percent CPU and not responding after a while
Hello!
The first thing one should ask is what is the cause of the 100% CPU utilization. It can be an issue with the memory or your queries may be quite resource intensive. Can you provide some more information about your issues ? Do you see any OutOfMemory exceptions in your logs ? How are JVM garbage collector behaving when you are experiencing Solr being unresponsive ?
--
Regards,
Rafał Kuć
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - ElasticSearch
> Hello all,
> We have solr 3.6 in a tomcat server with 16 GB memory allocated to it
> in a Linux server. It is a multicore solr and one of the core has over
> 100 million records. We have a web application that is fed by this
> solr. We don't use sharding, different cores are for different
> purposes. There are all kind of queries involved: simple queries,
> queries with filters, queries with facets, queries with pagination (
> deep pagination as well) and a combination of them . When there are
> 100 concurrent users and quite some queries thrown to the solr the CPU
> utilization by the solr exceeds 100% and it becomes unresponsive.
> Has someone from this group have had this problem and found a
> solution. We did look at the queries from the logs and there is some
> optimization to be done there but is there anything we can do in the
> configuration. I know that there is a parameter called MergeFactor
> which can be set low in the solrconfig.xml for enabling faster
> searching but I don't see anything about CPU utilization related to
> that settings. So my question in a jest is that is there any CPU
> utilization related configuration settings for SOLR.
> Any help will be appreciated a lot
> Thank you a bunch
> Biva
Re: SOLR USING 100% percent CPU and not responding after a while
Posted by Rafał Kuć <r....@solr.pl>.
Hello!
The first thing one should ask is what is the cause of the 100% CPU
utilization. It can be an issue with the memory or your queries may be
quite resource intensive. Can you provide some more information about
your issues ? Do you see any OutOfMemory exceptions in your logs ? How
are JVM garbage collector behaving when you are experiencing Solr
being unresponsive ?
--
Regards,
Rafał Kuć
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - ElasticSearch
> Hello all,
> We have solr 3.6 in a tomcat server with 16 GB memory allocated to
> it in a Linux server. It is a multicore solr and one of the core has
> over 100 million records. We have a web application that is fed by
> this solr. We don't use sharding, different cores are for different
> purposes. There are all kind of queries involved: simple queries,
> queries with filters, queries with facets, queries with pagination (
> deep pagination as well) and a combination of them . When there are
> 100 concurrent users and quite some queries thrown to the solr the
> CPU utilization by the solr exceeds 100% and it becomes unresponsive.
> Has someone from this group have had this problem and found a
> solution. We did look at the queries from the logs and there is some
> optimization to be done there but is there anything we can do in the
> configuration. I know that there is a parameter called MergeFactor
> which can be set low in the solrconfig.xml for enabling faster
> searching but I don't see anything about CPU utilization related to
> that settings. So my question in a jest is that is there any CPU
> utilization related configuration settings for SOLR.
> Any help will be appreciated a lot
> Thank you a bunch
> Biva