You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by dipti khullar <di...@gmail.com> on 2010/03/25 18:13:06 UTC
Threads blocking on solr slave servers
Hi
Since last 2 days we are facing extremely slow behaviour with solr slave
servers. The threads are either in blocking or waiting state and require a
restart of servers to keep the production running.
Following are the relevant details:
There are 2 search servers in a virtualized VMware environment. Each has 2
instances of Solr running on separates ports in tomcat.
Server 1: hosts 1 master(application 1), 1 slave (application 1) 6 GB RAM,
JVM 2GB
Server 2: hosts 1 master (application 2), 1 slave (application 1) 6 GB RAM,
JVM 2GB
Maximum number of threads each server can handle is 250. The number of
requests is maxing out and the response time from solr is so slow that the
threads are getting blocked.
We have upgraded slaves to Solr 1.4 but master is still on Solr 1.3 as we do
not want to play much around with the production systems. Yet the behavior
is almost same.
Solrj client which is used to query slave solr,gets timedout and there is
high CPU usage/load avg.T he problem is reported on slaves for application
1. The SolrJ client which queries Solr over HTTP times out (10 sec is the
timeout value) though in the Solr tomcat access log we find all requests
have 200 response.
During the tme, requests timeout the load avg. of the server goes extremely
high (10-20).
The issue gets resolved as soon as we optimize the slave index. In the solr
admin, it shows only 4 requests/sec is handled with 400 ms response time.
Blocked thread:
'http-9090-219' Id=248, BLOCKED on
lock=org.apache.lucene.index.SegmentReader$CoreReaders@1546c85, total cpu
time=3930.0000ms user time=2830.0000ms
at
org.apache.lucene.index.SegmentReader$CoreReaders.getTermsReader(SegmentReader.java:161)
at org.apache.lucene.index.SegmentTermDocs.seek(SegmentTermDocs.java:57)
at
org.apache.lucene.search.MultiTermQueryWrapperFilter$TermGenerator.generate(MultiTermQueryWrapperFilter.java:110)
at
org.apache.lucene.search.MultiTermQueryWrapperFilter.getDocIdSet(MultiTermQueryWrapperFilter.java:167)
at
org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.<init>(ConstantScoreQuery.java:116)
at
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:81)
at
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:297)
at
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:297)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:246)
at org.apache.lucene.search.Searcher.search(Searcher.java:171)
at
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:988)
at
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:884)
at
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:341)
at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:182)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at
org.jstripe.tomcat.probe.Tomcat55AgentValve.invoke(Tomcat55AgentValve.java:20)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
at java.lang.Thread.run(Thread.java:595)
Thanks
Dipti
Re: Threads blocking on solr slave servers
Posted by Lance Norskog <go...@gmail.com>.
The Solr instance can only handle so many queries per second. It
should be possible to configure the web app container to limit the
total number of active threads.
Also, have you watched the garbage collection statistics with
'jconsole' or another interactive tool? You might discover that it is
spending all of its time doing garbage collection.
On Thu, Mar 25, 2010 at 10:13 AM, dipti khullar <di...@gmail.com> wrote:
> Hi
>
> Since last 2 days we are facing extremely slow behaviour with solr slave
> servers. The threads are either in blocking or waiting state and require a
> restart of servers to keep the production running.
>
> Following are the relevant details:
>
> There are 2 search servers in a virtualized VMware environment. Each has 2
> instances of Solr running on separates ports in tomcat.
> Server 1: hosts 1 master(application 1), 1 slave (application 1) 6 GB RAM,
> JVM 2GB
> Server 2: hosts 1 master (application 2), 1 slave (application 1) 6 GB RAM,
> JVM 2GB
>
> Maximum number of threads each server can handle is 250. The number of
> requests is maxing out and the response time from solr is so slow that the
> threads are getting blocked.
>
> We have upgraded slaves to Solr 1.4 but master is still on Solr 1.3 as we do
> not want to play much around with the production systems. Yet the behavior
> is almost same.
>
> Solrj client which is used to query slave solr,gets timedout and there is
> high CPU usage/load avg.T he problem is reported on slaves for application
> 1. The SolrJ client which queries Solr over HTTP times out (10 sec is the
> timeout value) though in the Solr tomcat access log we find all requests
> have 200 response.
> During the tme, requests timeout the load avg. of the server goes extremely
> high (10-20).
> The issue gets resolved as soon as we optimize the slave index. In the solr
> admin, it shows only 4 requests/sec is handled with 400 ms response time.
>
> Blocked thread:
>
> 'http-9090-219' Id=248, BLOCKED on
> lock=org.apache.lucene.index.SegmentReader$CoreReaders@1546c85, total cpu
> time=3930.0000ms user time=2830.0000ms
> at
> org.apache.lucene.index.SegmentReader$CoreReaders.getTermsReader(SegmentReader.java:161)
>
> at org.apache.lucene.index.SegmentTermDocs.seek(SegmentTermDocs.java:57)
> at
> org.apache.lucene.search.MultiTermQueryWrapperFilter$TermGenerator.generate(MultiTermQueryWrapperFilter.java:110)
>
> at
> org.apache.lucene.search.MultiTermQueryWrapperFilter.getDocIdSet(MultiTermQueryWrapperFilter.java:167)
>
> at
> org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.<init>(ConstantScoreQuery.java:116)
>
> at
> org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:81)
>
> at
> org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:297)
>
> at
> org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:297)
>
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:246)
> at org.apache.lucene.search.Searcher.search(Searcher.java:171)
> at
> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:988)
>
> at
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:884)
>
> at
> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:341)
> at
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:182)
>
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
>
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
>
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
>
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
>
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
>
> at
> org.jstripe.tomcat.probe.Tomcat55AgentValve.invoke(Tomcat55AgentValve.java:20)
>
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>
> at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567)
>
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
> at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
> at
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
>
> at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
> at java.lang.Thread.run(Thread.java:595)
>
> Thanks
> Dipti
>
--
Lance Norskog
goksron@gmail.com