You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Jason <hi...@gmail.com> on 2012/07/11 04:46:44 UTC
shard connection timeout
Hi, all
We're running multiple server(trunk version) with tomcat.
We'got connection timeout error as below when distribute searching.
So we've configured the shard timeout.
<shardHandlerFactory class="HttpShardHandlerFactory">
<int name="socketTimeout">300000</int>
<int name="connTimeout">300000</int>
</shardHandlerFactory>
But, error continues.
How can we fix this?
======================================================================
ERROR org.apache.solr.servlet.SolrDispatchFilter -
null:org.apache.solr.common.SolrException:
org.apache.solr.client.solrj.SolrServerException: Server refused connection
at: http://test.jt.pat.org:8080/solr_jt/jtp00
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:298)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1555)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:445)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:266)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at
org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
at
org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:732)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2262)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.solr.client.solrj.SolrServerException: Server refused
connection at: http://203.242.170.145:8080/solr_jt/jtp00
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:430)
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:209)
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:165)
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:132)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
... 3 more
Caused by: org.apache.http.conn.HttpHostConnectException: Connection to
http://test.jt.pat.org:8080 refused
at
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:158)
at
org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:150)
at
org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121)
at
org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:575)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:425)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732)
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:379)
... 11 more
Caused by: java.net.ConnectException: Connection timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at
java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at
org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:123)
at
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148)
... 19 more
--
View this message in context: http://lucene.472066.n3.nabble.com/shard-connection-timeout-tp3994301.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: shard connection timeout
Posted by "Trym R. Møller" <tr...@sigmat.dk>.
Hi Jason
We are running with -XX:MaxGCPauseMillis=5000 as well, which might also
help you.
Best regards Trym
Den 11-07-2012 07:55, Jason skrev:
> Actually we got this error when remote server is executing garbage collecting
> and that time is over about 1 minute.
> Solr server sometimes is frozen during gc and occurred connection refused
> error.
> Our gc option is -XX:+UseParallelGC -XX:+UseParallelOldGC
> -XX:+AggressiveOpts
> Response waiting is better than connection refuse for us.
> is there any solution?
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/shard-connection-timeout-tp3994301p3994320.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: shard connection timeout
Posted by Erick Erickson <er...@gmail.com>.
Right, as Michael says you've allocated a massive amount of memory.
I suspect you're just hitting a full-pause garbage collection and it's
taking a loooong time. Do you know whether you actually need all
that memory or was it allocated on the theory that "more is better"?
We often recommend that you _start_ with about 1/2 the memory to the
JVM and the rest to the OS. In your case, I'd back that off even more
unless you have evidence that this much memory is required.
Best
Erick
On Thu, Jul 12, 2012 at 8:54 AM, Michael Della Bitta
<mi...@appinions.com> wrote:
> Hi, Jason,
>
> That's a huge heap. Which Directory implementation are you using? It
> might make more sense to drastically reduce that heap and let the OS
> buffer the index at a lower level.
>
> Michael Della Bitta
>
> ------------------------------------------------
> Appinions, Inc. -- Where Influence Isn’t a Game.
> http://www.appinions.com
>
>
> On Wed, Jul 11, 2012 at 10:05 PM, Jason <hi...@gmail.com> wrote:
>> Hi Erick
>> Our physical memory is 128g and jvm option is -Xms110g -Xmx110g
>> -XX:PermSize=512m -XX:MaxPermSize=512m
>> This is very huge memory allocation.
>> But we are servicing patent search and that is accepted very complex queries
>> used memory.
>>
>>
>> --
>> View this message in context: http://lucene.472066.n3.nabble.com/shard-connection-timeout-tp3994301p3994556.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
Re: shard connection timeout
Posted by Michael Della Bitta <mi...@appinions.com>.
Hi, Jason,
That's a huge heap. Which Directory implementation are you using? It
might make more sense to drastically reduce that heap and let the OS
buffer the index at a lower level.
Michael Della Bitta
------------------------------------------------
Appinions, Inc. -- Where Influence Isn’t a Game.
http://www.appinions.com
On Wed, Jul 11, 2012 at 10:05 PM, Jason <hi...@gmail.com> wrote:
> Hi Erick
> Our physical memory is 128g and jvm option is -Xms110g -Xmx110g
> -XX:PermSize=512m -XX:MaxPermSize=512m
> This is very huge memory allocation.
> But we are servicing patent search and that is accepted very complex queries
> used memory.
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/shard-connection-timeout-tp3994301p3994556.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: shard connection timeout
Posted by Jason <hi...@gmail.com>.
Hi Erick
Our physical memory is 128g and jvm option is -Xms110g -Xmx110g
-XX:PermSize=512m -XX:MaxPermSize=512m
This is very huge memory allocation.
But we are servicing patent search and that is accepted very complex queries
used memory.
--
View this message in context: http://lucene.472066.n3.nabble.com/shard-connection-timeout-tp3994301p3994556.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: shard connection timeout
Posted by Erick Erickson <er...@gmail.com>.
There are about a zillion garbage collection options, see:
http://www.lucidimagination.com/blog/2011/03/27/garbage-collection-bootcamp-1-0/
for a great intro.
Be a bit careful. Allocating more memory to the JVM can cause the GCs to take
a longer time when they do occur. How much memory are you allocating?
Best
Erick
On Wed, Jul 11, 2012 at 1:55 AM, Jason <hi...@gmail.com> wrote:
> Actually we got this error when remote server is executing garbage collecting
> and that time is over about 1 minute.
> Solr server sometimes is frozen during gc and occurred connection refused
> error.
> Our gc option is -XX:+UseParallelGC -XX:+UseParallelOldGC
> -XX:+AggressiveOpts
> Response waiting is better than connection refuse for us.
> is there any solution?
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/shard-connection-timeout-tp3994301p3994320.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: shard connection timeout
Posted by Jason <hi...@gmail.com>.
Actually we got this error when remote server is executing garbage collecting
and that time is over about 1 minute.
Solr server sometimes is frozen during gc and occurred connection refused
error.
Our gc option is -XX:+UseParallelGC -XX:+UseParallelOldGC
-XX:+AggressiveOpts
Response waiting is better than connection refuse for us.
is there any solution?
--
View this message in context: http://lucene.472066.n3.nabble.com/shard-connection-timeout-tp3994301p3994320.html
Sent from the Solr - User mailing list archive at Nabble.com.