You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Robert Brown <ro...@intelcompute.com> on 2016/04/28 23:13:54 UTC
Idle timeout expired: 50000/50000 ms
Hi,
I operate several collections (about 7-8) all using the same 5-node
ZooKeeper cluster. They've been in production for 3 months, with only 2
previous issues where a Solr node went down.
Tonight, during several updates to the various collections, a handful
failed due to the below error.
Could this be related to ZooKeeper in any way? If so, what could I
check to ensure everything is running smoothly?
The collections are a mix of 1 and 2 shards, all with 1 replica.
Updates are performed in batches of 1000 in JSON files.
Are there any other things I could/should be checking?
$VAR1 = {
'error' => {
'code' => 500,
'msg' => 'java.util.concurrent.TimeoutException:
Idle timeout expired: 50000/50000 ms',
'trace' => 'java.io.IOException:
java.util.concurrent.TimeoutException: Idle timeout expired: 50000/50000 ms
at
org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:234)
at
org.eclipse.jetty.server.HttpInputOverHTTP.blockForContent(HttpInputOverHTTP.java:66)
at
org.eclipse.jetty.server.HttpInput$1.waitForContent(HttpInput.java:476)
at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:121)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at org.noggit.JSONParser.fill(JSONParser.java:196)
at org.noggit.JSONParser.getMore(JSONParser.java:203)
at org.noggit.JSONParser.readStringChars2(JSONParser.java:646)
at org.noggit.JSONParser.readStringChars(JSONParser.java:626)
at org.noggit.JSONParser.getStringChars(JSONParser.java:1029)
at org.noggit.JSONParser.getString(JSONParser.java:1017)
at
org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.parseDoc(JsonLoader.java:501)
at
org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.handleAdds(JsonLoader.java:470)
at
org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.processUpdate(JsonLoader.java:135)
at
org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.load(JsonLoader.java:114)
at org.apache.solr.handler.loader.JsonLoader.load(JsonLoader.java:77)
at
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:95)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:70)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2073)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:658)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:457)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:222)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.TimeoutException: Idle timeout expired:
50000/50000 ms
at
org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:161)
at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more
'
},
'responseHeader' => {
'QTime' => 77068,
'status' => 500
}
};
Re: Idle timeout expired: 50000/50000 ms
Posted by Yago Riveiro <ya...@gmail.com>.
Recently I started to buffering docs and send them to Solr in blocks of 250
and 50 workers. But now I'm hitting this issue too with Solr 5.3.1
Googling a bit I found this
https://bugs.eclipse.org/bugs/show_bug.cgi?id=435322 that was fixes in 9.2
version.
There is another link
<https://www.digitalocean.com/community/questions/jetty-timeout-problem-when-serving-images>
that in comments people describes that the running out upgrading to 9.2
-----
Best regards
--
View this message in context: http://lucene.472066.n3.nabble.com/Idle-timeout-expired-50000-50000-ms-tp4273515p4287219.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Idle timeout expired: 50000/50000 ms
Posted by Robert Brown <ro...@intelcompute.com>.
Thanks Shawn,
I'm definitely not looking to just upping the timeout, like you say,
there's a bigger issue to be resolved.
My indexes are between 1m and up to 60m docs (30m per shard, ~70GB on
disk each).
All of these collections get completely refreshed at least once a day,
data may not actually be changing, but the JSON files are re-uploaded
currently.
I've used a 3GB heap for all of the nodes, perhaps that needs upping a bit.
Strange that I've not seen this issue before though.
Any good advice/guidance for analysing and tweaking GC?
On 29/04/16 18:52, Shawn Heisey wrote:
> On 4/28/2016 3:13 PM, Robert Brown wrote:
>> I operate several collections (about 7-8) all using the same 5-node
>> ZooKeeper cluster. They've been in production for 3 months, with only
>> 2 previous issues where a Solr node went down.
>>
>> Tonight, during several updates to the various collections, a handful
>> failed due to the below error.
>>
>> Could this be related to ZooKeeper in any way? If so, what could I
>> check to ensure everything is running smoothly?
>>
>> The collections are a mix of 1 and 2 shards, all with 1 replica.
>>
>> Updates are performed in batches of 1000 in JSON files.
>>
>> Are there any other things I could/should be checking?
>>
>>
>> $VAR1 = {
>> 'error' => {
>> 'code' => 500,
>> 'msg' => 'java.util.concurrent.TimeoutException:
>> Idle timeout expired: 50000/50000 ms',
> This idle timeout is configured in Jetty. The default setting in the
> jetty config provided with Solr 5.x is 50 seconds.
>
> If your update requests are taking too long for the Jetty idle timeout,
> then I think you're having a general performance problem with Solr.
> Increasing the timeout might help in the short term, but unless you fix
> the underlying performance issue, you'd probably just run into the new
> timeout at some point in the future.
>
> Most severe performance problems like this are memory related, and are
> solved by adding more memory. Sometimes that is Java heap memory,
> sometimes that is memory that is not allocated to a program. Sometimes
> both are required.
>
> Thanks,
> Shawn
>
Re: Idle timeout expired: 50000/50000 ms
Posted by Shawn Heisey <ap...@elyograg.org>.
On 4/28/2016 3:13 PM, Robert Brown wrote:
> I operate several collections (about 7-8) all using the same 5-node
> ZooKeeper cluster. They've been in production for 3 months, with only
> 2 previous issues where a Solr node went down.
>
> Tonight, during several updates to the various collections, a handful
> failed due to the below error.
>
> Could this be related to ZooKeeper in any way? If so, what could I
> check to ensure everything is running smoothly?
>
> The collections are a mix of 1 and 2 shards, all with 1 replica.
>
> Updates are performed in batches of 1000 in JSON files.
>
> Are there any other things I could/should be checking?
>
>
> $VAR1 = {
> 'error' => {
> 'code' => 500,
> 'msg' => 'java.util.concurrent.TimeoutException:
> Idle timeout expired: 50000/50000 ms',
This idle timeout is configured in Jetty. The default setting in the
jetty config provided with Solr 5.x is 50 seconds.
If your update requests are taking too long for the Jetty idle timeout,
then I think you're having a general performance problem with Solr.
Increasing the timeout might help in the short term, but unless you fix
the underlying performance issue, you'd probably just run into the new
timeout at some point in the future.
Most severe performance problems like this are memory related, and are
solved by adding more memory. Sometimes that is Java heap memory,
sometimes that is memory that is not allocated to a program. Sometimes
both are required.
Thanks,
Shawn