You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Markus Jelsma <ma...@openindex.io> on 2012/08/07 20:07:43 UTC
null:java.lang.RuntimeException: [was class
java.net.SocketTimeoutException] null
Hello,
We sometimes see the error below in our `master` when indexing. Our master is currently the node we send documents to - we've not yet implemented CloudSolrServer in Apache Nutch. This causes the indexer to crash when using Nutch locally, the task is retried when running on Hadoop. We're running it locally in this test set up so there's only one indexing thread.
Anyway, for me it's quite a cryptic error because i don't know what connection has timed out, i assume a connection from the indexing node to some other node in the cluster when it passes a document to the correct leader? Each node of the 10 node cluster has the same configuration, Tomcat is configured with maxThreads=512 and a time out of one second.
We're using today's trunk in this test set up and we cannot reliably reproduce the error. We've seen the error before so it's not a very recent issue. No errors are found in the other node's logs.
2012-08-07 17:52:05,260 ERROR [solr.servlet.SolrDispatchFilter] - [http-8080-exec-6] - : null:java.lang.RuntimeException: [was class java.net.SocketTimeoutException] null
at com.ctc.wstx.util.ExceptionUtil.throwRuntimeException(ExceptionUtil.java:18)
at com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:731)
at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3657)
at com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:809)
at org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:376)
at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:229)
at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:157)
at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1656)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:454)
at io.openindex.solr.servlet.HttpResponseSolrDispatchFilter.doFilter(HttpResponseSolrDispatchFilter.java:219)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:744)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.SocketTimeoutException
at org.apache.tomcat.util.net.NioBlockingSelector.read(NioBlockingSelector.java:185)
at org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:229)
at org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:210)
at org.apache.coyote.http11.InternalNioInputBuffer.readSocket(InternalNioInputBuffer.java:643)
at org.apache.coyote.http11.InternalNioInputBuffer.fill(InternalNioInputBuffer.java:945)
at org.apache.coyote.http11.InternalNioInputBuffer$SocketInputBuffer.doR
ead(InternalNioInputBuffer.java:969)
at org.apache.coyote.http11.filters.ChunkedInputFilter.readBytes(ChunkedInputFilter.java:268)
at org.apache.coyote.http11.filters.ChunkedInputFilter.doRead(ChunkedInputFilter.java:167)
at org.apache.coyote.http11.InternalNioInputBuffer.doRead(InternalNioInputBuffer.java:916)
at org.apache.coyote.Request.doRead(Request.java:427)
at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:304)
at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:419)
at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:327)
at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:162)
at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:365)
at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:110)
at com.ctc.wstx.io.MergedReader.read(MergedReader.java:101)
at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84)
at com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57)
at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:992)
at com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4628)
at com.ctc.wstx.sr.BasicStreamReader.readCoalescedText(BasicStreamReader.java:4126)
at com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3701)
at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3649)
... 24 more
Any thoughts to share? Is this a bug? A misconfiguration?
Thanks,
Markus
Re: java.lang.RuntimeException: [was class java.net.SocketTimeoutException] null
Posted by Jack Krupansky <ja...@basetechnology.com>.
I'm wondering if the timeout occurs due to a JVM garbage collection due to a
large number of Lucene segments to merge.
What is the JVM heap usage like, compared to the total heap space available?
In other words, maybe the JVM needs more heap memory.
-- Jack Krupansky
-----Original Message-----
From: Markus Jelsma
Sent: Tuesday, August 07, 2012 5:39 PM
To: solr-user@lucene.apache.org ; Markus Jelsma
Subject: RE: null:java.lang.RuntimeException: [was class
java.net.SocketTimeoutException] null
A signicant detail is the batch size which we set to 64 documents due to
earlier memory limitations. We index segments of roughly 300-500k records
each time. Lowering the batch size to 32 lead to an early internal server
error and the stack trace below. Increasing it to 128 allowed us to index
some more records but it still throws the error after 200k+ indexed records.
Increasing it even more to 256 records per batch allowed us to index an
entire segment without errors.
Another detail is that we do not restart the cluster between indexing
attempts so it seems that something only builds up during indexing (nothing
seems to leak afterwards) and throws an error.
Any hints?
Thanks,
Markus
-----Original message-----
> From:Markus Jelsma <ma...@openindex.io>
> Sent: Tue 07-Aug-2012 20:08
> To: solr-user@lucene.apache.org
> Subject: null:java.lang.RuntimeException: [was class
> java.net.SocketTimeoutException] null
>
> Hello,
>
> We sometimes see the error below in our `master` when indexing. Our master
> is currently the node we send documents to - we've not yet implemented
> CloudSolrServer in Apache Nutch. This causes the indexer to crash when
> using Nutch locally, the task is retried when running on Hadoop. We're
> running it locally in this test set up so there's only one indexing
> thread.
>
> Anyway, for me it's quite a cryptic error because i don't know what
> connection has timed out, i assume a connection from the indexing node to
> some other node in the cluster when it passes a document to the correct
> leader? Each node of the 10 node cluster has the same configuration,
> Tomcat is configured with maxThreads=512 and a time out of one second.
>
> We're using today's trunk in this test set up and we cannot reliably
> reproduce the error. We've seen the error before so it's not a very recent
> issue. No errors are found in the other node's logs.
>
> 2012-08-07 17:52:05,260 ERROR [solr.servlet.SolrDispatchFilter] -
> [http-8080-exec-6] - : null:java.lang.RuntimeException: [was class
> java.net.SocketTimeoutException] null
> at
> com.ctc.wstx.util.ExceptionUtil.throwRuntimeException(ExceptionUtil.java:18)
> at
> com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:731)
> at
> com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3657)
> at
> com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:809)
> at
> org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:376)
> at
> org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:229)
> at
> org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:157)
> at
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
> at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1656)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:454)
> at
> io.openindex.solr.servlet.HttpResponseSolrDispatchFilter.doFilter(HttpResponseSolrDispatchFilter.java:219)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
> at
> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
> at
> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:744)
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.net.SocketTimeoutException
> at
> org.apache.tomcat.util.net.NioBlockingSelector.read(NioBlockingSelector.java:185)
> at
> org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:229)
> at
> org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:210)
> at
> org.apache.coyote.http11.InternalNioInputBuffer.readSocket(InternalNioInputBuffer.java:643)
> at
> org.apache.coyote.http11.InternalNioInputBuffer.fill(InternalNioInputBuffer.java:945)
> at
> org.apache.coyote.http11.InternalNioInputBuffer$SocketInputBuffer.doR
> ead(InternalNioInputBuffer.java:969)
> at
> org.apache.coyote.http11.filters.ChunkedInputFilter.readBytes(ChunkedInputFilter.java:268)
> at
> org.apache.coyote.http11.filters.ChunkedInputFilter.doRead(ChunkedInputFilter.java:167)
> at
> org.apache.coyote.http11.InternalNioInputBuffer.doRead(InternalNioInputBuffer.java:916)
> at org.apache.coyote.Request.doRead(Request.java:427)
> at
> org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:304)
> at
> org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:419)
> at
> org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:327)
> at
> org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:162)
> at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:365)
> at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:110)
> at com.ctc.wstx.io.MergedReader.read(MergedReader.java:101)
> at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84)
> at
> com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57)
> at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:992)
> at
> com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4628)
> at
> com.ctc.wstx.sr.BasicStreamReader.readCoalescedText(BasicStreamReader.java:4126)
> at
> com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3701)
> at
> com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3649)
> ... 24 more
>
> Any thoughts to share? Is this a bug? A misconfiguration?
>
> Thanks,
> Markus
>
Re: null:java.lang.RuntimeException: [was class java.net.SocketTimeoutException]
null
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Thu, Aug 9, 2012 at 10:11 AM, Markus Jelsma
<ma...@openindex.io> wrote:
> I've increased the connection time out on all 10 Tomcats from 1000ms to 5000ms. Indexing a larger amount of batches seems to run fine now. This, however, does not really answer the issue. What is exactly timing out here and why?
It can be any communication with tomcat for any reason.
For example, a commit needs to flush and fsync all segments, applying
buffered deletes, etc, then open a new searcher and run any configured
warming queries or autowarming. That can take some time. It's even
longer if you want to optimize. Or a long GC pause could cause a
socket timeout.
For the stock jetty server, we set it to 50,000ms, which still may be
too short for some things frankly.
Here's the jetty documentation for the parameter:
"""
maxIdleTime:
Set the maximum Idle time for a connection, which roughly translates
to the Socket.setSoTimeout(int) call, although with NIO
implementations other mechanisms may be used to implement the timeout.
The max idle time is applied: when waiting for a new request to be
received on a connection; when reading the headers and content of a
request; when writing the headers and content of a response. Jetty
interprets this value as the maximum time between some progress being
made on the connection. So if a single byte is read or written, then
the timeout (if implemented by jetty) is reset. However, in many
instances, the reading/writing is delegated to the JVM, and the
semantic is more strictly enforced as the maximum time a single
read/write operation can take. Note, that as Jetty supports writes of
memory mapped file buffers, then a write may take many 10s of seconds
for large content written to a slow device.
"""
-Yonik
http://lucidimagination.com
I assume its the forwarding of documents from the `indexing node` to
the correct shard leader but with 512 maxThreads it should be fine.
>
> Any hints?
>
> Thanks
>
>
>
> -----Original message-----
>> From:Markus Jelsma <ma...@openindex.io>
>> Sent: Wed 08-Aug-2012 00:10
>> To: solr-user@lucene.apache.org
>> Subject: RE: null:java.lang.RuntimeException: [was class java.net.SocketTimeoutException] null
>>
>> Jack,
>>
>> There are no peculiarities in the JVM graphs. Only increase in used threads and GC time. Heap space is collected quickly and doesn't suddenly increase. There's only 256MB available for the heap but it's fine.
>>
>>
>> Yonik,
>>
>> I'll increase the time out to five seconds tomorrow and try to reproduce it with a low batch size of 32. Juding from what i've seen it should throw an error quickly with such a low batch size. However, what is timing out here? My client connection to the indexing node or something else that i don't see?
>>
>> Unfortunately no Jetty here (yet).
>>
>> Thanks
>> Markus
>>
>>
>> -----Original message-----
>> > From:Yonik Seeley <yo...@lucidimagination.com>
>> > Sent: Tue 07-Aug-2012 23:54
>> > To: solr-user@lucene.apache.org
>> > Subject: Re: null:java.lang.RuntimeException: [was class java.net.SocketTimeoutException] null
>> >
>> > Could this be just a simple case of a socket timeout? Can you raise
>> > the timout on request threads in Tomcat?
>> > It's a lot easier to reproduce/diagnose stuff like this when people
>> > use the stock jetty server shipped with Solr.
>> >
>> > -Yonik
>> > http://lucidimagination.com
>> >
>> >
>> > On Tue, Aug 7, 2012 at 5:39 PM, Markus Jelsma
>> > <ma...@openindex.io> wrote:
>> > > A signicant detail is the batch size which we set to 64 documents due to earlier memory limitations. We index segments of roughly 300-500k records each time. Lowering the batch size to 32 lead to an early internal server error and the stack trace below. Increasing it to 128 allowed us to index some more records but it still throws the error after 200k+ indexed records.
>> > >
>> > > Increasing it even more to 256 records per batch allowed us to index an entire segment without errors.
>> > >
>> > > Another detail is that we do not restart the cluster between indexing attempts so it seems that something only builds up during indexing (nothing seems to leak afterwards) and throws an error.
>> > >
>> > > Any hints?
>> > >
>> > > Thanks,
>> > > Markus
>> > >
>> > >
>> > >
>> > > -----Original message-----
>> > >> From:Markus Jelsma <ma...@openindex.io>
>> > >> Sent: Tue 07-Aug-2012 20:08
>> > >> To: solr-user@lucene.apache.org
>> > >> Subject: null:java.lang.RuntimeException: [was class java.net.SocketTimeoutException] null
>> > >>
>> > >> Hello,
>> > >>
>> > >> We sometimes see the error below in our `master` when indexing. Our master is currently the node we send documents to - we've not yet implemented CloudSolrServer in Apache Nutch. This causes the indexer to crash when using Nutch locally, the task is retried when running on Hadoop. We're running it locally in this test set up so there's only one indexing thread.
>> > >>
>> > >> Anyway, for me it's quite a cryptic error because i don't know what connection has timed out, i assume a connection from the indexing node to some other node in the cluster when it passes a document to the correct leader? Each node of the 10 node cluster has the same configuration, Tomcat is configured with maxThreads=512 and a time out of one second.
>> > >>
>> > >> We're using today's trunk in this test set up and we cannot reliably reproduce the error. We've seen the error before so it's not a very recent issue. No errors are found in the other node's logs.
>> > >>
>> > >> 2012-08-07 17:52:05,260 ERROR [solr.servlet.SolrDispatchFilter] - [http-8080-exec-6] - : null:java.lang.RuntimeException: [was class java.net.SocketTimeoutException] null
>> > >> at com.ctc.wstx.util.ExceptionUtil.throwRuntimeException(ExceptionUtil.java:18)
>> > >> at com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:731)
>> > >> at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3657)
>> > >> at com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:809)
>> > >> at org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:376)
>> > >> at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:229)
>> > >> at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:157)
>> > >> at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
>> > >> at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
>> > >> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
>> > >> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1656)
>> > >> at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:454)
>> > >> at io.openindex.solr.servlet.HttpResponseSolrDispatchFilter.doFilter(HttpResponseSolrDispatchFilter.java:219)
>> > >> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>> > >> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>> > >> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>> > >> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>> > >> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>> > >> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>> > >> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>> > >> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
>> > >> at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
>> > >> at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:744)
>> > >> at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274)
>> > >> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> > >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> > >> at java.lang.Thread.run(Thread.java:662)
>> > >> Caused by: java.net.SocketTimeoutException
>> > >> at org.apache.tomcat.util.net.NioBlockingSelector.read(NioBlockingSelector.java:185)
>> > >> at org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:229)
>> > >> at org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:210)
>> > >> at org.apache.coyote.http11.InternalNioInputBuffer.readSocket(InternalNioInputBuffer.java:643)
>> > >> at org.apache.coyote.http11.InternalNioInputBuffer.fill(InternalNioInputBuffer.java:945)
>> > >> at org.apache.coyote.http11.InternalNioInputBuffer$SocketInputBuffer.doR
>> > >> ead(InternalNioInputBuffer.java:969)
>> > >> at org.apache.coyote.http11.filters.ChunkedInputFilter.readBytes(ChunkedInputFilter.java:268)
>> > >> at org.apache.coyote.http11.filters.ChunkedInputFilter.doRead(ChunkedInputFilter.java:167)
>> > >> at org.apache.coyote.http11.InternalNioInputBuffer.doRead(InternalNioInputBuffer.java:916)
>> > >> at org.apache.coyote.Request.doRead(Request.java:427)
>> > >> at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:304)
>> > >> at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:419)
>> > >> at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:327)
>> > >> at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:162)
>> > >> at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:365)
>> > >> at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:110)
>> > >> at com.ctc.wstx.io.MergedReader.read(MergedReader.java:101)
>> > >> at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84)
>> > >> at com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57)
>> > >> at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:992)
>> > >> at com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4628)
>> > >> at com.ctc.wstx.sr.BasicStreamReader.readCoalescedText(BasicStreamReader.java:4126)
>> > >> at com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3701)
>> > >> at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3649)
>> > >> ... 24 more
>> > >>
>> > >> Any thoughts to share? Is this a bug? A misconfiguration?
>> > >>
>> > >> Thanks,
>> > >> Markus
>> > >>
>> >
>>
RE: null:java.lang.RuntimeException: [was class
java.net.SocketTimeoutException] null
Posted by Markus Jelsma <ma...@openindex.io>.
I've increased the connection time out on all 10 Tomcats from 1000ms to 5000ms. Indexing a larger amount of batches seems to run fine now. This, however, does not really answer the issue. What is exactly timing out here and why? I assume its the forwarding of documents from the `indexing node` to the correct shard leader but with 512 maxThreads it should be fine.
Any hints?
Thanks
-----Original message-----
> From:Markus Jelsma <ma...@openindex.io>
> Sent: Wed 08-Aug-2012 00:10
> To: solr-user@lucene.apache.org
> Subject: RE: null:java.lang.RuntimeException: [was class java.net.SocketTimeoutException] null
>
> Jack,
>
> There are no peculiarities in the JVM graphs. Only increase in used threads and GC time. Heap space is collected quickly and doesn't suddenly increase. There's only 256MB available for the heap but it's fine.
>
>
> Yonik,
>
> I'll increase the time out to five seconds tomorrow and try to reproduce it with a low batch size of 32. Juding from what i've seen it should throw an error quickly with such a low batch size. However, what is timing out here? My client connection to the indexing node or something else that i don't see?
>
> Unfortunately no Jetty here (yet).
>
> Thanks
> Markus
>
>
> -----Original message-----
> > From:Yonik Seeley <yo...@lucidimagination.com>
> > Sent: Tue 07-Aug-2012 23:54
> > To: solr-user@lucene.apache.org
> > Subject: Re: null:java.lang.RuntimeException: [was class java.net.SocketTimeoutException] null
> >
> > Could this be just a simple case of a socket timeout? Can you raise
> > the timout on request threads in Tomcat?
> > It's a lot easier to reproduce/diagnose stuff like this when people
> > use the stock jetty server shipped with Solr.
> >
> > -Yonik
> > http://lucidimagination.com
> >
> >
> > On Tue, Aug 7, 2012 at 5:39 PM, Markus Jelsma
> > <ma...@openindex.io> wrote:
> > > A signicant detail is the batch size which we set to 64 documents due to earlier memory limitations. We index segments of roughly 300-500k records each time. Lowering the batch size to 32 lead to an early internal server error and the stack trace below. Increasing it to 128 allowed us to index some more records but it still throws the error after 200k+ indexed records.
> > >
> > > Increasing it even more to 256 records per batch allowed us to index an entire segment without errors.
> > >
> > > Another detail is that we do not restart the cluster between indexing attempts so it seems that something only builds up during indexing (nothing seems to leak afterwards) and throws an error.
> > >
> > > Any hints?
> > >
> > > Thanks,
> > > Markus
> > >
> > >
> > >
> > > -----Original message-----
> > >> From:Markus Jelsma <ma...@openindex.io>
> > >> Sent: Tue 07-Aug-2012 20:08
> > >> To: solr-user@lucene.apache.org
> > >> Subject: null:java.lang.RuntimeException: [was class java.net.SocketTimeoutException] null
> > >>
> > >> Hello,
> > >>
> > >> We sometimes see the error below in our `master` when indexing. Our master is currently the node we send documents to - we've not yet implemented CloudSolrServer in Apache Nutch. This causes the indexer to crash when using Nutch locally, the task is retried when running on Hadoop. We're running it locally in this test set up so there's only one indexing thread.
> > >>
> > >> Anyway, for me it's quite a cryptic error because i don't know what connection has timed out, i assume a connection from the indexing node to some other node in the cluster when it passes a document to the correct leader? Each node of the 10 node cluster has the same configuration, Tomcat is configured with maxThreads=512 and a time out of one second.
> > >>
> > >> We're using today's trunk in this test set up and we cannot reliably reproduce the error. We've seen the error before so it's not a very recent issue. No errors are found in the other node's logs.
> > >>
> > >> 2012-08-07 17:52:05,260 ERROR [solr.servlet.SolrDispatchFilter] - [http-8080-exec-6] - : null:java.lang.RuntimeException: [was class java.net.SocketTimeoutException] null
> > >> at com.ctc.wstx.util.ExceptionUtil.throwRuntimeException(ExceptionUtil.java:18)
> > >> at com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:731)
> > >> at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3657)
> > >> at com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:809)
> > >> at org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:376)
> > >> at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:229)
> > >> at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:157)
> > >> at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
> > >> at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
> > >> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
> > >> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1656)
> > >> at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:454)
> > >> at io.openindex.solr.servlet.HttpResponseSolrDispatchFilter.doFilter(HttpResponseSolrDispatchFilter.java:219)
> > >> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> > >> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> > >> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> > >> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> > >> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> > >> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> > >> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> > >> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
> > >> at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
> > >> at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:744)
> > >> at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274)
> > >> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > >> at java.lang.Thread.run(Thread.java:662)
> > >> Caused by: java.net.SocketTimeoutException
> > >> at org.apache.tomcat.util.net.NioBlockingSelector.read(NioBlockingSelector.java:185)
> > >> at org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:229)
> > >> at org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:210)
> > >> at org.apache.coyote.http11.InternalNioInputBuffer.readSocket(InternalNioInputBuffer.java:643)
> > >> at org.apache.coyote.http11.InternalNioInputBuffer.fill(InternalNioInputBuffer.java:945)
> > >> at org.apache.coyote.http11.InternalNioInputBuffer$SocketInputBuffer.doR
> > >> ead(InternalNioInputBuffer.java:969)
> > >> at org.apache.coyote.http11.filters.ChunkedInputFilter.readBytes(ChunkedInputFilter.java:268)
> > >> at org.apache.coyote.http11.filters.ChunkedInputFilter.doRead(ChunkedInputFilter.java:167)
> > >> at org.apache.coyote.http11.InternalNioInputBuffer.doRead(InternalNioInputBuffer.java:916)
> > >> at org.apache.coyote.Request.doRead(Request.java:427)
> > >> at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:304)
> > >> at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:419)
> > >> at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:327)
> > >> at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:162)
> > >> at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:365)
> > >> at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:110)
> > >> at com.ctc.wstx.io.MergedReader.read(MergedReader.java:101)
> > >> at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84)
> > >> at com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57)
> > >> at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:992)
> > >> at com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4628)
> > >> at com.ctc.wstx.sr.BasicStreamReader.readCoalescedText(BasicStreamReader.java:4126)
> > >> at com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3701)
> > >> at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3649)
> > >> ... 24 more
> > >>
> > >> Any thoughts to share? Is this a bug? A misconfiguration?
> > >>
> > >> Thanks,
> > >> Markus
> > >>
> >
>
RE: null:java.lang.RuntimeException: [was class
java.net.SocketTimeoutException] null
Posted by Markus Jelsma <ma...@openindex.io>.
Jack,
There are no peculiarities in the JVM graphs. Only increase in used threads and GC time. Heap space is collected quickly and doesn't suddenly increase. There's only 256MB available for the heap but it's fine.
Yonik,
I'll increase the time out to five seconds tomorrow and try to reproduce it with a low batch size of 32. Juding from what i've seen it should throw an error quickly with such a low batch size. However, what is timing out here? My client connection to the indexing node or something else that i don't see?
Unfortunately no Jetty here (yet).
Thanks
Markus
-----Original message-----
> From:Yonik Seeley <yo...@lucidimagination.com>
> Sent: Tue 07-Aug-2012 23:54
> To: solr-user@lucene.apache.org
> Subject: Re: null:java.lang.RuntimeException: [was class java.net.SocketTimeoutException] null
>
> Could this be just a simple case of a socket timeout? Can you raise
> the timout on request threads in Tomcat?
> It's a lot easier to reproduce/diagnose stuff like this when people
> use the stock jetty server shipped with Solr.
>
> -Yonik
> http://lucidimagination.com
>
>
> On Tue, Aug 7, 2012 at 5:39 PM, Markus Jelsma
> <ma...@openindex.io> wrote:
> > A signicant detail is the batch size which we set to 64 documents due to earlier memory limitations. We index segments of roughly 300-500k records each time. Lowering the batch size to 32 lead to an early internal server error and the stack trace below. Increasing it to 128 allowed us to index some more records but it still throws the error after 200k+ indexed records.
> >
> > Increasing it even more to 256 records per batch allowed us to index an entire segment without errors.
> >
> > Another detail is that we do not restart the cluster between indexing attempts so it seems that something only builds up during indexing (nothing seems to leak afterwards) and throws an error.
> >
> > Any hints?
> >
> > Thanks,
> > Markus
> >
> >
> >
> > -----Original message-----
> >> From:Markus Jelsma <ma...@openindex.io>
> >> Sent: Tue 07-Aug-2012 20:08
> >> To: solr-user@lucene.apache.org
> >> Subject: null:java.lang.RuntimeException: [was class java.net.SocketTimeoutException] null
> >>
> >> Hello,
> >>
> >> We sometimes see the error below in our `master` when indexing. Our master is currently the node we send documents to - we've not yet implemented CloudSolrServer in Apache Nutch. This causes the indexer to crash when using Nutch locally, the task is retried when running on Hadoop. We're running it locally in this test set up so there's only one indexing thread.
> >>
> >> Anyway, for me it's quite a cryptic error because i don't know what connection has timed out, i assume a connection from the indexing node to some other node in the cluster when it passes a document to the correct leader? Each node of the 10 node cluster has the same configuration, Tomcat is configured with maxThreads=512 and a time out of one second.
> >>
> >> We're using today's trunk in this test set up and we cannot reliably reproduce the error. We've seen the error before so it's not a very recent issue. No errors are found in the other node's logs.
> >>
> >> 2012-08-07 17:52:05,260 ERROR [solr.servlet.SolrDispatchFilter] - [http-8080-exec-6] - : null:java.lang.RuntimeException: [was class java.net.SocketTimeoutException] null
> >> at com.ctc.wstx.util.ExceptionUtil.throwRuntimeException(ExceptionUtil.java:18)
> >> at com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:731)
> >> at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3657)
> >> at com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:809)
> >> at org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:376)
> >> at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:229)
> >> at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:157)
> >> at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
> >> at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
> >> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
> >> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1656)
> >> at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:454)
> >> at io.openindex.solr.servlet.HttpResponseSolrDispatchFilter.doFilter(HttpResponseSolrDispatchFilter.java:219)
> >> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> >> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> >> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> >> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> >> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> >> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> >> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> >> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
> >> at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
> >> at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:744)
> >> at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274)
> >> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> at java.lang.Thread.run(Thread.java:662)
> >> Caused by: java.net.SocketTimeoutException
> >> at org.apache.tomcat.util.net.NioBlockingSelector.read(NioBlockingSelector.java:185)
> >> at org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:229)
> >> at org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:210)
> >> at org.apache.coyote.http11.InternalNioInputBuffer.readSocket(InternalNioInputBuffer.java:643)
> >> at org.apache.coyote.http11.InternalNioInputBuffer.fill(InternalNioInputBuffer.java:945)
> >> at org.apache.coyote.http11.InternalNioInputBuffer$SocketInputBuffer.doR
> >> ead(InternalNioInputBuffer.java:969)
> >> at org.apache.coyote.http11.filters.ChunkedInputFilter.readBytes(ChunkedInputFilter.java:268)
> >> at org.apache.coyote.http11.filters.ChunkedInputFilter.doRead(ChunkedInputFilter.java:167)
> >> at org.apache.coyote.http11.InternalNioInputBuffer.doRead(InternalNioInputBuffer.java:916)
> >> at org.apache.coyote.Request.doRead(Request.java:427)
> >> at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:304)
> >> at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:419)
> >> at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:327)
> >> at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:162)
> >> at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:365)
> >> at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:110)
> >> at com.ctc.wstx.io.MergedReader.read(MergedReader.java:101)
> >> at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84)
> >> at com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57)
> >> at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:992)
> >> at com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4628)
> >> at com.ctc.wstx.sr.BasicStreamReader.readCoalescedText(BasicStreamReader.java:4126)
> >> at com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3701)
> >> at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3649)
> >> ... 24 more
> >>
> >> Any thoughts to share? Is this a bug? A misconfiguration?
> >>
> >> Thanks,
> >> Markus
> >>
>
Re: null:java.lang.RuntimeException: [was class java.net.SocketTimeoutException]
null
Posted by Yonik Seeley <yo...@lucidimagination.com>.
Could this be just a simple case of a socket timeout? Can you raise
the timout on request threads in Tomcat?
It's a lot easier to reproduce/diagnose stuff like this when people
use the stock jetty server shipped with Solr.
-Yonik
http://lucidimagination.com
On Tue, Aug 7, 2012 at 5:39 PM, Markus Jelsma
<ma...@openindex.io> wrote:
> A signicant detail is the batch size which we set to 64 documents due to earlier memory limitations. We index segments of roughly 300-500k records each time. Lowering the batch size to 32 lead to an early internal server error and the stack trace below. Increasing it to 128 allowed us to index some more records but it still throws the error after 200k+ indexed records.
>
> Increasing it even more to 256 records per batch allowed us to index an entire segment without errors.
>
> Another detail is that we do not restart the cluster between indexing attempts so it seems that something only builds up during indexing (nothing seems to leak afterwards) and throws an error.
>
> Any hints?
>
> Thanks,
> Markus
>
>
>
> -----Original message-----
>> From:Markus Jelsma <ma...@openindex.io>
>> Sent: Tue 07-Aug-2012 20:08
>> To: solr-user@lucene.apache.org
>> Subject: null:java.lang.RuntimeException: [was class java.net.SocketTimeoutException] null
>>
>> Hello,
>>
>> We sometimes see the error below in our `master` when indexing. Our master is currently the node we send documents to - we've not yet implemented CloudSolrServer in Apache Nutch. This causes the indexer to crash when using Nutch locally, the task is retried when running on Hadoop. We're running it locally in this test set up so there's only one indexing thread.
>>
>> Anyway, for me it's quite a cryptic error because i don't know what connection has timed out, i assume a connection from the indexing node to some other node in the cluster when it passes a document to the correct leader? Each node of the 10 node cluster has the same configuration, Tomcat is configured with maxThreads=512 and a time out of one second.
>>
>> We're using today's trunk in this test set up and we cannot reliably reproduce the error. We've seen the error before so it's not a very recent issue. No errors are found in the other node's logs.
>>
>> 2012-08-07 17:52:05,260 ERROR [solr.servlet.SolrDispatchFilter] - [http-8080-exec-6] - : null:java.lang.RuntimeException: [was class java.net.SocketTimeoutException] null
>> at com.ctc.wstx.util.ExceptionUtil.throwRuntimeException(ExceptionUtil.java:18)
>> at com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:731)
>> at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3657)
>> at com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:809)
>> at org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:376)
>> at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:229)
>> at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:157)
>> at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
>> at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
>> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1656)
>> at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:454)
>> at io.openindex.solr.servlet.HttpResponseSolrDispatchFilter.doFilter(HttpResponseSolrDispatchFilter.java:219)
>> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
>> at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
>> at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:744)
>> at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.net.SocketTimeoutException
>> at org.apache.tomcat.util.net.NioBlockingSelector.read(NioBlockingSelector.java:185)
>> at org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:229)
>> at org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:210)
>> at org.apache.coyote.http11.InternalNioInputBuffer.readSocket(InternalNioInputBuffer.java:643)
>> at org.apache.coyote.http11.InternalNioInputBuffer.fill(InternalNioInputBuffer.java:945)
>> at org.apache.coyote.http11.InternalNioInputBuffer$SocketInputBuffer.doR
>> ead(InternalNioInputBuffer.java:969)
>> at org.apache.coyote.http11.filters.ChunkedInputFilter.readBytes(ChunkedInputFilter.java:268)
>> at org.apache.coyote.http11.filters.ChunkedInputFilter.doRead(ChunkedInputFilter.java:167)
>> at org.apache.coyote.http11.InternalNioInputBuffer.doRead(InternalNioInputBuffer.java:916)
>> at org.apache.coyote.Request.doRead(Request.java:427)
>> at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:304)
>> at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:419)
>> at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:327)
>> at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:162)
>> at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:365)
>> at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:110)
>> at com.ctc.wstx.io.MergedReader.read(MergedReader.java:101)
>> at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84)
>> at com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57)
>> at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:992)
>> at com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4628)
>> at com.ctc.wstx.sr.BasicStreamReader.readCoalescedText(BasicStreamReader.java:4126)
>> at com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3701)
>> at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3649)
>> ... 24 more
>>
>> Any thoughts to share? Is this a bug? A misconfiguration?
>>
>> Thanks,
>> Markus
>>
RE: null:java.lang.RuntimeException: [was class
java.net.SocketTimeoutException] null
Posted by Markus Jelsma <ma...@openindex.io>.
A signicant detail is the batch size which we set to 64 documents due to earlier memory limitations. We index segments of roughly 300-500k records each time. Lowering the batch size to 32 lead to an early internal server error and the stack trace below. Increasing it to 128 allowed us to index some more records but it still throws the error after 200k+ indexed records.
Increasing it even more to 256 records per batch allowed us to index an entire segment without errors.
Another detail is that we do not restart the cluster between indexing attempts so it seems that something only builds up during indexing (nothing seems to leak afterwards) and throws an error.
Any hints?
Thanks,
Markus
-----Original message-----
> From:Markus Jelsma <ma...@openindex.io>
> Sent: Tue 07-Aug-2012 20:08
> To: solr-user@lucene.apache.org
> Subject: null:java.lang.RuntimeException: [was class java.net.SocketTimeoutException] null
>
> Hello,
>
> We sometimes see the error below in our `master` when indexing. Our master is currently the node we send documents to - we've not yet implemented CloudSolrServer in Apache Nutch. This causes the indexer to crash when using Nutch locally, the task is retried when running on Hadoop. We're running it locally in this test set up so there's only one indexing thread.
>
> Anyway, for me it's quite a cryptic error because i don't know what connection has timed out, i assume a connection from the indexing node to some other node in the cluster when it passes a document to the correct leader? Each node of the 10 node cluster has the same configuration, Tomcat is configured with maxThreads=512 and a time out of one second.
>
> We're using today's trunk in this test set up and we cannot reliably reproduce the error. We've seen the error before so it's not a very recent issue. No errors are found in the other node's logs.
>
> 2012-08-07 17:52:05,260 ERROR [solr.servlet.SolrDispatchFilter] - [http-8080-exec-6] - : null:java.lang.RuntimeException: [was class java.net.SocketTimeoutException] null
> at com.ctc.wstx.util.ExceptionUtil.throwRuntimeException(ExceptionUtil.java:18)
> at com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:731)
> at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3657)
> at com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:809)
> at org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:376)
> at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:229)
> at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:157)
> at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
> at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1656)
> at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:454)
> at io.openindex.solr.servlet.HttpResponseSolrDispatchFilter.doFilter(HttpResponseSolrDispatchFilter.java:219)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
> at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
> at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:744)
> at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.net.SocketTimeoutException
> at org.apache.tomcat.util.net.NioBlockingSelector.read(NioBlockingSelector.java:185)
> at org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:229)
> at org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:210)
> at org.apache.coyote.http11.InternalNioInputBuffer.readSocket(InternalNioInputBuffer.java:643)
> at org.apache.coyote.http11.InternalNioInputBuffer.fill(InternalNioInputBuffer.java:945)
> at org.apache.coyote.http11.InternalNioInputBuffer$SocketInputBuffer.doR
> ead(InternalNioInputBuffer.java:969)
> at org.apache.coyote.http11.filters.ChunkedInputFilter.readBytes(ChunkedInputFilter.java:268)
> at org.apache.coyote.http11.filters.ChunkedInputFilter.doRead(ChunkedInputFilter.java:167)
> at org.apache.coyote.http11.InternalNioInputBuffer.doRead(InternalNioInputBuffer.java:916)
> at org.apache.coyote.Request.doRead(Request.java:427)
> at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:304)
> at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:419)
> at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:327)
> at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:162)
> at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:365)
> at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:110)
> at com.ctc.wstx.io.MergedReader.read(MergedReader.java:101)
> at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84)
> at com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57)
> at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:992)
> at com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4628)
> at com.ctc.wstx.sr.BasicStreamReader.readCoalescedText(BasicStreamReader.java:4126)
> at com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3701)
> at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3649)
> ... 24 more
>
> Any thoughts to share? Is this a bug? A misconfiguration?
>
> Thanks,
> Markus
>