You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Anoop Bhatti <an...@gmail.com> on 2008/11/24 20:45:00 UTC
broken socket in Jetty causing invalid XML ?
Hello Solr Community,
I'm getting the stracktrace below when adding docs using the
CommonsHttpSolrServer.add(Collection<org.apache.solr.common.SolrInputDocument>
docs)
method. The server doesn't seem to be able to recover from this error.
We are adding a collection with 1,000 SolrInputDocument's at a time.
I'm using Solr 1.3.0 and Java 1.6.0_07.
It seems that this problem occurs in Jetty when the TCP connection is
broken while the stream (from the add(...) method) is being read. The
XML read from the broken stream is not valid. Is this a correct
diagnosis?
Could this stacktrace be occurring when the max POST size has been
exceeded? I'm referring to the example/etc/jetty.xml file, which has
the setting:
<!-- Increase the maximum POST size to 1 MB to be able to handle
large shard requests -->
<Call class="java.lang.System" name="setProperty">
<Arg>org.mortbay.jetty.Request.maxFormContentSize</Arg>
<Arg>1000000</Arg>
</Call>
Right now maxFormContentSize is set to the default 1 MB on my server.
Also, in some cases I have two clients, could Jetty be blocking one
client and causing it to finally timeout?
This stacktrace doesn't happen right away, is occurs once the Lucene
indexes are about 30 GB.
Could the periodic merging of segments be the culprit?
I was also thinking that the problem could be with writing back the
response (the UpdateRequest).
Here's the gist of my Java client code:
CommonsHttpSolrServer solrServer = new CommonsHttpSolrServer(solrServerURL);
solrServer.setConnectionTimeout(100);
solrServer.setDefaultMaxConnectionsPerHost(100);
solrServer.setMaxTotalConnections(100);
solrServer.add(solrDocs); //the collection of docs
solrServer.commit();
And here's the stacktrace:
Nov 20, 2008 5:25:33 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/update params={wt=javabin&version=2.2}
status=0 QTime=469
Nov 20, 2008 5:25:37 PM org.apache.solr.common.SolrException log
SEVERE: com.ctc.wstx.exc.WstxIOException: null
at com.ctc.wstx.sr.StreamScanner.throwFromIOE(StreamScanner.java:708)
at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1086)
at org.apache.solr.handler.XmlUpdateRequestHandler.readDoc(XmlUpdateRequestHandler.java:321)
at org.apache.solr.handler.XmlUpdateRequestHandler.processUpdate(XmlUpdateRequestHandler.java:195)
at org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody(XmlUpdateRequestHandler.java:123)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
at org.mortbay.jetty.Server.handle(Server.java:285)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:835)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:641)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:202)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
Caused by: org.mortbay.jetty.EofException
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:302)
at org.mortbay.jetty.HttpParser$Input.blockForContent(HttpParser.java:919)
at org.mortbay.jetty.HttpParser$Input.read(HttpParser.java:897)
at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
at sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
at java.io.InputStreamReader.read(InputStreamReader.java:167)
at com.ctc.wstx.io.MergedReader.read(MergedReader.java:101)
at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84)
at com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57)
at com.ctc.wstx.sr.StreamScanner.loadMoreFromCurrent(StreamScanner.java:1046)
at com.ctc.wstx.sr.StreamScanner.loadMoreFromCurrent(StreamScanner.java:1053)
at com.ctc.wstx.sr.StreamScanner.getNextCharFromCurrent(StreamScanner.java:811)
at com.ctc.wstx.sr.BasicStreamReader.readEndElem(BasicStreamReader.java:3206)
at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2832)
at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1019)
... 24 more
Nov 20, 2008 5:25:37 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/update params={wt=javabin&version=2.2}
status=500 QTime=3215
Nov 20, 2008 5:25:37 PM org.apache.solr.common.SolrException log
SEVERE: com.ctc.wstx.exc.WstxIOException: null
at com.ctc.wstx.sr.StreamScanner.throwFromIOE(StreamScanner.java:708)
at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1086)
at org.apache.solr.handler.XmlUpdateRequestHandler.readDoc(XmlUpdateRequestHandler.java:321)
at org.apache.solr.handler.XmlUpdateRequestHandler.processUpdate(XmlUpdateRequestHandler.java:195)
at org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody(XmlUpdateRequestHandler.java:123)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
at org.mortbay.jetty.Server.handle(Server.java:285)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:835)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:641)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:202)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
Caused by: org.mortbay.jetty.EofException
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:302)
at org.mortbay.jetty.HttpParser$Input.blockForContent(HttpParser.java:919)
at org.mortbay.jetty.HttpParser$Input.read(HttpParser.java:897)
at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
at sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
at java.io.InputStreamReader.read(InputStreamReader.java:167)
at com.ctc.wstx.io.MergedReader.read(MergedReader.java:101)
at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84)
at com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57)
at com.ctc.wstx.sr.StreamScanner.loadMoreFromCurrent(StreamScanner.java:1046)
at com.ctc.wstx.sr.StreamScanner.loadMoreFromCurrent(StreamScanner.java:1053)
at com.ctc.wstx.sr.StreamScanner.getNextCharFromCurrent(StreamScanner.java:811)
at com.ctc.wstx.sr.BasicStreamReader.readEndElem(BasicStreamReader.java:3206)
at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2832)
at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1019)
... 24 more
Thanks,
Anoop Bhatti
--
Committed to open source technology.
Re: broken socket in Jetty causing invalid XML ?
Posted by Anoop Bhatti <an...@gmail.com>.
In case anyone has the same issue, it looks like switching Solr over
from Jetty to Tomcat fixed the problem.
I am using Tomcat v. 6.0.18.
Regards,
Anoop Bhatti
--
Committed to open source technology.
On Mon, Dec 1, 2008 at 10:51 AM, Anoop Bhatti <an...@gmail.com> wrote:
> I increased this param to 10MB and still got the same exception. I
> doubt my HTTP requests are exceeding 10MB. I can rerun everything
> again and log the sizes of the requests, just to be 100% sure, but
> this will take some time. The stacktrace appears when
> the Lucene index is around 30 GB. Could the size of the indexes be
> the cause of the problem? Any other ideas?
>
>
> Thanks again,
>
> Anoop Bhatti
> --
> Committed to open source technology.
>
>
>
>
> On Mon, Nov 24, 2008 at 11:44 PM, Yonik Seeley <yo...@apache.org> wrote:
>> I thought the Jetty maxFormContentSize was only for form data (not for
>> the POST body).
>> Does increasing this param help?
>>
>> -Yonik
>>
>>
>> On Mon, Nov 24, 2008 at 2:45 PM, Anoop Bhatti <an...@gmail.com> wrote:
>>> Hello Solr Community,
>>>
>>> I'm getting the stracktrace below when adding docs using the
>>> CommonsHttpSolrServer.add(Collection<org.apache.solr.common.SolrInputDocument>
>>> docs)
>>> method. The server doesn't seem to be able to recover from this error.
>>> We are adding a collection with 1,000 SolrInputDocument's at a time.
>>> I'm using Solr 1.3.0 and Java 1.6.0_07.
>>>
>>> It seems that this problem occurs in Jetty when the TCP connection is
>>> broken while the stream (from the add(...) method) is being read. The
>>> XML read from the broken stream is not valid. Is this a correct
>>> diagnosis?
>>>
>>> Could this stacktrace be occurring when the max POST size has been
>>> exceeded? I'm referring to the example/etc/jetty.xml file, which has
>>> the setting:
>>> <!-- Increase the maximum POST size to 1 MB to be able to handle
>>> large shard requests -->
>>> <Call class="java.lang.System" name="setProperty">
>>> <Arg>org.mortbay.jetty.Request.maxFormContentSize</Arg>
>>> <Arg>1000000</Arg>
>>> </Call>
>>> Right now maxFormContentSize is set to the default 1 MB on my server.
>>>
>>> Also, in some cases I have two clients, could Jetty be blocking one
>>> client and causing it to finally timeout?
>>>
>>> This stacktrace doesn't happen right away, is occurs once the Lucene
>>> indexes are about 30 GB.
>>> Could the periodic merging of segments be the culprit?
>>>
>>> I was also thinking that the problem could be with writing back the
>>> response (the UpdateRequest).
>>>
>>> Here's the gist of my Java client code:
>>>
>>> CommonsHttpSolrServer solrServer = new CommonsHttpSolrServer(solrServerURL);
>>> solrServer.setConnectionTimeout(100);
>>> solrServer.setDefaultMaxConnectionsPerHost(100);
>>> solrServer.setMaxTotalConnections(100);
>>> solrServer.add(solrDocs); //the collection of docs
>>> solrServer.commit();
>>>
>>> And here's the stacktrace:
>>>
>>> Nov 20, 2008 5:25:33 PM org.apache.solr.core.SolrCore execute
>>> INFO: [] webapp=/solr path=/update params={wt=javabin&version=2.2}
>>> status=0 QTime=469
>>> Nov 20, 2008 5:25:37 PM org.apache.solr.common.SolrException log
>>> SEVERE: com.ctc.wstx.exc.WstxIOException: null
>>> at com.ctc.wstx.sr.StreamScanner.throwFromIOE(StreamScanner.java:708)
>>> at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1086)
>>> at org.apache.solr.handler.XmlUpdateRequestHandler.readDoc(XmlUpdateRequestHandler.java:321)
>>> at org.apache.solr.handler.XmlUpdateRequestHandler.processUpdate(XmlUpdateRequestHandler.java:195)
>>> at org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody(XmlUpdateRequestHandler.java:123)
>>> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204)
>>> at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
>>> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
>>> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
>>> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
>>> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>>> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
>>> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
>>> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
>>> at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
>>> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
>>> at org.mortbay.jetty.Server.handle(Server.java:285)
>>> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
>>> at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:835)
>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:641)
>>> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:202)
>>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
>>> at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
>>> at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
>>> Caused by: org.mortbay.jetty.EofException
>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:302)
>>> at org.mortbay.jetty.HttpParser$Input.blockForContent(HttpParser.java:919)
>>> at org.mortbay.jetty.HttpParser$Input.read(HttpParser.java:897)
>>> at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
>>> at sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
>>> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
>>> at java.io.InputStreamReader.read(InputStreamReader.java:167)
>>> at com.ctc.wstx.io.MergedReader.read(MergedReader.java:101)
>>> at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84)
>>> at com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57)
>>> at com.ctc.wstx.sr.StreamScanner.loadMoreFromCurrent(StreamScanner.java:1046)
>>> at com.ctc.wstx.sr.StreamScanner.loadMoreFromCurrent(StreamScanner.java:1053)
>>> at com.ctc.wstx.sr.StreamScanner.getNextCharFromCurrent(StreamScanner.java:811)
>>> at com.ctc.wstx.sr.BasicStreamReader.readEndElem(BasicStreamReader.java:3206)
>>> at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2832)
>>> at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1019)
>>> ... 24 more
>>>
>>> Nov 20, 2008 5:25:37 PM org.apache.solr.core.SolrCore execute
>>> INFO: [] webapp=/solr path=/update params={wt=javabin&version=2.2}
>>> status=500 QTime=3215
>>> Nov 20, 2008 5:25:37 PM org.apache.solr.common.SolrException log
>>> SEVERE: com.ctc.wstx.exc.WstxIOException: null
>>> at com.ctc.wstx.sr.StreamScanner.throwFromIOE(StreamScanner.java:708)
>>> at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1086)
>>> at org.apache.solr.handler.XmlUpdateRequestHandler.readDoc(XmlUpdateRequestHandler.java:321)
>>> at org.apache.solr.handler.XmlUpdateRequestHandler.processUpdate(XmlUpdateRequestHandler.java:195)
>>> at org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody(XmlUpdateRequestHandler.java:123)
>>> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204)
>>> at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
>>> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
>>> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
>>> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
>>> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>>> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
>>> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
>>> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
>>> at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
>>> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
>>> at org.mortbay.jetty.Server.handle(Server.java:285)
>>> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
>>> at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:835)
>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:641)
>>> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:202)
>>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
>>> at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
>>> at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
>>> Caused by: org.mortbay.jetty.EofException
>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:302)
>>> at org.mortbay.jetty.HttpParser$Input.blockForContent(HttpParser.java:919)
>>> at org.mortbay.jetty.HttpParser$Input.read(HttpParser.java:897)
>>> at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
>>> at sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
>>> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
>>> at java.io.InputStreamReader.read(InputStreamReader.java:167)
>>> at com.ctc.wstx.io.MergedReader.read(MergedReader.java:101)
>>> at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84)
>>> at com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57)
>>> at com.ctc.wstx.sr.StreamScanner.loadMoreFromCurrent(StreamScanner.java:1046)
>>> at com.ctc.wstx.sr.StreamScanner.loadMoreFromCurrent(StreamScanner.java:1053)
>>> at com.ctc.wstx.sr.StreamScanner.getNextCharFromCurrent(StreamScanner.java:811)
>>> at com.ctc.wstx.sr.BasicStreamReader.readEndElem(BasicStreamReader.java:3206)
>>> at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2832)
>>> at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1019)
>>> ... 24 more
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Anoop Bhatti
>>> --
>>> Committed to open source technology.
>>>
>>
>
Re: broken socket in Jetty causing invalid XML ?
Posted by Anoop Bhatti <an...@gmail.com>.
I increased this param to 10MB and still got the same exception. I
doubt my HTTP requests are exceeding 10MB. I can rerun everything
again and log the sizes of the requests, just to be 100% sure, but
this will take some time. The stacktrace appears when
the Lucene index is around 30 GB. Could the size of the indexes be
the cause of the problem? Any other ideas?
Thanks again,
Anoop Bhatti
--
Committed to open source technology.
On Mon, Nov 24, 2008 at 11:44 PM, Yonik Seeley <yo...@apache.org> wrote:
> I thought the Jetty maxFormContentSize was only for form data (not for
> the POST body).
> Does increasing this param help?
>
> -Yonik
>
>
> On Mon, Nov 24, 2008 at 2:45 PM, Anoop Bhatti <an...@gmail.com> wrote:
>> Hello Solr Community,
>>
>> I'm getting the stracktrace below when adding docs using the
>> CommonsHttpSolrServer.add(Collection<org.apache.solr.common.SolrInputDocument>
>> docs)
>> method. The server doesn't seem to be able to recover from this error.
>> We are adding a collection with 1,000 SolrInputDocument's at a time.
>> I'm using Solr 1.3.0 and Java 1.6.0_07.
>>
>> It seems that this problem occurs in Jetty when the TCP connection is
>> broken while the stream (from the add(...) method) is being read. The
>> XML read from the broken stream is not valid. Is this a correct
>> diagnosis?
>>
>> Could this stacktrace be occurring when the max POST size has been
>> exceeded? I'm referring to the example/etc/jetty.xml file, which has
>> the setting:
>> <!-- Increase the maximum POST size to 1 MB to be able to handle
>> large shard requests -->
>> <Call class="java.lang.System" name="setProperty">
>> <Arg>org.mortbay.jetty.Request.maxFormContentSize</Arg>
>> <Arg>1000000</Arg>
>> </Call>
>> Right now maxFormContentSize is set to the default 1 MB on my server.
>>
>> Also, in some cases I have two clients, could Jetty be blocking one
>> client and causing it to finally timeout?
>>
>> This stacktrace doesn't happen right away, is occurs once the Lucene
>> indexes are about 30 GB.
>> Could the periodic merging of segments be the culprit?
>>
>> I was also thinking that the problem could be with writing back the
>> response (the UpdateRequest).
>>
>> Here's the gist of my Java client code:
>>
>> CommonsHttpSolrServer solrServer = new CommonsHttpSolrServer(solrServerURL);
>> solrServer.setConnectionTimeout(100);
>> solrServer.setDefaultMaxConnectionsPerHost(100);
>> solrServer.setMaxTotalConnections(100);
>> solrServer.add(solrDocs); //the collection of docs
>> solrServer.commit();
>>
>> And here's the stacktrace:
>>
>> Nov 20, 2008 5:25:33 PM org.apache.solr.core.SolrCore execute
>> INFO: [] webapp=/solr path=/update params={wt=javabin&version=2.2}
>> status=0 QTime=469
>> Nov 20, 2008 5:25:37 PM org.apache.solr.common.SolrException log
>> SEVERE: com.ctc.wstx.exc.WstxIOException: null
>> at com.ctc.wstx.sr.StreamScanner.throwFromIOE(StreamScanner.java:708)
>> at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1086)
>> at org.apache.solr.handler.XmlUpdateRequestHandler.readDoc(XmlUpdateRequestHandler.java:321)
>> at org.apache.solr.handler.XmlUpdateRequestHandler.processUpdate(XmlUpdateRequestHandler.java:195)
>> at org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody(XmlUpdateRequestHandler.java:123)
>> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204)
>> at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
>> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
>> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
>> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
>> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
>> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
>> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
>> at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
>> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
>> at org.mortbay.jetty.Server.handle(Server.java:285)
>> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
>> at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:835)
>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:641)
>> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:202)
>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
>> at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
>> at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
>> Caused by: org.mortbay.jetty.EofException
>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:302)
>> at org.mortbay.jetty.HttpParser$Input.blockForContent(HttpParser.java:919)
>> at org.mortbay.jetty.HttpParser$Input.read(HttpParser.java:897)
>> at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
>> at sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
>> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
>> at java.io.InputStreamReader.read(InputStreamReader.java:167)
>> at com.ctc.wstx.io.MergedReader.read(MergedReader.java:101)
>> at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84)
>> at com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57)
>> at com.ctc.wstx.sr.StreamScanner.loadMoreFromCurrent(StreamScanner.java:1046)
>> at com.ctc.wstx.sr.StreamScanner.loadMoreFromCurrent(StreamScanner.java:1053)
>> at com.ctc.wstx.sr.StreamScanner.getNextCharFromCurrent(StreamScanner.java:811)
>> at com.ctc.wstx.sr.BasicStreamReader.readEndElem(BasicStreamReader.java:3206)
>> at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2832)
>> at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1019)
>> ... 24 more
>>
>> Nov 20, 2008 5:25:37 PM org.apache.solr.core.SolrCore execute
>> INFO: [] webapp=/solr path=/update params={wt=javabin&version=2.2}
>> status=500 QTime=3215
>> Nov 20, 2008 5:25:37 PM org.apache.solr.common.SolrException log
>> SEVERE: com.ctc.wstx.exc.WstxIOException: null
>> at com.ctc.wstx.sr.StreamScanner.throwFromIOE(StreamScanner.java:708)
>> at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1086)
>> at org.apache.solr.handler.XmlUpdateRequestHandler.readDoc(XmlUpdateRequestHandler.java:321)
>> at org.apache.solr.handler.XmlUpdateRequestHandler.processUpdate(XmlUpdateRequestHandler.java:195)
>> at org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody(XmlUpdateRequestHandler.java:123)
>> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204)
>> at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
>> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
>> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
>> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
>> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
>> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
>> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
>> at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
>> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
>> at org.mortbay.jetty.Server.handle(Server.java:285)
>> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
>> at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:835)
>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:641)
>> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:202)
>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
>> at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
>> at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
>> Caused by: org.mortbay.jetty.EofException
>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:302)
>> at org.mortbay.jetty.HttpParser$Input.blockForContent(HttpParser.java:919)
>> at org.mortbay.jetty.HttpParser$Input.read(HttpParser.java:897)
>> at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
>> at sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
>> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
>> at java.io.InputStreamReader.read(InputStreamReader.java:167)
>> at com.ctc.wstx.io.MergedReader.read(MergedReader.java:101)
>> at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84)
>> at com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57)
>> at com.ctc.wstx.sr.StreamScanner.loadMoreFromCurrent(StreamScanner.java:1046)
>> at com.ctc.wstx.sr.StreamScanner.loadMoreFromCurrent(StreamScanner.java:1053)
>> at com.ctc.wstx.sr.StreamScanner.getNextCharFromCurrent(StreamScanner.java:811)
>> at com.ctc.wstx.sr.BasicStreamReader.readEndElem(BasicStreamReader.java:3206)
>> at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2832)
>> at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1019)
>> ... 24 more
>>
>>
>>
>> Thanks,
>>
>> Anoop Bhatti
>> --
>> Committed to open source technology.
>>
>
Re: broken socket in Jetty causing invalid XML ?
Posted by Yonik Seeley <yo...@apache.org>.
I thought the Jetty maxFormContentSize was only for form data (not for
the POST body).
Does increasing this param help?
-Yonik
On Mon, Nov 24, 2008 at 2:45 PM, Anoop Bhatti <an...@gmail.com> wrote:
> Hello Solr Community,
>
> I'm getting the stracktrace below when adding docs using the
> CommonsHttpSolrServer.add(Collection<org.apache.solr.common.SolrInputDocument>
> docs)
> method. The server doesn't seem to be able to recover from this error.
> We are adding a collection with 1,000 SolrInputDocument's at a time.
> I'm using Solr 1.3.0 and Java 1.6.0_07.
>
> It seems that this problem occurs in Jetty when the TCP connection is
> broken while the stream (from the add(...) method) is being read. The
> XML read from the broken stream is not valid. Is this a correct
> diagnosis?
>
> Could this stacktrace be occurring when the max POST size has been
> exceeded? I'm referring to the example/etc/jetty.xml file, which has
> the setting:
> <!-- Increase the maximum POST size to 1 MB to be able to handle
> large shard requests -->
> <Call class="java.lang.System" name="setProperty">
> <Arg>org.mortbay.jetty.Request.maxFormContentSize</Arg>
> <Arg>1000000</Arg>
> </Call>
> Right now maxFormContentSize is set to the default 1 MB on my server.
>
> Also, in some cases I have two clients, could Jetty be blocking one
> client and causing it to finally timeout?
>
> This stacktrace doesn't happen right away, is occurs once the Lucene
> indexes are about 30 GB.
> Could the periodic merging of segments be the culprit?
>
> I was also thinking that the problem could be with writing back the
> response (the UpdateRequest).
>
> Here's the gist of my Java client code:
>
> CommonsHttpSolrServer solrServer = new CommonsHttpSolrServer(solrServerURL);
> solrServer.setConnectionTimeout(100);
> solrServer.setDefaultMaxConnectionsPerHost(100);
> solrServer.setMaxTotalConnections(100);
> solrServer.add(solrDocs); //the collection of docs
> solrServer.commit();
>
> And here's the stacktrace:
>
> Nov 20, 2008 5:25:33 PM org.apache.solr.core.SolrCore execute
> INFO: [] webapp=/solr path=/update params={wt=javabin&version=2.2}
> status=0 QTime=469
> Nov 20, 2008 5:25:37 PM org.apache.solr.common.SolrException log
> SEVERE: com.ctc.wstx.exc.WstxIOException: null
> at com.ctc.wstx.sr.StreamScanner.throwFromIOE(StreamScanner.java:708)
> at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1086)
> at org.apache.solr.handler.XmlUpdateRequestHandler.readDoc(XmlUpdateRequestHandler.java:321)
> at org.apache.solr.handler.XmlUpdateRequestHandler.processUpdate(XmlUpdateRequestHandler.java:195)
> at org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody(XmlUpdateRequestHandler.java:123)
> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204)
> at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
> at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
> at org.mortbay.jetty.Server.handle(Server.java:285)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
> at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:835)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:641)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:202)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
> at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
> at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
> Caused by: org.mortbay.jetty.EofException
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:302)
> at org.mortbay.jetty.HttpParser$Input.blockForContent(HttpParser.java:919)
> at org.mortbay.jetty.HttpParser$Input.read(HttpParser.java:897)
> at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
> at sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
> at java.io.InputStreamReader.read(InputStreamReader.java:167)
> at com.ctc.wstx.io.MergedReader.read(MergedReader.java:101)
> at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84)
> at com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57)
> at com.ctc.wstx.sr.StreamScanner.loadMoreFromCurrent(StreamScanner.java:1046)
> at com.ctc.wstx.sr.StreamScanner.loadMoreFromCurrent(StreamScanner.java:1053)
> at com.ctc.wstx.sr.StreamScanner.getNextCharFromCurrent(StreamScanner.java:811)
> at com.ctc.wstx.sr.BasicStreamReader.readEndElem(BasicStreamReader.java:3206)
> at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2832)
> at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1019)
> ... 24 more
>
> Nov 20, 2008 5:25:37 PM org.apache.solr.core.SolrCore execute
> INFO: [] webapp=/solr path=/update params={wt=javabin&version=2.2}
> status=500 QTime=3215
> Nov 20, 2008 5:25:37 PM org.apache.solr.common.SolrException log
> SEVERE: com.ctc.wstx.exc.WstxIOException: null
> at com.ctc.wstx.sr.StreamScanner.throwFromIOE(StreamScanner.java:708)
> at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1086)
> at org.apache.solr.handler.XmlUpdateRequestHandler.readDoc(XmlUpdateRequestHandler.java:321)
> at org.apache.solr.handler.XmlUpdateRequestHandler.processUpdate(XmlUpdateRequestHandler.java:195)
> at org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody(XmlUpdateRequestHandler.java:123)
> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204)
> at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
> at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
> at org.mortbay.jetty.Server.handle(Server.java:285)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
> at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:835)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:641)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:202)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
> at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
> at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
> Caused by: org.mortbay.jetty.EofException
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:302)
> at org.mortbay.jetty.HttpParser$Input.blockForContent(HttpParser.java:919)
> at org.mortbay.jetty.HttpParser$Input.read(HttpParser.java:897)
> at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
> at sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
> at java.io.InputStreamReader.read(InputStreamReader.java:167)
> at com.ctc.wstx.io.MergedReader.read(MergedReader.java:101)
> at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84)
> at com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57)
> at com.ctc.wstx.sr.StreamScanner.loadMoreFromCurrent(StreamScanner.java:1046)
> at com.ctc.wstx.sr.StreamScanner.loadMoreFromCurrent(StreamScanner.java:1053)
> at com.ctc.wstx.sr.StreamScanner.getNextCharFromCurrent(StreamScanner.java:811)
> at com.ctc.wstx.sr.BasicStreamReader.readEndElem(BasicStreamReader.java:3206)
> at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2832)
> at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1019)
> ... 24 more
>
>
>
> Thanks,
>
> Anoop Bhatti
> --
> Committed to open source technology.
>