You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Dušan Omerčević (JIRA)" <ji...@apache.org> on 2012/06/14 22:36:42 UTC
[jira] [Created] (SOLR-3544) Under heavy load json response is cut
at some arbitrary position
Dušan Omerčević created SOLR-3544:
-------------------------------------
Summary: Under heavy load json response is cut at some arbitrary position
Key: SOLR-3544
URL: https://issues.apache.org/jira/browse/SOLR-3544
Project: Solr
Issue Type: Bug
Components: search
Affects Versions: 3.1
Environment: Linux version 2.6.32-5-amd64 (Debian 2.6.32-38) (ben@decadent.org.uk) (gcc version 4.3.5 (Debian 4.3.5-4) )
Reporter: Dušan Omerčević
We query solr for 30K documents using json as the response format. Normally this works perfectly fine. But when the machine comes under heavy load (all cores utilized) the response got interrupted at arbitrary position. We circumvented the problem by switching to xml response format.
I've written the full description here: http://restreaming.wordpress.com/2012/06/14/the-curious-case-of-solr-malfunction/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3544) Under heavy load json response is
cut at some arbitrary position
Posted by "Dušan Omerčević (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13295913#comment-13295913 ]
Dušan Omerčević commented on SOLR-3544:
---------------------------------------
Ad 1) We're using Tomcat.
Ad 2) We fetch up to 40K documents with each document being a few KB in size. Total response is some 100MB in size. The cut off really happened at arbitrary position, from a few KB into the response up to several MB into the response.
Ad 3) I've checked it thoroughly and I couldn't find any pattern. It really seems that the problem is independent of the data.
Ad 4) The full message is:
2012-06-11 12:23:34 ERROR: ClientAbortException: java.net.SocketException: Broken pipe
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:370)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:323)
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:396)
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:385)
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:89)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:263)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:106)
at java.io.OutputStreamWriter.write(OutputStreamWriter.java:190)
at org.apache.solr.common.util.FastWriter.flush(FastWriter.java:113)
at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:344)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:265)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:873)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:750)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:432)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:347)
at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:773)
at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:127)
at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:583)
at org.apache.coyote.Response.doWrite(Response.java:560)
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:365)
... 25 more
2012-06-11 12:23:34 ERROR: Servlet.service() for servlet default threw exception
java.lang.IllegalStateException
at org.apache.catalina.connector.ResponseFacade.sendError(ResponseFacade.java:405)
at org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:380)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:283)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:873)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Thread.java:662)
But I believe that this is just a secondary effect of client closing the connection. It seems that the client is closing the connection because he doesn't have anything left to read. And it seems that it doesn't have anything to read because server abruptly stops sending data. In catalina.out there were no other error messages that would indicate why the server stopped sending data.
I hope this helps at least a bit. I didn't post this bug report because I expected it to be resolved (it's quite weird indeed), but mostly as reference point if somebody else is experiencing similar problems. But if it would get resolved that would be even better, of course :)
Dušan
> Under heavy load json response is cut at some arbitrary position
> ----------------------------------------------------------------
>
> Key: SOLR-3544
> URL: https://issues.apache.org/jira/browse/SOLR-3544
> Project: Solr
> Issue Type: Bug
> Components: search
> Affects Versions: 3.1
> Environment: Linux version 2.6.32-5-amd64 (Debian 2.6.32-38) (ben@decadent.org.uk) (gcc version 4.3.5 (Debian 4.3.5-4) )
> Reporter: Dušan Omerčević
>
> We query solr for 30K documents using json as the response format. Normally this works perfectly fine. But when the machine comes under heavy load (all cores utilized) the response got interrupted at arbitrary position. We circumvented the problem by switching to xml response format.
> I've written the full description here: http://restreaming.wordpress.com/2012/06/14/the-curious-case-of-solr-malfunction/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3544) Under heavy load json response is
cut at some arbitrary position
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13295307#comment-13295307 ]
Hoss Man commented on SOLR-3544:
--------------------------------
can you provide some more details please...
1) what servlet container are you using?
2) how big (in bytes) are the responses when they work? how big are they when "cut off"?
3) does the "cut off" always happen on/around a specific piece of markup? (ie: when closing a list or an object) or in the middle of arbitrary string values? is it possible there are certain byte sequences that always occur just before/at/after the cutoff happens?
4) your blog post mentions...
bq. Unfortunately, there was no indication of any malfunction in Solr except for the “Broken Pipe” notification that the client has closed the connection.
...where are you seeing this? in packet sniffing tool? in the solr logs? ... what exactly is the full message?
> Under heavy load json response is cut at some arbitrary position
> ----------------------------------------------------------------
>
> Key: SOLR-3544
> URL: https://issues.apache.org/jira/browse/SOLR-3544
> Project: Solr
> Issue Type: Bug
> Components: search
> Affects Versions: 3.1
> Environment: Linux version 2.6.32-5-amd64 (Debian 2.6.32-38) (ben@decadent.org.uk) (gcc version 4.3.5 (Debian 4.3.5-4) )
> Reporter: Dušan Omerčević
>
> We query solr for 30K documents using json as the response format. Normally this works perfectly fine. But when the machine comes under heavy load (all cores utilized) the response got interrupted at arbitrary position. We circumvented the problem by switching to xml response format.
> I've written the full description here: http://restreaming.wordpress.com/2012/06/14/the-curious-case-of-solr-malfunction/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org