You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Jim Murphy <ji...@pobox.com> on 2009/03/01 16:31:08 UTC
Tomcat 6 HTTP Connector Threads all blocked
I have a 100 thread HTTP connector pool that for some reason ends up with all
its threads blocked here:
java.net.SocketOutputStream.socketWrite0(Native Method)
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
java.net.SocketOutputStream.write(SocketOutputStream.java:136)
org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:737)
org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:434)
org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:349)
org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:761)
org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:126)
org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:570)
org.apache.coyote.Response.doWrite(Response.java:560)
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:353)
org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:434)
org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:309)
org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:273)
org.apache.catalina.connector.Response.finishResponse(Response.java:486)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:287)
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:584)
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
java.lang.Thread.run(Thread.java:619)
Any hints on what to try to diagnose?
Regards
Jim
--
View this message in context: http://www.nabble.com/Tomcat-6-HTTP-Connector-Threads-all-blocked-tp22274107p22274107.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Tomcat 6 HTTP Connector Threads all blocked
Posted by Jim Murphy <ji...@pobox.com>.
Thanks Yonik,
1. We're using rsync, snappuller/snapinstaller scripts for syncing masters
and slaves.
2. Ran jstack and found 2 kinds of stacks blocke on Socket.write0. The
first is the one I sent previously - and seems difficult to figure the
request type. This one is serializing a SolrDocumentList as JSON which makes
me believe its a query result on the slave side of things. Our inserts use
XML but our queries use JSON.
I suppose this question might be better posed on the Tomcat list, but I'm
surprised that the server sockets just block if its misbehaving clients. We
have ruby processes that are the only clients to Solr, so its possible that
there's bugs in ruby's Net:Http. But is there something I can enable
server-side to make the server more aggresive on closing connections?
The "other" stack trace:
- java.net.SocketOutputStream.socketWrite0(java.io.FileDescriptor, byte[],
int, int) @bci=0 (Interpreted frame)
- java.net.SocketOutputStream.socketWrite(byte[], int, int) @bci=44,
line=92 (Interpreted frame)
- java.net.SocketOutputStream.write(byte[], int, int) @bci=4, line=136
(Interpreted frame)
- org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(byte[], int,
int) @bci=11, line=737 (Interpreted frame)
- org.apache.tomcat.util.buf.ByteChunk.flushBuffer() @bci=71, line=434
(Compiled frame)
- org.apache.tomcat.util.buf.ByteChunk.append(byte[], int, int) @bci=159,
line=349 (Compiled frame)
-
org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(org.apache.tomcat.util.buf.ByteChunk,
org.apache.coyote.Response) @bci=31, line=761 (Interpreted frame)
-
org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(org.apache.tomcat.util.buf.ByteChunk,
org.apache.coyote.Response) @bci=107, line=127 (Interpreted frame)
-
org.apache.coyote.http11.InternalOutputBuffer.doWrite(org.apache.tomcat.util.buf.ByteChunk,
org.apache.coyote.Response) @bci=49, line=570 (Interpreted frame)
- org.apache.coyote.Response.doWrite(org.apache.tomcat.util.buf.ByteChunk)
@bci=6, line=560 (Compiled frame)
- org.apache.catalina.connector.OutputBuffer.realWriteBytes(byte[], int,
int) @bci=38, line=353 (Compiled frame)
- org.apache.tomcat.util.buf.ByteChunk.append(byte[], int, int) @bci=77,
line=325 (Compiled frame)
- org.apache.tomcat.util.buf.IntermediateOutputStream.write(byte[], int,
int) @bci=14, line=242 (Compiled frame)
- sun.nio.cs.StreamEncoder.writeBytes() @bci=120, line=202 (Compiled frame)
- sun.nio.cs.StreamEncoder.implFlushBuffer() @bci=11, line=272 (Compiled
frame)
- sun.nio.cs.StreamEncoder.implFlush() @bci=1, line=276 (Compiled frame)
- sun.nio.cs.StreamEncoder.flush() @bci=12, line=122 (Compiled frame)
- java.io.OutputStreamWriter.flush() @bci=4, line=212 (Compiled frame)
- org.apache.tomcat.util.buf.WriteConvertor.flush() @bci=1, line=191
(Compiled frame)
- org.apache.tomcat.util.buf.C2BConverter.flushBuffer() @bci=4, line=134
(Compiled frame)
- org.apache.catalina.connector.OutputBuffer.write(char[], int, int)
@bci=22, line=439 (Interpreted frame)
- org.apache.catalina.connector.CoyoteWriter.write(char[], int, int)
@bci=15, line=143 (Interpreted frame)
- org.apache.solr.common.util.FastWriter.write(char) @bci=25, line=55
(Compiled frame)
- org.apache.solr.request.JSONWriter.writeStr(java.lang.String,
java.lang.String, boolean) @bci=60, line=610 (Compiled frame)
- org.apache.solr.request.TextResponseWriter.writeVal(java.lang.String,
java.lang.Object) @bci=26, line=118 (Compiled frame)
- org.apache.solr.request.JSONWriter.writeSolrDocument(java.lang.String,
org.apache.solr.common.SolrDocument, java.util.Set, java.util.Map) @bci=148,
line=419 (Compiled frame)
-
org.apache.solr.request.JSONWriter.writeSolrDocumentList(java.lang.String,
org.apache.solr.common.SolrDocumentList, java.util.Set, java.util.Map)
@bci=240, line=547 (Interpreted frame)
- org.apache.solr.request.TextResponseWriter.writeVal(java.lang.String,
java.lang.Object) @bci=243, line=147 (Compiled frame)
-
org.apache.solr.request.JSONWriter.writeNamedListAsMapWithDups(java.lang.String,
org.apache.solr.common.util.NamedList) @bci=70, line=175 (Compiled frame)
- org.apache.solr.request.JSONWriter.writeNamedList(java.lang.String,
org.apache.solr.common.util.NamedList) @bci=10, line=288 (Interpreted frame)
- org.apache.solr.request.JSONWriter.writeResponse() @bci=45, line=88
(Interpreted frame)
- org.apache.solr.request.JSONResponseWriter.write(java.io.Writer,
org.apache.solr.request.SolrQueryRequest,
org.apache.solr.request.SolrQueryResponse) @bci=14, line=49 (Interpreted
frame)
-
org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=767, line=257
(Interpreted frame)
-
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse) @bci=117, line=235 (Interpreted frame)
-
org.apache.catalina.core.ApplicationFilterChain.doFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse) @bci=101, line=206 (Interpreted frame)
-
org.apache.catalina.core.StandardWrapperValve.invoke(org.apache.catalina.connector.Request,
org.apache.catalina.connector.Response) @bci=804, line=233 (Interpreted
frame)
-
org.apache.catalina.core.StandardContextValve.invoke(org.apache.catalina.connector.Request,
org.apache.catalina.connector.Response) @bci=285, line=175 (Interpreted
frame)
-
org.apache.catalina.core.StandardHostValve.invoke(org.apache.catalina.connector.Request,
org.apache.catalina.connector.Response) @bci=64, line=128 (Interpreted
frame)
-
org.apache.catalina.valves.ErrorReportValve.invoke(org.apache.catalina.connector.Request,
org.apache.catalina.connector.Response) @bci=6, line=102 (Interpreted frame)
-
org.apache.catalina.valves.AccessLogValve.invoke(org.apache.catalina.connector.Request,
org.apache.catalina.connector.Response) @bci=24, line=563 (Interpreted
frame)
-
org.apache.catalina.core.StandardEngineValve.invoke(org.apache.catalina.connector.Request,
org.apache.catalina.connector.Response) @bci=42, line=109 (Interpreted
frame)
-
org.apache.catalina.connector.CoyoteAdapter.service(org.apache.coyote.Request,
org.apache.coyote.Response) @bci=157, line=263 (Interpreted frame)
- org.apache.coyote.http11.Http11Processor.process(java.net.Socket)
@bci=432, line=844 (Interpreted frame)
-
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(java.net.Socket)
@bci=82, line=584 (Interpreted frame)
- org.apache.tomcat.util.net.JIoEndpoint$Worker.run() @bci=41, line=447
(Interpreted frame)
- java.lang.Thread.run() @bci=11, line=619 (Interpreted frame)
Yonik Seeley-2 wrote:
>
> On Sun, Mar 1, 2009 at 10:32 AM, Jim Murphy <ji...@pobox.com> wrote:
>> I should have said - tomcat is hosting 2 webapps a solr 1.3 master and
>> slave
>> - as separate web apps.
>
> Given the the socket writes are blocked, it appears like whatever is
> supposed to be reading the other endpoint isn't doing it's job.
>
> Are you using java-based replication? Do you know if these sockets
> that are blocking are from client queries or from replication
> requests? Splitting up the master and slave into separate JVMs might
> help shed some light on the situation.
>
> -Yonik
> http://www.lucidimagination.com
>
>
--
View this message in context: http://www.nabble.com/Tomcat-6-HTTP-Connector-Threads-all-blocked-tp22274107p22278035.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Tomcat 6 HTTP Connector Threads all blocked
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Sun, Mar 1, 2009 at 10:32 AM, Jim Murphy <ji...@pobox.com> wrote:
> I should have said - tomcat is hosting 2 webapps a solr 1.3 master and slave
> - as separate web apps.
Given the the socket writes are blocked, it appears like whatever is
supposed to be reading the other endpoint isn't doing it's job.
Are you using java-based replication? Do you know if these sockets
that are blocking are from client queries or from replication
requests? Splitting up the master and slave into separate JVMs might
help shed some light on the situation.
-Yonik
http://www.lucidimagination.com
Re: Tomcat 6 HTTP Connector Threads all blocked
Posted by Jim Murphy <ji...@pobox.com>.
I should have said - tomcat is hosting 2 webapps a solr 1.3 master and slave
- as separate web apps.
Looking for anything to try.
Jim
Jim Murphy wrote:
>
> I have a 100 thread HTTP connector pool that for some reason ends up with
> all its threads blocked here:
>
> java.net.SocketOutputStream.socketWrite0(Native Method)
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
> java.net.SocketOutputStream.write(SocketOutputStream.java:136)
> org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:737)
> org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:434)
> org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:349)
> org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:761)
> org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:126)
> org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:570)
> org.apache.coyote.Response.doWrite(Response.java:560)
> org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:353)
> org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:434)
> org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:309)
> org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:273)
> org.apache.catalina.connector.Response.finishResponse(Response.java:486)
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:287)
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:584)
> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
> java.lang.Thread.run(Thread.java:619)
>
>
> Any hints on what to try to diagnose?
>
> Regards
>
> Jim
>
--
View this message in context: http://www.nabble.com/Tomcat-6-HTTP-Connector-Threads-all-blocked-tp22274107p22274129.html
Sent from the Solr - User mailing list archive at Nabble.com.