You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hc.apache.org by Supun Kamburugamuva <su...@gmail.com> on 2010/05/15 10:32:50 UTC
I/O Dispatcher thread blocks on a read
Hi,
One of the I/O dispatcher threads in Synapse is blocked
in SharedInputBuffer.read method. Here is the thread dump. I would really
appreciate if you can point me to the correct path for figuring out the root
cause of this.
This dispatcher is sending the message out from synapse. In this case it
tries to read the incoming message read by the listeners to send it out.
Thanks,
Supun..
"http-Sender I/O dispatcher-2" prio=10 tid=0x08505800 nid=0x1718 in
Object.wait() [0x613ac000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x8213c928> (a java.lang.Object)
at java.lang.Object.wait(Object.java:485)
at
org.apache.http.nio.util.SharedInputBuffer.waitForData(SharedInputBuffer.java:132)
- locked <0x8213c928> (a java.lang.Object)
at
org.apache.http.nio.util.SharedInputBuffer.read(SharedInputBuffer.java:192)
- locked <0x8213c928> (a java.lang.Object)
at
org.apache.http.nio.entity.ContentInputStream.read(ContentInputStream.java:64)
at
java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:221)
at
java.util.zip.InflaterInputStream.read(InflaterInputStream.java:141)
at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:92)
at java.io.FilterInputStream.read(FilterInputStream.java:90)
at
org.wso2.carbon.relay.BinaryRelayBuilder.readAllFromInputSteam(BinaryRelayBuilder.java:48)
at
org.wso2.carbon.relay.StreamingOnRequestDataSource.getInputStream(StreamingOnRequestDataSource.java:59)
at javax.activation.DataHandler.writeTo(DataHandler.java:305)
at
org.wso2.carbon.relay.ExpandingMessageFormatter.findAndWrite2OutputStream(ExpandingMessageFormatter.java:170)
at
org.wso2.carbon.relay.ExpandingMessageFormatter.writeTo(ExpandingMessageFormatter.java:99)
at
org.apache.synapse.transport.nhttp.util.GzipMessageFormatterDecorator.writeTo(GzipMessageFormatterDecorator.java:66)
at
org.apache.synapse.transport.nhttp.Axis2HttpRequest.setStreamAsTempData(Axis2HttpRequest.java:332)
at
org.apache.synapse.transport.nhttp.Axis2HttpRequest.getRequest(Axis2HttpRequest.java:169)
at
org.apache.synapse.transport.nhttp.ClientHandler.processConnection(ClientHandler.java:194)
at
org.apache.synapse.transport.nhttp.ClientHandler.connected(ClientHandler.java:160)
at
org.apache.http.impl.nio.DefaultClientIOEventDispatch.connected(DefaultClientIOEventDispatch.java:134)
at
org.apache.http.impl.nio.reactor.BaseIOReactor.sessionCreated(BaseIOReactor.java:284)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processNewChannels(AbstractIOReactor.java:423)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:286)
at
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:528)
at java.lang.Thread.run(Thread.java:619)
--
Software Engineer, WSO2 Inc
http://wso2.org
supunk.blogspot.com
Re: I/O Dispatcher thread blocks on a read
Posted by Supun Kamburugamuva <su...@gmail.com>.
We could resolve the issue. It was a issue with converting HTTP 1.1 to HTTP
1.0. We were trying to stream a HTTP 1.1 response as a HTTP 1.0 request.
This caused the IO Dispatchers to block.
Thanks Oleg for all the help.
Supun..
On Mon, May 17, 2010 at 3:37 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
> On Mon, 2010-05-17 at 15:08 +0530, Supun Kamburugamuva wrote:
> > On Sun, May 16, 2010 at 2:41 PM, Oleg Kalnichevski <ol...@apache.org>
> wrote:
> >
> > > On Sun, 2010-05-16 at 10:16 +0530, Supun Kamburugamuva wrote:
> > > > Hi Oleg,
> > > >
> > > > A quick question. In this system we have two IO dispatchers. Only
> > > > one dispatcher is blocked. But all the workers are waiting on a lock
> > > > unlocked by the dispatchers. If one dispatcher is blocked, what
> happen to
> > > > the other dispatcher? Is it going to work normally?
> > > >
> > >
> > > No, it is not. A dispatch thread can be responsible for selecting I/O
> > > events for many non-blocking channels. If the tread gets stuck, all its
> > > connections become blocked. For example, if you have 2 dispatch threads
> > > and one gets stuck, 50% of open connections are dead.
> > >
> > > To be more clear I will as the question in a different way,
> >
> > Lets say a dispatcher thread gets blocked and there are some consumer
> > threads working on the connections processed by this dispatcher. Now the
> > dispatcher is blocked and consumers are waiting for a event fired by a
> > dispatcher to continue.
> >
> > Do you say that the consumers will block fore-ever because this
> particular
> > dispatcher is blocked even though there is another dispatcher running in
> the
> > system.
> >
>
> Yes, they will. Blocked dispatch thread spells big trouble.
>
> Oleg
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
>
--
Software Engineer, WSO2 Inc
http://wso2.org
supunk.blogspot.com
Re: I/O Dispatcher thread blocks on a read
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Mon, 2010-05-17 at 15:08 +0530, Supun Kamburugamuva wrote:
> On Sun, May 16, 2010 at 2:41 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
>
> > On Sun, 2010-05-16 at 10:16 +0530, Supun Kamburugamuva wrote:
> > > Hi Oleg,
> > >
> > > A quick question. In this system we have two IO dispatchers. Only
> > > one dispatcher is blocked. But all the workers are waiting on a lock
> > > unlocked by the dispatchers. If one dispatcher is blocked, what happen to
> > > the other dispatcher? Is it going to work normally?
> > >
> >
> > No, it is not. A dispatch thread can be responsible for selecting I/O
> > events for many non-blocking channels. If the tread gets stuck, all its
> > connections become blocked. For example, if you have 2 dispatch threads
> > and one gets stuck, 50% of open connections are dead.
> >
> > To be more clear I will as the question in a different way,
>
> Lets say a dispatcher thread gets blocked and there are some consumer
> threads working on the connections processed by this dispatcher. Now the
> dispatcher is blocked and consumers are waiting for a event fired by a
> dispatcher to continue.
>
> Do you say that the consumers will block fore-ever because this particular
> dispatcher is blocked even though there is another dispatcher running in the
> system.
>
Yes, they will. Blocked dispatch thread spells big trouble.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: I/O Dispatcher thread blocks on a read
Posted by Supun Kamburugamuva <su...@gmail.com>.
On Sun, May 16, 2010 at 2:41 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
> On Sun, 2010-05-16 at 10:16 +0530, Supun Kamburugamuva wrote:
> > Hi Oleg,
> >
> > A quick question. In this system we have two IO dispatchers. Only
> > one dispatcher is blocked. But all the workers are waiting on a lock
> > unlocked by the dispatchers. If one dispatcher is blocked, what happen to
> > the other dispatcher? Is it going to work normally?
> >
>
> No, it is not. A dispatch thread can be responsible for selecting I/O
> events for many non-blocking channels. If the tread gets stuck, all its
> connections become blocked. For example, if you have 2 dispatch threads
> and one gets stuck, 50% of open connections are dead.
>
> To be more clear I will as the question in a different way,
Lets say a dispatcher thread gets blocked and there are some consumer
threads working on the connections processed by this dispatcher. Now the
dispatcher is blocked and consumers are waiting for a event fired by a
dispatcher to continue.
Do you say that the consumers will block fore-ever because this particular
dispatcher is blocked even though there is another dispatcher running in the
system.
Thanks,
Supun..
> Oleg
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
>
--
Software Engineer, WSO2 Inc
http://wso2.org
supunk.blogspot.com
Re: I/O Dispatcher thread blocks on a read
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Sun, 2010-05-16 at 10:16 +0530, Supun Kamburugamuva wrote:
> Hi Oleg,
>
> A quick question. In this system we have two IO dispatchers. Only
> one dispatcher is blocked. But all the workers are waiting on a lock
> unlocked by the dispatchers. If one dispatcher is blocked, what happen to
> the other dispatcher? Is it going to work normally?
>
No, it is not. A dispatch thread can be responsible for selecting I/O
events for many non-blocking channels. If the tread gets stuck, all its
connections become blocked. For example, if you have 2 dispatch threads
and one gets stuck, 50% of open connections are dead.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: I/O Dispatcher thread blocks on a read
Posted by Supun Kamburugamuva <su...@gmail.com>.
Hi Oleg,
A quick question. In this system we have two IO dispatchers. Only
one dispatcher is blocked. But all the workers are waiting on a lock
unlocked by the dispatchers. If one dispatcher is blocked, what happen to
the other dispatcher? Is it going to work normally?
Thanks,
Supun..
On Sat, May 15, 2010 at 6:49 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
> On Sat, 2010-05-15 at 18:28 +0530, Supun Kamburugamuva wrote:
> > Hi Oleg,
> >
> > There is an exception in the listening side of the transport. Do you
> think
> > this is the cause of the error?
> >
> > Thanks,
> > Supun..
> >
>
> Most likely it is. The real problem is, however, that the producer
> thread failed to shut down the shared buffer, thus leaving the consumer
> thread stuck unaware of the error condition.
>
> Hope this helps
>
> Oleg
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
>
--
Software Engineer, WSO2 Inc
http://wso2.org
supunk.blogspot.com
Re: I/O Dispatcher thread blocks on a read
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Sat, 2010-05-15 at 18:28 +0530, Supun Kamburugamuva wrote:
> Hi Oleg,
>
> There is an exception in the listening side of the transport. Do you think
> this is the cause of the error?
>
> Thanks,
> Supun..
>
Most likely it is. The real problem is, however, that the producer
thread failed to shut down the shared buffer, thus leaving the consumer
thread stuck unaware of the error condition.
Hope this helps
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: I/O Dispatcher thread blocks on a read
Posted by Supun Kamburugamuva <su...@gmail.com>.
Hi Oleg,
There is an exception in the listening side of the transport. Do you think
this is the cause of the error?
Thanks,
Supun..
2010-05-13 23:39:40,076 [-] [http-Listener I/O dispatcher-1] ERROR
ServerHandler I/O error: Input length = 1
java.nio.charset.MalformedInputException: Input length = 1
at java.nio.charset.CoderResult.throwException(CoderResult.java:260)
at
org.apache.http.impl.nio.reactor.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:189)
at
org.apache.http.impl.nio.codecs.AbstractMessageParser.parse(AbstractMessageParser.java:169)
at
org.apache.http.impl.nio.DefaultNHttpServerConnection.consumeInput(DefaultNHttpServerConnection.java:152)
at
org.apache.http.impl.nio.DefaultServerIOEventDispatch.inputReady(DefaultServerIOEventDispatch.java:147)
at
org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:161)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:339)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:319)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:275)
at
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:528)
at java.lang.Thread.run(Thread.java:619)
On Sat, May 15, 2010 at 4:28 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
> On Sat, 2010-05-15 at 14:02 +0530, Supun Kamburugamuva wrote:
> > Hi,
> >
> > One of the I/O dispatcher threads in Synapse is blocked
> > in SharedInputBuffer.read method. Here is the thread dump. I would really
> > appreciate if you can point me to the correct path for figuring out the
> root
> > cause of this.
> >
> > This dispatcher is sending the message out from synapse. In this case it
> > tries to read the incoming message read by the listeners to send it out.
> >
> > Thanks,
> > Supun..
>
> Supun,
>
> SharedInputBuffer is shared by two (or more) threads: a data consumer
> and a data producer. The data consumer thread is stuck waiting for more
> data to arrive because the data producer thread does not fill the buffer
> with more data.
>
> Try to find out why there is no data in the shared buffer.
>
> Oleg
>
> >
> > "http-Sender I/O dispatcher-2" prio=10 tid=0x08505800 nid=0x1718 in
> > Object.wait() [0x613ac000]
> > java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <0x8213c928> (a java.lang.Object)
> > at java.lang.Object.wait(Object.java:485)
> > at
> >
> org.apache.http.nio.util.SharedInputBuffer.waitForData(SharedInputBuffer.java:132)
> > - locked <0x8213c928> (a java.lang.Object)
> > at
> >
> org.apache.http.nio.util.SharedInputBuffer.read(SharedInputBuffer.java:192)
> > - locked <0x8213c928> (a java.lang.Object)
> > at
> >
> org.apache.http.nio.entity.ContentInputStream.read(ContentInputStream.java:64)
> > at
> > java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:221)
> > at
> > java.util.zip.InflaterInputStream.read(InflaterInputStream.java:141)
> > at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:92)
> > at java.io.FilterInputStream.read(FilterInputStream.java:90)
> > at
> >
> org.wso2.carbon.relay.BinaryRelayBuilder.readAllFromInputSteam(BinaryRelayBuilder.java:48)
> > at
> >
> org.wso2.carbon.relay.StreamingOnRequestDataSource.getInputStream(StreamingOnRequestDataSource.java:59)
> > at javax.activation.DataHandler.writeTo(DataHandler.java:305)
> > at
> >
> org.wso2.carbon.relay.ExpandingMessageFormatter.findAndWrite2OutputStream(ExpandingMessageFormatter.java:170)
> > at
> >
> org.wso2.carbon.relay.ExpandingMessageFormatter.writeTo(ExpandingMessageFormatter.java:99)
> > at
> >
> org.apache.synapse.transport.nhttp.util.GzipMessageFormatterDecorator.writeTo(GzipMessageFormatterDecorator.java:66)
> > at
> >
> org.apache.synapse.transport.nhttp.Axis2HttpRequest.setStreamAsTempData(Axis2HttpRequest.java:332)
> > at
> >
> org.apache.synapse.transport.nhttp.Axis2HttpRequest.getRequest(Axis2HttpRequest.java:169)
> > at
> >
> org.apache.synapse.transport.nhttp.ClientHandler.processConnection(ClientHandler.java:194)
> > at
> >
> org.apache.synapse.transport.nhttp.ClientHandler.connected(ClientHandler.java:160)
> > at
> >
> org.apache.http.impl.nio.DefaultClientIOEventDispatch.connected(DefaultClientIOEventDispatch.java:134)
> > at
> >
> org.apache.http.impl.nio.reactor.BaseIOReactor.sessionCreated(BaseIOReactor.java:284)
> > at
> >
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processNewChannels(AbstractIOReactor.java:423)
> > at
> >
> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:286)
> > at
> >
> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
> > at
> >
> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:528)
> > at java.lang.Thread.run(Thread.java:619)
> >
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
>
--
Software Engineer, WSO2 Inc
http://wso2.org
supunk.blogspot.com
Re: I/O Dispatcher thread blocks on a read
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Sat, 2010-05-15 at 14:02 +0530, Supun Kamburugamuva wrote:
> Hi,
>
> One of the I/O dispatcher threads in Synapse is blocked
> in SharedInputBuffer.read method. Here is the thread dump. I would really
> appreciate if you can point me to the correct path for figuring out the root
> cause of this.
>
> This dispatcher is sending the message out from synapse. In this case it
> tries to read the incoming message read by the listeners to send it out.
>
> Thanks,
> Supun..
Supun,
SharedInputBuffer is shared by two (or more) threads: a data consumer
and a data producer. The data consumer thread is stuck waiting for more
data to arrive because the data producer thread does not fill the buffer
with more data.
Try to find out why there is no data in the shared buffer.
Oleg
>
> "http-Sender I/O dispatcher-2" prio=10 tid=0x08505800 nid=0x1718 in
> Object.wait() [0x613ac000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x8213c928> (a java.lang.Object)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.http.nio.util.SharedInputBuffer.waitForData(SharedInputBuffer.java:132)
> - locked <0x8213c928> (a java.lang.Object)
> at
> org.apache.http.nio.util.SharedInputBuffer.read(SharedInputBuffer.java:192)
> - locked <0x8213c928> (a java.lang.Object)
> at
> org.apache.http.nio.entity.ContentInputStream.read(ContentInputStream.java:64)
> at
> java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:221)
> at
> java.util.zip.InflaterInputStream.read(InflaterInputStream.java:141)
> at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:92)
> at java.io.FilterInputStream.read(FilterInputStream.java:90)
> at
> org.wso2.carbon.relay.BinaryRelayBuilder.readAllFromInputSteam(BinaryRelayBuilder.java:48)
> at
> org.wso2.carbon.relay.StreamingOnRequestDataSource.getInputStream(StreamingOnRequestDataSource.java:59)
> at javax.activation.DataHandler.writeTo(DataHandler.java:305)
> at
> org.wso2.carbon.relay.ExpandingMessageFormatter.findAndWrite2OutputStream(ExpandingMessageFormatter.java:170)
> at
> org.wso2.carbon.relay.ExpandingMessageFormatter.writeTo(ExpandingMessageFormatter.java:99)
> at
> org.apache.synapse.transport.nhttp.util.GzipMessageFormatterDecorator.writeTo(GzipMessageFormatterDecorator.java:66)
> at
> org.apache.synapse.transport.nhttp.Axis2HttpRequest.setStreamAsTempData(Axis2HttpRequest.java:332)
> at
> org.apache.synapse.transport.nhttp.Axis2HttpRequest.getRequest(Axis2HttpRequest.java:169)
> at
> org.apache.synapse.transport.nhttp.ClientHandler.processConnection(ClientHandler.java:194)
> at
> org.apache.synapse.transport.nhttp.ClientHandler.connected(ClientHandler.java:160)
> at
> org.apache.http.impl.nio.DefaultClientIOEventDispatch.connected(DefaultClientIOEventDispatch.java:134)
> at
> org.apache.http.impl.nio.reactor.BaseIOReactor.sessionCreated(BaseIOReactor.java:284)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processNewChannels(AbstractIOReactor.java:423)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:286)
> at
> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
> at
> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:528)
> at java.lang.Thread.run(Thread.java:619)
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org