You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-dev@axis.apache.org by "Hubert, Eric" <er...@jamba.net> on 2008/07/20 16:39:44 UTC
Understanding asynchrounous request processing in Apache Synapse/Axis2/Http Core NIO
Hi devs,
we are facing an issue in Apache Synapse where HttpClientWorkers are in
the process of writing a response back to the client using
HttpCoreNIOSender.sendAsyncResponse(), but are then waiting at
org.apache.http.nio.util.SharedOutputBuffer.flushContent() to get
notified by someone...
I'd like to take this issue as a start to gather more knowledge about
the internal working of Apache Synapse and http core nio in general.
Could someone please point me to some documentation, which describes,
how the request/response processing is working. Please also consider my
limited background in asynchronous event processing. I started with a
presentation by Doug Lea about the reactor design pattern. I guess this
is a good starting point, but obviously this is not enough.
When looking at a threaddump I see the following special thread types in
Apache Synapse:
- ClientWorker
- HttpServerWorker
- HttpCoreNIOListener
- HttpCoreNIOSender
- I/O dispatcher
- http-<sslport>Monitor
- http-<sslport>Processor
- ContainerBackgroundProcessor
- Axis2 Task
- Thread-XX (currently unnamed StatisticsReporterThread)
(maybe the code could be changed to assign a name to that thread)
Now I would like to understand the responsibility of each of these
Threads and how the interact. Who is responsible to wake up whom and so
on. So any information which helps me to understand this is greatly
appreciated.
My current understanding is as follows:
Incoming request from clients are processed from the
HttpCoreNIOListener, which is the in Axis2 configured non blocking
transport receiver for the http transport.
When starting this Listener, it starts an IOReactor
(org.apache.http.impl.nio.reactor.DefaultListeningIOReactor)-Thread with
name HttpCoreNIOListener. The IO reactor listens at the configured http
port and dispatches to the ServerHandler (NHttpServiceHandler). The
actual request process work is than delegated to a ServerWorker-Thread
from the pool. The ServerWorker delegates the request processing to
Axis2, where a SOAP message is created and the request is sent to the
endpoint. Right now the Axis2 engine is a kind of black box to me. I'm
not to sure about the whole handler chain between AxisEngine.receive()
and AxisEngine.send(). I hope this is not necessary for the general
understanding of the flow.
A TransportSender (HttpCoreNIOSender.sendAsyncRequest()) is used to send
the constructed soap message over the wire. The connection is
established using an inner IOReactor-Thread
(org.apache.http.impl.nio.reactor.DefaultListeningIOReactor).
The request is streamed using a nio ContentOutputBuffer.
Now the service at the specified endpoint processes the requests and
sends back a reply.
The response is processed by the ClientWorker who delegates the response
processing to Axis2 (again the blackbox of AxisEngine.receive,
AxisEngine.send) and then the response shall be send back to the client
over the MessageFormatter which again uses some NIO buffer. Here we have
the trouble that the code in
org.apache.http.nio.util.SharedOutputBuffer.flushContent() is waiting
for some notification. Who is responsible for that notification? The
HttpCoreNIOSender.reactor? I couldn't find any useful API-doc on this.
I'm also looking for some httpcore-nio-4.0-beta1-sources.jar from any
Maven Repo to attach to my synapse-IDE project.
Does anybody know something about possible causes for not getting such a
notification?
I would be very helpful for any help and pointers to more documentation
to understand more about the nio stuff.
Ah, and who creates the I/O Dispatcher? Are this the IOReactors? And are
the I/O Dispatcher again IOReactors?
Thanks for your help!
Eric
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
RE: Understanding asynchrounous request processing in ApacheSynapse/Axis2/Http Core NIO
Posted by "Hubert, Eric" <er...@jamba.net>.
Hi Oleg,
Thanks for your reply, which already helped me a little bit to improve my understanding.
> I am very sorry but we currently have almost nothing in terms of
> documentation for HttpCore. I am planning to start working on an
> HttpCore tutorial in August.
Oh yes, a tutorial would be very helpful. I'll be one of the first who is going to read it. :-)
> Asankha knows best about the Synapse NIO transport. I'll happily chip
> in information about HttpCore internal stuff
Thanks Oleg! That's why I included the http components dev list.
> > org.apache.http.nio.util.SharedOutputBuffer.flushContent() is waiting
> > for some notification. Who is responsible for that notification?
> It expects a notification from the I/O reactor it is ready to accept
> more output.
> > Does anybody know something about possible causes for not getting such a
> > notification?
> The most likely explanation the output notifications have been suspended
> by the protocol handler (Synapse it this case). Apparently Synapse
> thinks it is not meant to be producing any output at this point)
Hmm, this sounds interesting. So to translate this, the protocol handler implementation is: HttpCoreNIOSender or ClientHandler?
Asankha wrote me, that he is also already working on our issue and has some idea. So'll wait what he finds out.
I would be really happy if I were able to reproduce the issue at will. Unfortunately I'm not able to do this, so I have no test case. It just happens from time to time and then Synapse accepts new requests which get executed at the destination service, but the client never receives a reply, as the client worker threads are all waiting to sent out their message.
Regards,
Eric
Re: Understanding asynchrounous request processing in Apache Synapse/Axis2/Http
Core NIO
Posted by "Asankha C. Perera" <as...@wso2.com>.
Hi Eric
Oleg Kalnichevski wrote:
> On Sun, 2008-07-20 at 16:39 +0200, Hubert, Eric wrote:
>
>> Hi devs,
>>
>> we are facing an issue in Apache Synapse where HttpClientWorkers are in
>> the process of writing a response back to the client using
>> HttpCoreNIOSender.sendAsyncResponse(), but are then waiting at
>> org.apache.http.nio.util.SharedOutputBuffer.flushContent() to get
>> notified by someone...
>>
>> I'd like to take this issue as a start to gather more knowledge about
>> the internal working of Apache Synapse and http core nio in general.
>> Could someone please point me to some documentation, which describes,
>> how the request/response processing is working.
>>
>
> Eric,
>
> I am very sorry but we currently have almost nothing in terms of
> documentation for HttpCore. I am planning to start working on an
> HttpCore tutorial in August.
>
> Asankha knows best about the Synapse NIO transport. I'll happily chip
> in information about HttpCore internal stuff
>
> ...
>
I know that ideally I should have made more documentation available on
the NIO transport implementation, but due to other work I am involved
with, this has gone down in my TODO list.. I will try to come back to
this in the near future.. However, HttpCore/NIO is where the really cool
code exists, and Synapse/Axis2 is merely using it..
>> The response is processed by the ClientWorker who delegates the response
>> processing to Axis2 (again the blackbox of AxisEngine.receive,
>> AxisEngine.send) and then the response shall be send back to the client
>> over the MessageFormatter which again uses some NIO buffer. Here we have
>> the trouble that the code in
>> org.apache.http.nio.util.SharedOutputBuffer.flushContent() is waiting
>> for some notification. Who is responsible for that notification?
>>
>
> It expects a notification from the I/O reactor it is ready to accept
> more output.
This is because we recently enhanced the way we converted from NIO
channels to Java streams, using the shared buffers available in
HttpCore/NIO. In addition, we now throttle the connection, and suspend
IO on the socket when the corresponding buffer is filled up - and
prevents out of memory errors, and allows the transport to operate with
constant memory.
>> The
>> HttpCoreNIOSender.reactor? I couldn't find any useful API-doc on this.
>> I'm also looking for some httpcore-nio-4.0-beta1-sources.jar from any
>> Maven Repo to attach to my synapse-IDE project.
>>
>> Does anybody know something about possible causes for not getting such a
>> notification?
>>
I was able to reproduce a 'simulated' case of what you experienced, and
this was caused by a connection close while the worker thread was
writing data and waiting on the flushContent() for a large response.
Once you confirm the temporary fix I provided you, I will check it into
the trunk with a JIRA so that it goes into the next release
asankha
--
Asankha C. Perera
WSO2 - http://wso2.org
http://esbmagic.blogspot.com
Re: Understanding asynchrounous request processing in Apache Synapse/Axis2/Http
Core NIO
Posted by "Asankha C. Perera" <as...@wso2.com>.
Hi Eric
Oleg Kalnichevski wrote:
> On Sun, 2008-07-20 at 16:39 +0200, Hubert, Eric wrote:
>
>> Hi devs,
>>
>> we are facing an issue in Apache Synapse where HttpClientWorkers are in
>> the process of writing a response back to the client using
>> HttpCoreNIOSender.sendAsyncResponse(), but are then waiting at
>> org.apache.http.nio.util.SharedOutputBuffer.flushContent() to get
>> notified by someone...
>>
>> I'd like to take this issue as a start to gather more knowledge about
>> the internal working of Apache Synapse and http core nio in general.
>> Could someone please point me to some documentation, which describes,
>> how the request/response processing is working.
>>
>
> Eric,
>
> I am very sorry but we currently have almost nothing in terms of
> documentation for HttpCore. I am planning to start working on an
> HttpCore tutorial in August.
>
> Asankha knows best about the Synapse NIO transport. I'll happily chip
> in information about HttpCore internal stuff
>
> ...
>
I know that ideally I should have made more documentation available on
the NIO transport implementation, but due to other work I am involved
with, this has gone down in my TODO list.. I will try to come back to
this in the near future.. However, HttpCore/NIO is where the really cool
code exists, and Synapse/Axis2 is merely using it..
>> The response is processed by the ClientWorker who delegates the response
>> processing to Axis2 (again the blackbox of AxisEngine.receive,
>> AxisEngine.send) and then the response shall be send back to the client
>> over the MessageFormatter which again uses some NIO buffer. Here we have
>> the trouble that the code in
>> org.apache.http.nio.util.SharedOutputBuffer.flushContent() is waiting
>> for some notification. Who is responsible for that notification?
>>
>
> It expects a notification from the I/O reactor it is ready to accept
> more output.
This is because we recently enhanced the way we converted from NIO
channels to Java streams, using the shared buffers available in
HttpCore/NIO. In addition, we now throttle the connection, and suspend
IO on the socket when the corresponding buffer is filled up - and
prevents out of memory errors, and allows the transport to operate with
constant memory.
>> The
>> HttpCoreNIOSender.reactor? I couldn't find any useful API-doc on this.
>> I'm also looking for some httpcore-nio-4.0-beta1-sources.jar from any
>> Maven Repo to attach to my synapse-IDE project.
>>
>> Does anybody know something about possible causes for not getting such a
>> notification?
>>
I was able to reproduce a 'simulated' case of what you experienced, and
this was caused by a connection close while the worker thread was
writing data and waiting on the flushContent() for a large response.
Once you confirm the temporary fix I provided you, I will check it into
the trunk with a JIRA so that it goes into the next release
asankha
--
Asankha C. Perera
WSO2 - http://wso2.org
http://esbmagic.blogspot.com
RE: Understanding asynchrounous request processing in ApacheSynapse/Axis2/Http Core NIO
Posted by "Hubert, Eric" <er...@jamba.net>.
Hi Oleg,
Thanks for your reply, which already helped me a little bit to improve my understanding.
> I am very sorry but we currently have almost nothing in terms of
> documentation for HttpCore. I am planning to start working on an
> HttpCore tutorial in August.
Oh yes, a tutorial would be very helpful. I'll be one of the first who is going to read it. :-)
> Asankha knows best about the Synapse NIO transport. I'll happily chip
> in information about HttpCore internal stuff
Thanks Oleg! That's why I included the http components dev list.
> > org.apache.http.nio.util.SharedOutputBuffer.flushContent() is waiting
> > for some notification. Who is responsible for that notification?
> It expects a notification from the I/O reactor it is ready to accept
> more output.
> > Does anybody know something about possible causes for not getting such a
> > notification?
> The most likely explanation the output notifications have been suspended
> by the protocol handler (Synapse it this case). Apparently Synapse
> thinks it is not meant to be producing any output at this point)
Hmm, this sounds interesting. So to translate this, the protocol handler implementation is: HttpCoreNIOSender or ClientHandler?
Asankha wrote me, that he is also already working on our issue and has some idea. So'll wait what he finds out.
I would be really happy if I were able to reproduce the issue at will. Unfortunately I'm not able to do this, so I have no test case. It just happens from time to time and then Synapse accepts new requests which get executed at the destination service, but the client never receives a reply, as the client worker threads are all waiting to sent out their message.
Regards,
Eric
Re: Understanding asynchrounous request processing in Apache Synapse/Axis2/Http
Core NIO
Posted by "Asankha C. Perera" <as...@wso2.com>.
Hi Eric
Oleg Kalnichevski wrote:
> On Sun, 2008-07-20 at 16:39 +0200, Hubert, Eric wrote:
>
>> Hi devs,
>>
>> we are facing an issue in Apache Synapse where HttpClientWorkers are in
>> the process of writing a response back to the client using
>> HttpCoreNIOSender.sendAsyncResponse(), but are then waiting at
>> org.apache.http.nio.util.SharedOutputBuffer.flushContent() to get
>> notified by someone...
>>
>> I'd like to take this issue as a start to gather more knowledge about
>> the internal working of Apache Synapse and http core nio in general.
>> Could someone please point me to some documentation, which describes,
>> how the request/response processing is working.
>>
>
> Eric,
>
> I am very sorry but we currently have almost nothing in terms of
> documentation for HttpCore. I am planning to start working on an
> HttpCore tutorial in August.
>
> Asankha knows best about the Synapse NIO transport. I'll happily chip
> in information about HttpCore internal stuff
>
> ...
>
I know that ideally I should have made more documentation available on
the NIO transport implementation, but due to other work I am involved
with, this has gone down in my TODO list.. I will try to come back to
this in the near future.. However, HttpCore/NIO is where the really cool
code exists, and Synapse/Axis2 is merely using it..
>> The response is processed by the ClientWorker who delegates the response
>> processing to Axis2 (again the blackbox of AxisEngine.receive,
>> AxisEngine.send) and then the response shall be send back to the client
>> over the MessageFormatter which again uses some NIO buffer. Here we have
>> the trouble that the code in
>> org.apache.http.nio.util.SharedOutputBuffer.flushContent() is waiting
>> for some notification. Who is responsible for that notification?
>>
>
> It expects a notification from the I/O reactor it is ready to accept
> more output.
This is because we recently enhanced the way we converted from NIO
channels to Java streams, using the shared buffers available in
HttpCore/NIO. In addition, we now throttle the connection, and suspend
IO on the socket when the corresponding buffer is filled up - and
prevents out of memory errors, and allows the transport to operate with
constant memory.
>> The
>> HttpCoreNIOSender.reactor? I couldn't find any useful API-doc on this.
>> I'm also looking for some httpcore-nio-4.0-beta1-sources.jar from any
>> Maven Repo to attach to my synapse-IDE project.
>>
>> Does anybody know something about possible causes for not getting such a
>> notification?
>>
I was able to reproduce a 'simulated' case of what you experienced, and
this was caused by a connection close while the worker thread was
writing data and waiting on the flushContent() for a large response.
Once you confirm the temporary fix I provided you, I will check it into
the trunk with a JIRA so that it goes into the next release
asankha
--
Asankha C. Perera
WSO2 - http://wso2.org
http://esbmagic.blogspot.com
RE: Understanding asynchrounous request processing in ApacheSynapse/Axis2/Http Core NIO
Posted by "Hubert, Eric" <er...@jamba.net>.
Hi Oleg,
Thanks for your reply, which already helped me a little bit to improve my understanding.
> I am very sorry but we currently have almost nothing in terms of
> documentation for HttpCore. I am planning to start working on an
> HttpCore tutorial in August.
Oh yes, a tutorial would be very helpful. I'll be one of the first who is going to read it. :-)
> Asankha knows best about the Synapse NIO transport. I'll happily chip
> in information about HttpCore internal stuff
Thanks Oleg! That's why I included the http components dev list.
> > org.apache.http.nio.util.SharedOutputBuffer.flushContent() is waiting
> > for some notification. Who is responsible for that notification?
> It expects a notification from the I/O reactor it is ready to accept
> more output.
> > Does anybody know something about possible causes for not getting such a
> > notification?
> The most likely explanation the output notifications have been suspended
> by the protocol handler (Synapse it this case). Apparently Synapse
> thinks it is not meant to be producing any output at this point)
Hmm, this sounds interesting. So to translate this, the protocol handler implementation is: HttpCoreNIOSender or ClientHandler?
Asankha wrote me, that he is also already working on our issue and has some idea. So'll wait what he finds out.
I would be really happy if I were able to reproduce the issue at will. Unfortunately I'm not able to do this, so I have no test case. It just happens from time to time and then Synapse accepts new requests which get executed at the destination service, but the client never receives a reply, as the client worker threads are all waiting to sent out their message.
Regards,
Eric
Re: Understanding asynchrounous request processing in Apache
Synapse/Axis2/Http Core NIO
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Sun, 2008-07-20 at 16:39 +0200, Hubert, Eric wrote:
> Hi devs,
>
> we are facing an issue in Apache Synapse where HttpClientWorkers are in
> the process of writing a response back to the client using
> HttpCoreNIOSender.sendAsyncResponse(), but are then waiting at
> org.apache.http.nio.util.SharedOutputBuffer.flushContent() to get
> notified by someone...
>
> I'd like to take this issue as a start to gather more knowledge about
> the internal working of Apache Synapse and http core nio in general.
> Could someone please point me to some documentation, which describes,
> how the request/response processing is working.
Eric,
I am very sorry but we currently have almost nothing in terms of
documentation for HttpCore. I am planning to start working on an
HttpCore tutorial in August.
Asankha knows best about the Synapse NIO transport. I'll happily chip
in information about HttpCore internal stuff
...
> The response is processed by the ClientWorker who delegates the response
> processing to Axis2 (again the blackbox of AxisEngine.receive,
> AxisEngine.send) and then the response shall be send back to the client
> over the MessageFormatter which again uses some NIO buffer. Here we have
> the trouble that the code in
> org.apache.http.nio.util.SharedOutputBuffer.flushContent() is waiting
> for some notification. Who is responsible for that notification?
It expects a notification from the I/O reactor it is ready to accept
more output.
> The
> HttpCoreNIOSender.reactor? I couldn't find any useful API-doc on this.
> I'm also looking for some httpcore-nio-4.0-beta1-sources.jar from any
> Maven Repo to attach to my synapse-IDE project.
>
> Does anybody know something about possible causes for not getting such a
> notification?
>
The most likely explanation the output notifications have been suspended
by the protocol handler (Synapse it this case). Apparently Synapse
thinks it is not meant to be producing any output at this point)
> I would be very helpful for any help and pointers to more documentation
> to understand more about the nio stuff.
>
> Ah, and who creates the I/O Dispatcher?
Listening or connection I/O reactors
> Are this the IOReactors?
Yes, it is
> And are
> the I/O Dispatcher again IOReactors?
>
Yes, they are
Can you reproduce the problem with a test case?
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: Understanding asynchrounous request processing in Apache
Synapse/Axis2/Http Core NIO
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Sun, 2008-07-20 at 16:39 +0200, Hubert, Eric wrote:
> Hi devs,
>
> we are facing an issue in Apache Synapse where HttpClientWorkers are in
> the process of writing a response back to the client using
> HttpCoreNIOSender.sendAsyncResponse(), but are then waiting at
> org.apache.http.nio.util.SharedOutputBuffer.flushContent() to get
> notified by someone...
>
> I'd like to take this issue as a start to gather more knowledge about
> the internal working of Apache Synapse and http core nio in general.
> Could someone please point me to some documentation, which describes,
> how the request/response processing is working.
Eric,
I am very sorry but we currently have almost nothing in terms of
documentation for HttpCore. I am planning to start working on an
HttpCore tutorial in August.
Asankha knows best about the Synapse NIO transport. I'll happily chip
in information about HttpCore internal stuff
...
> The response is processed by the ClientWorker who delegates the response
> processing to Axis2 (again the blackbox of AxisEngine.receive,
> AxisEngine.send) and then the response shall be send back to the client
> over the MessageFormatter which again uses some NIO buffer. Here we have
> the trouble that the code in
> org.apache.http.nio.util.SharedOutputBuffer.flushContent() is waiting
> for some notification. Who is responsible for that notification?
It expects a notification from the I/O reactor it is ready to accept
more output.
> The
> HttpCoreNIOSender.reactor? I couldn't find any useful API-doc on this.
> I'm also looking for some httpcore-nio-4.0-beta1-sources.jar from any
> Maven Repo to attach to my synapse-IDE project.
>
> Does anybody know something about possible causes for not getting such a
> notification?
>
The most likely explanation the output notifications have been suspended
by the protocol handler (Synapse it this case). Apparently Synapse
thinks it is not meant to be producing any output at this point)
> I would be very helpful for any help and pointers to more documentation
> to understand more about the nio stuff.
>
> Ah, and who creates the I/O Dispatcher?
Listening or connection I/O reactors
> Are this the IOReactors?
Yes, it is
> And are
> the I/O Dispatcher again IOReactors?
>
Yes, they are
Can you reproduce the problem with a test case?
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org