You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hc.apache.org by Oleg Kalnichevski <ol...@apache.org> on 2008/07/20 17:49:36 UTC
Re: Understanding asynchrounous request processing in Apache
Synapse/Axis2/Http Core NIO
On Sun, 2008-07-20 at 16:39 +0200, Hubert, Eric wrote:
> Hi devs,
>
> we are facing an issue in Apache Synapse where HttpClientWorkers are in
> the process of writing a response back to the client using
> HttpCoreNIOSender.sendAsyncResponse(), but are then waiting at
> org.apache.http.nio.util.SharedOutputBuffer.flushContent() to get
> notified by someone...
>
> I'd like to take this issue as a start to gather more knowledge about
> the internal working of Apache Synapse and http core nio in general.
> Could someone please point me to some documentation, which describes,
> how the request/response processing is working.
Eric,
I am very sorry but we currently have almost nothing in terms of
documentation for HttpCore. I am planning to start working on an
HttpCore tutorial in August.
Asankha knows best about the Synapse NIO transport. I'll happily chip
in information about HttpCore internal stuff
...
> The response is processed by the ClientWorker who delegates the response
> processing to Axis2 (again the blackbox of AxisEngine.receive,
> AxisEngine.send) and then the response shall be send back to the client
> over the MessageFormatter which again uses some NIO buffer. Here we have
> the trouble that the code in
> org.apache.http.nio.util.SharedOutputBuffer.flushContent() is waiting
> for some notification. Who is responsible for that notification?
It expects a notification from the I/O reactor it is ready to accept
more output.
> The
> HttpCoreNIOSender.reactor? I couldn't find any useful API-doc on this.
> I'm also looking for some httpcore-nio-4.0-beta1-sources.jar from any
> Maven Repo to attach to my synapse-IDE project.
>
> Does anybody know something about possible causes for not getting such a
> notification?
>
The most likely explanation the output notifications have been suspended
by the protocol handler (Synapse it this case). Apparently Synapse
thinks it is not meant to be producing any output at this point)
> I would be very helpful for any help and pointers to more documentation
> to understand more about the nio stuff.
>
> Ah, and who creates the I/O Dispatcher?
Listening or connection I/O reactors
> Are this the IOReactors?
Yes, it is
> And are
> the I/O Dispatcher again IOReactors?
>
Yes, they are
Can you reproduce the problem with a test case?
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
RE: Understanding asynchrounous request processing in ApacheSynapse/Axis2/Http Core NIO
Posted by "Hubert, Eric" <er...@jamba.net>.
Hi Oleg,
Thanks for your reply, which already helped me a little bit to improve my understanding.
> I am very sorry but we currently have almost nothing in terms of
> documentation for HttpCore. I am planning to start working on an
> HttpCore tutorial in August.
Oh yes, a tutorial would be very helpful. I'll be one of the first who is going to read it. :-)
> Asankha knows best about the Synapse NIO transport. I'll happily chip
> in information about HttpCore internal stuff
Thanks Oleg! That's why I included the http components dev list.
> > org.apache.http.nio.util.SharedOutputBuffer.flushContent() is waiting
> > for some notification. Who is responsible for that notification?
> It expects a notification from the I/O reactor it is ready to accept
> more output.
> > Does anybody know something about possible causes for not getting such a
> > notification?
> The most likely explanation the output notifications have been suspended
> by the protocol handler (Synapse it this case). Apparently Synapse
> thinks it is not meant to be producing any output at this point)
Hmm, this sounds interesting. So to translate this, the protocol handler implementation is: HttpCoreNIOSender or ClientHandler?
Asankha wrote me, that he is also already working on our issue and has some idea. So'll wait what he finds out.
I would be really happy if I were able to reproduce the issue at will. Unfortunately I'm not able to do this, so I have no test case. It just happens from time to time and then Synapse accepts new requests which get executed at the destination service, but the client never receives a reply, as the client worker threads are all waiting to sent out their message.
Regards,
Eric
Re: Understanding asynchrounous request processing in Apache Synapse/Axis2/Http
Core NIO
Posted by "Asankha C. Perera" <as...@wso2.com>.
Hi Eric
Oleg Kalnichevski wrote:
> On Sun, 2008-07-20 at 16:39 +0200, Hubert, Eric wrote:
>
>> Hi devs,
>>
>> we are facing an issue in Apache Synapse where HttpClientWorkers are in
>> the process of writing a response back to the client using
>> HttpCoreNIOSender.sendAsyncResponse(), but are then waiting at
>> org.apache.http.nio.util.SharedOutputBuffer.flushContent() to get
>> notified by someone...
>>
>> I'd like to take this issue as a start to gather more knowledge about
>> the internal working of Apache Synapse and http core nio in general.
>> Could someone please point me to some documentation, which describes,
>> how the request/response processing is working.
>>
>
> Eric,
>
> I am very sorry but we currently have almost nothing in terms of
> documentation for HttpCore. I am planning to start working on an
> HttpCore tutorial in August.
>
> Asankha knows best about the Synapse NIO transport. I'll happily chip
> in information about HttpCore internal stuff
>
> ...
>
I know that ideally I should have made more documentation available on
the NIO transport implementation, but due to other work I am involved
with, this has gone down in my TODO list.. I will try to come back to
this in the near future.. However, HttpCore/NIO is where the really cool
code exists, and Synapse/Axis2 is merely using it..
>> The response is processed by the ClientWorker who delegates the response
>> processing to Axis2 (again the blackbox of AxisEngine.receive,
>> AxisEngine.send) and then the response shall be send back to the client
>> over the MessageFormatter which again uses some NIO buffer. Here we have
>> the trouble that the code in
>> org.apache.http.nio.util.SharedOutputBuffer.flushContent() is waiting
>> for some notification. Who is responsible for that notification?
>>
>
> It expects a notification from the I/O reactor it is ready to accept
> more output.
This is because we recently enhanced the way we converted from NIO
channels to Java streams, using the shared buffers available in
HttpCore/NIO. In addition, we now throttle the connection, and suspend
IO on the socket when the corresponding buffer is filled up - and
prevents out of memory errors, and allows the transport to operate with
constant memory.
>> The
>> HttpCoreNIOSender.reactor? I couldn't find any useful API-doc on this.
>> I'm also looking for some httpcore-nio-4.0-beta1-sources.jar from any
>> Maven Repo to attach to my synapse-IDE project.
>>
>> Does anybody know something about possible causes for not getting such a
>> notification?
>>
I was able to reproduce a 'simulated' case of what you experienced, and
this was caused by a connection close while the worker thread was
writing data and waiting on the flushContent() for a large response.
Once you confirm the temporary fix I provided you, I will check it into
the trunk with a JIRA so that it goes into the next release
asankha
--
Asankha C. Perera
WSO2 - http://wso2.org
http://esbmagic.blogspot.com
Re: Understanding asynchrounous request processing in Apache Synapse/Axis2/Http
Core NIO
Posted by "Asankha C. Perera" <as...@wso2.com>.
Hi Eric
Oleg Kalnichevski wrote:
> On Sun, 2008-07-20 at 16:39 +0200, Hubert, Eric wrote:
>
>> Hi devs,
>>
>> we are facing an issue in Apache Synapse where HttpClientWorkers are in
>> the process of writing a response back to the client using
>> HttpCoreNIOSender.sendAsyncResponse(), but are then waiting at
>> org.apache.http.nio.util.SharedOutputBuffer.flushContent() to get
>> notified by someone...
>>
>> I'd like to take this issue as a start to gather more knowledge about
>> the internal working of Apache Synapse and http core nio in general.
>> Could someone please point me to some documentation, which describes,
>> how the request/response processing is working.
>>
>
> Eric,
>
> I am very sorry but we currently have almost nothing in terms of
> documentation for HttpCore. I am planning to start working on an
> HttpCore tutorial in August.
>
> Asankha knows best about the Synapse NIO transport. I'll happily chip
> in information about HttpCore internal stuff
>
> ...
>
I know that ideally I should have made more documentation available on
the NIO transport implementation, but due to other work I am involved
with, this has gone down in my TODO list.. I will try to come back to
this in the near future.. However, HttpCore/NIO is where the really cool
code exists, and Synapse/Axis2 is merely using it..
>> The response is processed by the ClientWorker who delegates the response
>> processing to Axis2 (again the blackbox of AxisEngine.receive,
>> AxisEngine.send) and then the response shall be send back to the client
>> over the MessageFormatter which again uses some NIO buffer. Here we have
>> the trouble that the code in
>> org.apache.http.nio.util.SharedOutputBuffer.flushContent() is waiting
>> for some notification. Who is responsible for that notification?
>>
>
> It expects a notification from the I/O reactor it is ready to accept
> more output.
This is because we recently enhanced the way we converted from NIO
channels to Java streams, using the shared buffers available in
HttpCore/NIO. In addition, we now throttle the connection, and suspend
IO on the socket when the corresponding buffer is filled up - and
prevents out of memory errors, and allows the transport to operate with
constant memory.
>> The
>> HttpCoreNIOSender.reactor? I couldn't find any useful API-doc on this.
>> I'm also looking for some httpcore-nio-4.0-beta1-sources.jar from any
>> Maven Repo to attach to my synapse-IDE project.
>>
>> Does anybody know something about possible causes for not getting such a
>> notification?
>>
I was able to reproduce a 'simulated' case of what you experienced, and
this was caused by a connection close while the worker thread was
writing data and waiting on the flushContent() for a large response.
Once you confirm the temporary fix I provided you, I will check it into
the trunk with a JIRA so that it goes into the next release
asankha
--
Asankha C. Perera
WSO2 - http://wso2.org
http://esbmagic.blogspot.com
RE: Understanding asynchrounous request processing in ApacheSynapse/Axis2/Http Core NIO
Posted by "Hubert, Eric" <er...@jamba.net>.
Hi Oleg,
Thanks for your reply, which already helped me a little bit to improve my understanding.
> I am very sorry but we currently have almost nothing in terms of
> documentation for HttpCore. I am planning to start working on an
> HttpCore tutorial in August.
Oh yes, a tutorial would be very helpful. I'll be one of the first who is going to read it. :-)
> Asankha knows best about the Synapse NIO transport. I'll happily chip
> in information about HttpCore internal stuff
Thanks Oleg! That's why I included the http components dev list.
> > org.apache.http.nio.util.SharedOutputBuffer.flushContent() is waiting
> > for some notification. Who is responsible for that notification?
> It expects a notification from the I/O reactor it is ready to accept
> more output.
> > Does anybody know something about possible causes for not getting such a
> > notification?
> The most likely explanation the output notifications have been suspended
> by the protocol handler (Synapse it this case). Apparently Synapse
> thinks it is not meant to be producing any output at this point)
Hmm, this sounds interesting. So to translate this, the protocol handler implementation is: HttpCoreNIOSender or ClientHandler?
Asankha wrote me, that he is also already working on our issue and has some idea. So'll wait what he finds out.
I would be really happy if I were able to reproduce the issue at will. Unfortunately I'm not able to do this, so I have no test case. It just happens from time to time and then Synapse accepts new requests which get executed at the destination service, but the client never receives a reply, as the client worker threads are all waiting to sent out their message.
Regards,
Eric
Re: Understanding asynchrounous request processing in Apache Synapse/Axis2/Http
Core NIO
Posted by "Asankha C. Perera" <as...@wso2.com>.
Hi Eric
Oleg Kalnichevski wrote:
> On Sun, 2008-07-20 at 16:39 +0200, Hubert, Eric wrote:
>
>> Hi devs,
>>
>> we are facing an issue in Apache Synapse where HttpClientWorkers are in
>> the process of writing a response back to the client using
>> HttpCoreNIOSender.sendAsyncResponse(), but are then waiting at
>> org.apache.http.nio.util.SharedOutputBuffer.flushContent() to get
>> notified by someone...
>>
>> I'd like to take this issue as a start to gather more knowledge about
>> the internal working of Apache Synapse and http core nio in general.
>> Could someone please point me to some documentation, which describes,
>> how the request/response processing is working.
>>
>
> Eric,
>
> I am very sorry but we currently have almost nothing in terms of
> documentation for HttpCore. I am planning to start working on an
> HttpCore tutorial in August.
>
> Asankha knows best about the Synapse NIO transport. I'll happily chip
> in information about HttpCore internal stuff
>
> ...
>
I know that ideally I should have made more documentation available on
the NIO transport implementation, but due to other work I am involved
with, this has gone down in my TODO list.. I will try to come back to
this in the near future.. However, HttpCore/NIO is where the really cool
code exists, and Synapse/Axis2 is merely using it..
>> The response is processed by the ClientWorker who delegates the response
>> processing to Axis2 (again the blackbox of AxisEngine.receive,
>> AxisEngine.send) and then the response shall be send back to the client
>> over the MessageFormatter which again uses some NIO buffer. Here we have
>> the trouble that the code in
>> org.apache.http.nio.util.SharedOutputBuffer.flushContent() is waiting
>> for some notification. Who is responsible for that notification?
>>
>
> It expects a notification from the I/O reactor it is ready to accept
> more output.
This is because we recently enhanced the way we converted from NIO
channels to Java streams, using the shared buffers available in
HttpCore/NIO. In addition, we now throttle the connection, and suspend
IO on the socket when the corresponding buffer is filled up - and
prevents out of memory errors, and allows the transport to operate with
constant memory.
>> The
>> HttpCoreNIOSender.reactor? I couldn't find any useful API-doc on this.
>> I'm also looking for some httpcore-nio-4.0-beta1-sources.jar from any
>> Maven Repo to attach to my synapse-IDE project.
>>
>> Does anybody know something about possible causes for not getting such a
>> notification?
>>
I was able to reproduce a 'simulated' case of what you experienced, and
this was caused by a connection close while the worker thread was
writing data and waiting on the flushContent() for a large response.
Once you confirm the temporary fix I provided you, I will check it into
the trunk with a JIRA so that it goes into the next release
asankha
--
Asankha C. Perera
WSO2 - http://wso2.org
http://esbmagic.blogspot.com
RE: Understanding asynchrounous request processing in ApacheSynapse/Axis2/Http Core NIO
Posted by "Hubert, Eric" <er...@jamba.net>.
Hi Oleg,
Thanks for your reply, which already helped me a little bit to improve my understanding.
> I am very sorry but we currently have almost nothing in terms of
> documentation for HttpCore. I am planning to start working on an
> HttpCore tutorial in August.
Oh yes, a tutorial would be very helpful. I'll be one of the first who is going to read it. :-)
> Asankha knows best about the Synapse NIO transport. I'll happily chip
> in information about HttpCore internal stuff
Thanks Oleg! That's why I included the http components dev list.
> > org.apache.http.nio.util.SharedOutputBuffer.flushContent() is waiting
> > for some notification. Who is responsible for that notification?
> It expects a notification from the I/O reactor it is ready to accept
> more output.
> > Does anybody know something about possible causes for not getting such a
> > notification?
> The most likely explanation the output notifications have been suspended
> by the protocol handler (Synapse it this case). Apparently Synapse
> thinks it is not meant to be producing any output at this point)
Hmm, this sounds interesting. So to translate this, the protocol handler implementation is: HttpCoreNIOSender or ClientHandler?
Asankha wrote me, that he is also already working on our issue and has some idea. So'll wait what he finds out.
I would be really happy if I were able to reproduce the issue at will. Unfortunately I'm not able to do this, so I have no test case. It just happens from time to time and then Synapse accepts new requests which get executed at the destination service, but the client never receives a reply, as the client worker threads are all waiting to sent out their message.
Regards,
Eric