You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-dev@axis.apache.org by Himakar Chennapragada <hi...@us.ibm.com> on 2011/08/17 23:43:02 UTC
how to avoid CLOSE_WAITs without the use of the REUSE_HTTP_CLIENT option?
Is there a way to prevent/avoid or minimize the CLOSE_WAITs from happening
when we DO NOT use the REUSE_HTTP_CLIENT option on the ServiceClient
interface.
We have multiple threads calling out to same or different web services. We
use a new ServiceClient for each invocation to an external web service
(same or different service) without the REUSE_HTTP_CLIENT option and that
produces multiple CLOSE_WAITs that seem to grow over time. We do
understand that using the REUSE_HTTP_CLIENT option helps in this area but
we want to explore other ways without using this option if any.
Best Regards,
-Himakar
Re: how to avoid CLOSE_WAITs without the use of the REUSE_HTTP_CLIENT option?
Posted by Afkham Azeez <af...@gmail.com>.
Nice blog post. Very informative. Thanks for sharing.
Azeez
On Thu, Aug 18, 2011 at 10:32 AM, shivendra tripathi
<sh...@gmail.com>wrote:
> Hope below link would be helpful
>
>
> http://shivendra-tripathi.blogspot.com/2011/05/http-transport-in-axis2-how-to-fix.html
>
>
> On Thu, Aug 18, 2011 at 3:13 AM, Himakar Chennapragada <himakar@us.ibm.com
> > wrote:
>
>>
>> Is there a way to prevent/avoid or minimize the CLOSE_WAITs from happening
>> when we DO NOT use the REUSE_HTTP_CLIENT option on the ServiceClient
>> interface.
>>
>> We have multiple threads calling out to *same or different *web services.
>> We use a new ServiceClient for *each invocation* to an external web
>> service (same or different service) without the REUSE_HTTP_CLIENT option and
>> that produces multiple CLOSE_WAITs that seem to grow over time. We do
>> understand that using the REUSE_HTTP_CLIENT option helps in this area but we
>> want to explore other ways without using this option if any.
>>
>> Best Regards,
>> -Himakar
>>
>
>
--
*Afkham Azeez*
Director of Architecture; WSO2, Inc.; http://wso2.com,
*Member; Apache Software Foundation;
**http://www.apache.org/*<http://www.apache.org/>
*
*
*email: **azeez@wso2.com* <az...@wso2.com>* cell: +94 77 3320919
blog: **http://blog.afkham.org* <http://blog.afkham.org>*
twitter: **http://twitter.com/afkham_azeez*<http://twitter.com/afkham_azeez>
*
linked-in: **http://lk.linkedin.com/in/afkhamazeez*
*
*
*Lean . Enterprise . Middleware*
*
*
Re: how to avoid CLOSE_WAITs without the use of the REUSE_HTTP_CLIENT option?
Posted by Andreas Veithen <an...@gmail.com>.
On Fri, Aug 19, 2011 at 11:44, Amila Suriarachchi
<am...@gmail.com> wrote:
>
>
> On Fri, Aug 19, 2011 at 12:43 AM, Andreas Veithen
> <an...@gmail.com> wrote:
>>
>> Although the practical conclusions of that blog post are correct, the
>> explanation is nevertheless based on a misunderstanding of what the
>> actual issue is. Quoting from the article:
>>
>> "idea behind recent change in axis2 is to use connection pool facility
>> of MTHCM to avoid new http connection every time you call webservice"
>>
>> "So if we are using axis2 1.5.1 onwards we need to release the
>> connection to the pool since we can not have more than one MTHCM"
>>
>> This is not entirely wrong, but it is still incomplete. The problem in
>> Axis2 versions before 1.5.1 was that in addition to creating a new
>> connection for every request, the HTTP transport was unable to ensure
>> that these connections are properly closed. This caused the CLOSE_WAIT
>> problem mentioned by Himakar.
>>
>> Another consequence of this bug was that for these versions of the
>> HTTP transport, it didn't really matter whether the application code
>> invoked ServiceClient#cleanupTransport() or not. There was a
>> connection leak anyway. In many cases this actually didn't cause an
>> incident because the connections would eventually be closed by the
>> garbage collector. The consequence of this is that there is lot of
>> application code written for Axis2 versions before 1.5.1 that fails to
>> call cleanupTransport() in the appropriate places.
>>
>> The connection leak was finally fixed in Axis2 1.5.1 (kudos to Glen)
>> and the HTTP transport now manages the connections properly. However,
>> the cleanup code relies on the assumption that the application code
>> uses the ServiceClient API properly, and in particular calls
>> cleanupTransport where necessary. If the code fails to do that, then
>> the connection pool used by the HTTP transport will quickly be
>> starved, resulting in a "Timeout waiting for connection" error.
>>
>> To summarize: application code that triggers a "Timeout waiting for
>> connection" with Axis2 >= 1.5.1 was already incorrect in previous
>> versions of Axis2, but the symptoms caused by this incorrect code are
>> different.
>>
>> With this knowledge, it is possible to give an answer to Himakar's
>> original question:
>>
>> a. In Axis2 >= 1.5.1, the CLOSE_WAIT issue never occurs and it is in
>> general not necessary to configure a custom connection manager or
>> HttpClient.
>
> Now the error is Timeout since default connection manager is only allow 2
> connections. I think this can be improved by initiating the connection
> manager at the transport initialization time and having parameters to define
> those at the axis2.xml transport sender level.
>
> The bottom line is still it does not fix the connection related problem out
> of the box. Users have to use the service Client API properly by calling the
> cleanUp transport.
>
>>
>> b. In Axis2 <= 1.5, Keep-Alive connections will always be leaked
>> unless one configures a custom connection manager.
>
> There is no connection leak in the earlier version. But what happen is it
> takes 30 seconds to close the connection.
That is only true if the connection is properly released, in which
case you won't get a "Timeout waiting for connection" error with Axis2
>= 1.5.1. As I said earlier, code that in Axis2 >= 1.5.1 produces a
"Timeout waiting for connection" error will leak connections in Axis2
< 1.5.1, and these connections are not closed after 30 seconds, but
when the garbage collector destroys the connection and the associated
connection pool. If the peer closes the connection before that, then
it will be in state CLOSE_WAIT.
> if the load is high this cause a
> CLOSE_WAIT sate connection accumulation.
If the connection is closed after 30 seconds, then this will primarily
result in connections in state TIME_WAIT, as described in AXIS2-2931.
Connections in state CLOSE_WAIT will only appear if the peer decides
to close them before the 30s timeout.
The TCP state diagram helps to understand this:
http://tangentsoft.net/wskfaq/articles/bitmaps/state-diagram-small.gif
> thanks,
> Amila.
>
>>
>> Another option may
>> be to disable Keep-Alive or switch to HTTP/1.0. In that case it is
>> likely that the call to cleanupTransport() will actually close the
>> connection (to be tested!).
>>
>> Andreas
>>
>> On Thu, Aug 18, 2011 at 07:02, shivendra tripathi <sh...@gmail.com>
>> wrote:
>> > Hope below link would be helpful
>> >
>> > http://shivendra-tripathi.blogspot.com/2011/05/http-transport-in-axis2-how-to-fix.html
>> >
>> > On Thu, Aug 18, 2011 at 3:13 AM, Himakar Chennapragada
>> > <hi...@us.ibm.com>
>> > wrote:
>> >>
>> >> Is there a way to prevent/avoid or minimize the CLOSE_WAITs from
>> >> happening
>> >> when we DO NOT use the REUSE_HTTP_CLIENT option on the ServiceClient
>> >> interface.
>> >>
>> >> We have multiple threads calling out to same or different web services.
>> >> We use a new ServiceClient for each invocation to an external web
>> >> service
>> >> (same or different service) without the REUSE_HTTP_CLIENT option and
>> >> that
>> >> produces multiple CLOSE_WAITs that seem to grow over time. We do
>> >> understand
>> >> that using the REUSE_HTTP_CLIENT option helps in this area but we want
>> >> to
>> >> explore other ways without using this option if any.
>> >>
>> >> Best Regards,
>> >> -Himakar
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
>> For additional commands, e-mail: java-dev-help@axis.apache.org
>>
>
>
>
> --
> Amila Suriarachchi
> WSO2 Inc.
> blog: http://amilachinthaka.blogspot.com/
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
Re: how to avoid CLOSE_WAITs without the use of the REUSE_HTTP_CLIENT option?
Posted by Amila Suriarachchi <am...@gmail.com>.
On Fri, Aug 19, 2011 at 12:43 AM, Andreas Veithen <andreas.veithen@gmail.com
> wrote:
> Although the practical conclusions of that blog post are correct, the
> explanation is nevertheless based on a misunderstanding of what the
> actual issue is. Quoting from the article:
>
> "idea behind recent change in axis2 is to use connection pool facility
> of MTHCM to avoid new http connection every time you call webservice"
>
> "So if we are using axis2 1.5.1 onwards we need to release the
> connection to the pool since we can not have more than one MTHCM"
>
> This is not entirely wrong, but it is still incomplete. The problem in
> Axis2 versions before 1.5.1 was that in addition to creating a new
> connection for every request, the HTTP transport was unable to ensure
> that these connections are properly closed. This caused the CLOSE_WAIT
> problem mentioned by Himakar.
>
> Another consequence of this bug was that for these versions of the
> HTTP transport, it didn't really matter whether the application code
> invoked ServiceClient#cleanupTransport() or not. There was a
> connection leak anyway. In many cases this actually didn't cause an
> incident because the connections would eventually be closed by the
> garbage collector. The consequence of this is that there is lot of
> application code written for Axis2 versions before 1.5.1 that fails to
> call cleanupTransport() in the appropriate places.
>
> The connection leak was finally fixed in Axis2 1.5.1 (kudos to Glen)
> and the HTTP transport now manages the connections properly. However,
> the cleanup code relies on the assumption that the application code
> uses the ServiceClient API properly, and in particular calls
> cleanupTransport where necessary. If the code fails to do that, then
> the connection pool used by the HTTP transport will quickly be
> starved, resulting in a "Timeout waiting for connection" error.
>
> To summarize: application code that triggers a "Timeout waiting for
> connection" with Axis2 >= 1.5.1 was already incorrect in previous
> versions of Axis2, but the symptoms caused by this incorrect code are
> different.
>
> With this knowledge, it is possible to give an answer to Himakar's
> original question:
>
> a. In Axis2 >= 1.5.1, the CLOSE_WAIT issue never occurs and it is in
> general not necessary to configure a custom connection manager or
> HttpClient.
>
Now the error is Timeout since default connection manager is only allow 2
connections. I think this can be improved by initiating the connection
manager at the transport initialization time and having parameters to define
those at the axis2.xml transport sender level.
The bottom line is still it does not fix the connection related problem out
of the box. Users have to use the service Client API properly by calling the
cleanUp transport.
> b. In Axis2 <= 1.5, Keep-Alive connections will always be leaked
> unless one configures a custom connection manager.
There is no connection leak in the earlier version. But what happen is it
takes 30 seconds to close the connection. if the load is high this cause a
CLOSE_WAIT sate connection accumulation.
thanks,
Amila.
> Another option may
> be to disable Keep-Alive or switch to HTTP/1.0. In that case it is
> likely that the call to cleanupTransport() will actually close the
> connection (to be tested!).
>
> Andreas
>
> On Thu, Aug 18, 2011 at 07:02, shivendra tripathi <sh...@gmail.com>
> wrote:
> > Hope below link would be helpful
> >
> http://shivendra-tripathi.blogspot.com/2011/05/http-transport-in-axis2-how-to-fix.html
> >
> > On Thu, Aug 18, 2011 at 3:13 AM, Himakar Chennapragada <
> himakar@us.ibm.com>
> > wrote:
> >>
> >> Is there a way to prevent/avoid or minimize the CLOSE_WAITs from
> happening
> >> when we DO NOT use the REUSE_HTTP_CLIENT option on the ServiceClient
> >> interface.
> >>
> >> We have multiple threads calling out to same or different web services.
> >> We use a new ServiceClient for each invocation to an external web
> service
> >> (same or different service) without the REUSE_HTTP_CLIENT option and
> that
> >> produces multiple CLOSE_WAITs that seem to grow over time. We do
> understand
> >> that using the REUSE_HTTP_CLIENT option helps in this area but we want
> to
> >> explore other ways without using this option if any.
> >>
> >> Best Regards,
> >> -Himakar
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
> For additional commands, e-mail: java-dev-help@axis.apache.org
>
>
--
Amila Suriarachchi
WSO2 Inc.
blog: http://amilachinthaka.blogspot.com/
Re: how to avoid CLOSE_WAITs without the use of the REUSE_HTTP_CLIENT option?
Posted by Andreas Veithen <an...@gmail.com>.
Although the practical conclusions of that blog post are correct, the
explanation is nevertheless based on a misunderstanding of what the
actual issue is. Quoting from the article:
"idea behind recent change in axis2 is to use connection pool facility
of MTHCM to avoid new http connection every time you call webservice"
"So if we are using axis2 1.5.1 onwards we need to release the
connection to the pool since we can not have more than one MTHCM"
This is not entirely wrong, but it is still incomplete. The problem in
Axis2 versions before 1.5.1 was that in addition to creating a new
connection for every request, the HTTP transport was unable to ensure
that these connections are properly closed. This caused the CLOSE_WAIT
problem mentioned by Himakar.
Another consequence of this bug was that for these versions of the
HTTP transport, it didn't really matter whether the application code
invoked ServiceClient#cleanupTransport() or not. There was a
connection leak anyway. In many cases this actually didn't cause an
incident because the connections would eventually be closed by the
garbage collector. The consequence of this is that there is lot of
application code written for Axis2 versions before 1.5.1 that fails to
call cleanupTransport() in the appropriate places.
The connection leak was finally fixed in Axis2 1.5.1 (kudos to Glen)
and the HTTP transport now manages the connections properly. However,
the cleanup code relies on the assumption that the application code
uses the ServiceClient API properly, and in particular calls
cleanupTransport where necessary. If the code fails to do that, then
the connection pool used by the HTTP transport will quickly be
starved, resulting in a "Timeout waiting for connection" error.
To summarize: application code that triggers a "Timeout waiting for
connection" with Axis2 >= 1.5.1 was already incorrect in previous
versions of Axis2, but the symptoms caused by this incorrect code are
different.
With this knowledge, it is possible to give an answer to Himakar's
original question:
a. In Axis2 >= 1.5.1, the CLOSE_WAIT issue never occurs and it is in
general not necessary to configure a custom connection manager or
HttpClient.
b. In Axis2 <= 1.5, Keep-Alive connections will always be leaked
unless one configures a custom connection manager. Another option may
be to disable Keep-Alive or switch to HTTP/1.0. In that case it is
likely that the call to cleanupTransport() will actually close the
connection (to be tested!).
Andreas
On Thu, Aug 18, 2011 at 07:02, shivendra tripathi <sh...@gmail.com> wrote:
> Hope below link would be helpful
> http://shivendra-tripathi.blogspot.com/2011/05/http-transport-in-axis2-how-to-fix.html
>
> On Thu, Aug 18, 2011 at 3:13 AM, Himakar Chennapragada <hi...@us.ibm.com>
> wrote:
>>
>> Is there a way to prevent/avoid or minimize the CLOSE_WAITs from happening
>> when we DO NOT use the REUSE_HTTP_CLIENT option on the ServiceClient
>> interface.
>>
>> We have multiple threads calling out to same or different web services.
>> We use a new ServiceClient for each invocation to an external web service
>> (same or different service) without the REUSE_HTTP_CLIENT option and that
>> produces multiple CLOSE_WAITs that seem to grow over time. We do understand
>> that using the REUSE_HTTP_CLIENT option helps in this area but we want to
>> explore other ways without using this option if any.
>>
>> Best Regards,
>> -Himakar
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
Re: how to avoid CLOSE_WAITs without the use of the REUSE_HTTP_CLIENT option?
Posted by shivendra tripathi <sh...@gmail.com>.
Hope below link would be helpful
http://shivendra-tripathi.blogspot.com/2011/05/http-transport-in-axis2-how-to-fix.html
On Thu, Aug 18, 2011 at 3:13 AM, Himakar Chennapragada
<hi...@us.ibm.com>wrote:
>
> Is there a way to prevent/avoid or minimize the CLOSE_WAITs from happening
> when we DO NOT use the REUSE_HTTP_CLIENT option on the ServiceClient
> interface.
>
> We have multiple threads calling out to *same or different *web services.
> We use a new ServiceClient for *each invocation* to an external web
> service (same or different service) without the REUSE_HTTP_CLIENT option and
> that produces multiple CLOSE_WAITs that seem to grow over time. We do
> understand that using the REUSE_HTTP_CLIENT option helps in this area but we
> want to explore other ways without using this option if any.
>
> Best Regards,
> -Himakar
>