Behavioural difference between 4.0b2 and 4.0-rc1

[Sam Crawford <sa...@gmail.com>]

Morning all,
We upgraded our application to use HttpClient 4.0-RC1 last weekend and
observed a change in behaviour from 4.0-beta2 that I wanted to make you
aware of (it may well be by design, just want to check!).

Firstly, I should say that there was a clear error in our code, meaning that
if the response code was non-2xx we would not read the entire response
stream from the HttpClient properly. This would in turn mean that the
connection was not released automatically and readded to the pool. This is
where the difference lies:

1. In 4.0-beta2 there appears to be some background collector that forcibly
closes connections that have ended up in this state (the log line reads
"Connection garbage collected...")
2. In 4.0-RC1 there is no such collecting process taking place (which in
turn led to the connections not being closed, resulting in the connection
pool being exhausted).

I've attached traces of the occurrence. The comparison was made by using
precisely the same application code and just swapping between the different
httpclient JARs.

Again, the problem was clearly mine with the handling of the response
stream. However, there is clearly a difference between beta2 and RC1, and I
just wanted to make sure it was expected behaviour.

Thanks,

Sam

Re: Behavioural difference between 4.0b2 and 4.0-rc1

[Sam Crawford <sa...@gmail.com>]

Perfect, thanks very much. Just wanted to make sure it wasn't an unexpected
last minute side-effect before the 4.0-final release.
Thanks once again,

Sam


2009/7/7 Oleg Kalnichevski <ol...@apache.org>

> On Tue, Jul 07, 2009 at 09:58:11AM +0100, Sam Crawford wrote:
> > Morning all,
> > We upgraded our application to use HttpClient 4.0-RC1 last weekend and
> > observed a change in behaviour from 4.0-beta2 that I wanted to make you
> > aware of (it may well be by design, just want to check!).
> >
> > Firstly, I should say that there was a clear error in our code, meaning
> that
> > if the response code was non-2xx we would not read the entire response
> > stream from the HttpClient properly. This would in turn mean that the
> > connection was not released automatically and readded to the pool. This
> is
> > where the difference lies:
> >
> > 1. In 4.0-beta2 there appears to be some background collector that
> forcibly
> > closes connections that have ended up in this state (the log line reads
> > "Connection garbage collected...")
> > 2. In 4.0-RC1 there is no such collecting process taking place (which in
> > turn led to the connections not being closed, resulting in the connection
> > pool being exhausted).
> >
> > I've attached traces of the occurrence. The comparison was made by using
> > precisely the same application code and just swapping between the
> different
> > httpclient JARs.
> >
> > Again, the problem was clearly mine with the handling of the response
> > stream. However, there is clearly a difference between beta2 and RC1, and
> I
> > just wanted to make sure it was expected behaviour.
> >
> > Thanks,
> >
> > Sam
>
> Hi Sam,
>
> This is expected behaviour. Until recently HttpClient used some non-trivial
> logic to detect leaked connections using weak references and garbage
> collection. However we got report about a fairly nasty memory leak caused
> by
> that particular bit of code under some rather special circumstances [1].
> The
> developer who contributed most of the new connection management code has
> left
> the project for good. The issue was discussed on the dev list and the
> majority
> of committers felt that the feature was too hacky to be 100% reliable and
> the
> decision was made to remove it completely. For details see the dev mail
> archive
> and HTTPCLIENT-841
>
> Oleg
>
>
> [1] https://issues.apache.org/jira/browse/HTTPCLIENT-841
>
>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>

Re: Behavioural difference between 4.0b2 and 4.0-rc1

[Oleg Kalnichevski <ol...@apache.org>]

On Tue, Jul 07, 2009 at 09:58:11AM +0100, Sam Crawford wrote:
> Morning all,
> We upgraded our application to use HttpClient 4.0-RC1 last weekend and
> observed a change in behaviour from 4.0-beta2 that I wanted to make you
> aware of (it may well be by design, just want to check!).
> 
> Firstly, I should say that there was a clear error in our code, meaning that
> if the response code was non-2xx we would not read the entire response
> stream from the HttpClient properly. This would in turn mean that the
> connection was not released automatically and readded to the pool. This is
> where the difference lies:
> 
> 1. In 4.0-beta2 there appears to be some background collector that forcibly
> closes connections that have ended up in this state (the log line reads
> "Connection garbage collected...")
> 2. In 4.0-RC1 there is no such collecting process taking place (which in
> turn led to the connections not being closed, resulting in the connection
> pool being exhausted).
> 
> I've attached traces of the occurrence. The comparison was made by using
> precisely the same application code and just swapping between the different
> httpclient JARs.
> 
> Again, the problem was clearly mine with the handling of the response
> stream. However, there is clearly a difference between beta2 and RC1, and I
> just wanted to make sure it was expected behaviour.
> 
> Thanks,
> 
> Sam

Hi Sam,

This is expected behaviour. Until recently HttpClient used some non-trivial
logic to detect leaked connections using weak references and garbage
collection. However we got report about a fairly nasty memory leak caused by
that particular bit of code under some rather special circumstances [1]. The
developer who contributed most of the new connection management code has left
the project for good. The issue was discussed on the dev list and the majority
of committers felt that the feature was too hacky to be 100% reliable and the
decision was made to remove it completely. For details see the dev mail archive
and HTTPCLIENT-841

Oleg


[1] https://issues.apache.org/jira/browse/HTTPCLIENT-841


> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org