You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hc.apache.org by Ryan Schmitt <rs...@apache.org> on 2025/05/30 01:08:18 UTC
Change in GC behavior since 5.2.1?
I've been debugging a load test regression, which turned out to be
caused by a client instance leak. The test started failing on 5.4.4
due to running out of file descriptors. What I realized is that leaked
connection pools are eventually cleaned up by garbage collection,
which causes the sockets' file descriptors to be released. What has
changed in 5.4.4 is that this process now takes TWO rounds of garbage
collection. This can be seen in the `lsof` output, which shows where
in the TCP state machine the socket is:
5.2: ESTABLISHED -(gc)-> (gone)
5.4: ESTABLISHED -(gc)-> CLOSE_WAIT -(gc)-> (gone)
Does anyone know what might have changed that would cause this? I'm
specifically asking about the synchronous client (I haven't tested the
async client).
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: Change in GC behavior since 5.2.1?
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Wed, 2025-06-04 at 10:52 -0700, Ryan Schmitt wrote:
> I've updated the reproducer to use a BasicHttpClientResponseHandler.
> The behavior is the same as before.
>
> Unrelated question, but do the HttpClientResponseHandler-variant
> `#execute` methods support response streaming? All the
> implementations
> I've looked at buffer the response entity in its entirety.
>
We do. See #executeOpen.
Oleg
> On Wed, Jun 4, 2025 at 5:47 AM Oleg Kalnichevski <ol...@apache.org>
> wrote:
> >
> > Hi Ryan
> >
> > I am not sure understand what exactly you are trying to do here? Do
> > you
> > intentionally use a deprecated method _and_ fail to correctly
> > return
> > connections back to the pool? Is this the scenario you are trying
> > to
> > test?
> >
> > Oleg
> >
> >
> > On Tue, 2025-06-03 at 14:15 -0700, Ryan Schmitt wrote:
> > > Furthermore, I can confirm that reverting ee0a10210 fixes the
> > > issue.
> > > No httpcore5 changes are required.
> > >
> > > On Tue, Jun 3, 2025 at 2:10 PM Ryan Schmitt <rs...@apache.org>
> > > wrote:
> > > >
> > > > Here's a reproducer:
> > > >
> > > > https://github.com/rschmitt/httpcomponents-client/tree/gc-repro
> > > >
> > > > The bug is exhibited on JDK11 and JDK21, not on JDK8 or JDK21.
> > > > I
> > > > didn't test with any non-LTS releases. Here's JDK17:
> > > >
> > > > [0] ~/src/httpcomponents-client # j17 ./run-example.sh
> > > > SocketLeak
> > > > This Java version SHOULD exhibit the bug where two garbage
> > > > collections are required to clean up leaked sockets.
> > > > Leaking client; sockets should be in ESTABLISHED
> > > > Tue Jun 3 14:04:40 PDT 2025:
> > > > Tue Jun 3 14:04:41 PDT 2025: 2 (ESTABLISHED)
> > > > Tue Jun 3 14:04:42 PDT 2025: 3 (ESTABLISHED)
> > > > Tue Jun 3 14:04:44 PDT 2025: 3 (ESTABLISHED)
> > > > Running garbage collector; sockets should be in CLOSE_WAIT
> > > > Tue Jun 3 14:04:45 PDT 2025: 3 (CLOSE_WAIT)
> > > > Tue Jun 3 14:04:46 PDT 2025: 3 (CLOSE_WAIT)
> > > > Tue Jun 3 14:04:47 PDT 2025: 3 (CLOSE_WAIT)
> > > > Running garbage collector again; no sockets should be
> > > > listed
> > > > Tue Jun 3 14:04:48 PDT 2025:
> > > > Tue Jun 3 14:04:50 PDT 2025:
> > > >
> > > > And here's JDK21:
> > > >
> > > > [0] ~/src/httpcomponents-client # j21 ./run-example.sh
> > > > SocketLeak
> > > > This Java version SHOULD NOT exhibit the bug where two
> > > > garbage
> > > > collections are required to clean up leaked sockets.
> > > > Leaking client; sockets should be in ESTABLISHED
> > > > Tue Jun 3 14:06:25 PDT 2025:
> > > > Tue Jun 3 14:06:26 PDT 2025: 3 (ESTABLISHED)
> > > > Tue Jun 3 14:06:28 PDT 2025: 3 (ESTABLISHED)
> > > > Tue Jun 3 14:06:29 PDT 2025: 3 (ESTABLISHED)
> > > > Running garbage collector; no sockets should be listed
> > > > Tue Jun 3 14:06:30 PDT 2025:
> > > > Tue Jun 3 14:06:31 PDT 2025:
> > > >
> > > > I tested this on both macOS and Linux.
> > > >
> > > > On Fri, May 30, 2025 at 5:14 AM Oleg Kalnichevski
> > > > <ol...@apache.org> wrote:
> > > > >
> > > > > On Fri, 2025-05-30 at 11:30 +0200, Oleg Kalnichevski wrote:
> > > > > > On Thu, 2025-05-29 at 22:54 -0700, Ryan Schmitt wrote:
> > > > > > > It's the `autoClose` setting. It used to be `true`:
> > > > > > >
> > > > > > > https://github.com/apache/httpcomponents-client/blob/d8f702fb4d44c746bb0edf00643aa7139cb8bdf7/httpclient5/src/main/java/org/apache/hc/client5/http/ssl/SSLConnectionSocketFactory.java#L274
> > > > > > >
> > > > > > > Now it's `false`:
> > > > > > >
> > > > > > > https://github.com/apache/httpcomponents-client/blob/ffe4a05f4faf3706a41ff660bcb3474c6b5101a3/httpclient5/src/main/java/org/apache/hc/client5/http/ssl/AbstractClientTlsStrategy.java#L208
> > > > > > >
> > > > > > > This was deliberately changed in this commit:
> > > > > > >
> > > > > > > https://github.com/apache/httpcomponents-client/commit/ee0a102104d03a8d1cfe18e571d179c41242182c
> > > > > > >
> > > > > >
> > > > > > This change was introduced in response to HTTPCLIENT-2328
> > > > > > [1].
> > > > > > I will
> > > > > > review the code for accidental mistakes. Conceptually the
> > > > > > `autoClose`
> > > > > > setting should be merely a convenience.
> > > > > >
> > > > > > Oleg
> > > > > >
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/browse/HTTPCLIENT-2328
> > > > > >
> > > > >
> > > > > I think this change in core is more relevant [1]. At the
> > > > > moment I
> > > > > cannot see anything obviously wrong with it.
> > > > >
> > > > > Would you be able to share (contribute) a reproducer for the
> > > > > defect?
> > > > >
> > > > > Oleg
> > > > >
> > > > > [1]
> > > > > https://github.com/apache/httpcomponents-core/commit/1c95b2994f22b91bc7f54cb66ed84a8e94111a61
> > > > >
> > > > > >
> > > > > >
> > > > > > > On Thu, May 29, 2025 at 6:08 PM Ryan Schmitt
> > > > > > > <rs...@apache.org>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > I've been debugging a load test regression, which
> > > > > > > > turned
> > > > > > > > out to
> > > > > > > > be
> > > > > > > > caused by a client instance leak. The test started
> > > > > > > > failing
> > > > > > > > on
> > > > > > > > 5.4.4
> > > > > > > > due to running out of file descriptors. What I realized
> > > > > > > > is
> > > > > > > > that
> > > > > > > > leaked
> > > > > > > > connection pools are eventually cleaned up by garbage
> > > > > > > > collection,
> > > > > > > > which causes the sockets' file descriptors to be
> > > > > > > > released.
> > > > > > > > What
> > > > > > > > has
> > > > > > > > changed in 5.4.4 is that this process now takes TWO
> > > > > > > > rounds
> > > > > > > > of
> > > > > > > > garbage
> > > > > > > > collection. This can be seen in the `lsof` output,
> > > > > > > > which
> > > > > > > > shows
> > > > > > > > where
> > > > > > > > in the TCP state machine the socket is:
> > > > > > > >
> > > > > > > > 5.2: ESTABLISHED -(gc)-> (gone)
> > > > > > > > 5.4: ESTABLISHED -(gc)-> CLOSE_WAIT -(gc)-> (gone)
> > > > > > > >
> > > > > > > > Does anyone know what might have changed that would
> > > > > > > > cause
> > > > > > > > this?
> > > > > > > > I'm
> > > > > > > > specifically asking about the synchronous client (I
> > > > > > > > haven't
> > > > > > > > tested
> > > > > > > > the
> > > > > > > > async client).
> > > > > > >
> > > > > > > ---------------------------------------------------------
> > > > > > > ----
> > > > > > > ------
> > > > > > > --
> > > > > > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > > > > > For additional commands, e-mail: dev-help@hc.apache.org
> > > > > > >
> > > > > >
> > > > > >
> > > > > > -----------------------------------------------------------
> > > > > > ----
> > > > > > ------
> > > > > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > > > > For additional commands, e-mail: dev-help@hc.apache.org
> > > > > >
> > > > >
> > > > >
> > > > > -------------------------------------------------------------
> > > > > ----
> > > > > ----
> > > > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > > > For additional commands, e-mail: dev-help@hc.apache.org
> > > > >
> > >
> > > -----------------------------------------------------------------
> > > ----
> > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: dev-help@hc.apache.org
> > >
> >
> >
> > -------------------------------------------------------------------
> > --
> > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > For additional commands, e-mail: dev-help@hc.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: Change in GC behavior since 5.2.1?
Posted by Ryan Schmitt <rs...@apache.org>.
Finalizers are deprecated for removal:
https://openjdk.org/jeps/421
I don't think there's any particular requirement for the HttpClient to
tolerate this sort of misuse, but it is nevertheless a change in
behavior and I wanted to get to the bottom of it.
On Thu, Jun 5, 2025 at 5:35 AM Oleg Kalnichevski <ol...@apache.org> wrote:
>
> Hi Ryan
>
> The reproducer stopped working for me (I use Ubuntu Linux) but this no
> longer matters. I think there is a reasonable explanation as to why it
> now may take two GC cycles to fully reclaim file descriptors in case of
> a connection pool going out of scope without being properly shut down.
>
> I think the first GC cycle releases connections and their SSLSockets.
> However SSLSockets with autoClose set to false no longer close out the
> socket they have been layered on top of. With some JREs it takes
> another GC cycle to release the underlying Socket objects and system
> resources allocated by them. This also explains why the issue appears
> to be JRE version specific.
>
> The best solution of course is to close HttpClient prior to letting it
> ho out of scope. That would be preferred. Alternatively we could add
> #finalize to managed connection classes
>
> https://github.com/ok2c/httpcomponents-client/commit/9cd760ee282f284824a6ce86621427babf6331fc
>
> Oleg
>
>
>
> On Wed, 2025-06-04 at 10:52 -0700, Ryan Schmitt wrote:
> > I've updated the reproducer to use a BasicHttpClientResponseHandler.
> > The behavior is the same as before.
> >
> > Unrelated question, but do the HttpClientResponseHandler-variant
> > `#execute` methods support response streaming? All the
> > implementations
> > I've looked at buffer the response entity in its entirety.
> >
> > On Wed, Jun 4, 2025 at 5:47 AM Oleg Kalnichevski <ol...@apache.org>
> > wrote:
> > >
> > > Hi Ryan
> > >
> > > I am not sure understand what exactly you are trying to do here? Do
> > > you
> > > intentionally use a deprecated method _and_ fail to correctly
> > > return
> > > connections back to the pool? Is this the scenario you are trying
> > > to
> > > test?
> > >
> > > Oleg
> > >
> > >
> > > On Tue, 2025-06-03 at 14:15 -0700, Ryan Schmitt wrote:
> > > > Furthermore, I can confirm that reverting ee0a10210 fixes the
> > > > issue.
> > > > No httpcore5 changes are required.
> > > >
> > > > On Tue, Jun 3, 2025 at 2:10 PM Ryan Schmitt <rs...@apache.org>
> > > > wrote:
> > > > >
> > > > > Here's a reproducer:
> > > > >
> > > > > https://github.com/rschmitt/httpcomponents-client/tree/gc-repro
> > > > >
> > > > > The bug is exhibited on JDK11 and JDK21, not on JDK8 or JDK21.
> > > > > I
> > > > > didn't test with any non-LTS releases. Here's JDK17:
> > > > >
> > > > > [0] ~/src/httpcomponents-client # j17 ./run-example.sh
> > > > > SocketLeak
> > > > > This Java version SHOULD exhibit the bug where two garbage
> > > > > collections are required to clean up leaked sockets.
> > > > > Leaking client; sockets should be in ESTABLISHED
> > > > > Tue Jun 3 14:04:40 PDT 2025:
> > > > > Tue Jun 3 14:04:41 PDT 2025: 2 (ESTABLISHED)
> > > > > Tue Jun 3 14:04:42 PDT 2025: 3 (ESTABLISHED)
> > > > > Tue Jun 3 14:04:44 PDT 2025: 3 (ESTABLISHED)
> > > > > Running garbage collector; sockets should be in CLOSE_WAIT
> > > > > Tue Jun 3 14:04:45 PDT 2025: 3 (CLOSE_WAIT)
> > > > > Tue Jun 3 14:04:46 PDT 2025: 3 (CLOSE_WAIT)
> > > > > Tue Jun 3 14:04:47 PDT 2025: 3 (CLOSE_WAIT)
> > > > > Running garbage collector again; no sockets should be
> > > > > listed
> > > > > Tue Jun 3 14:04:48 PDT 2025:
> > > > > Tue Jun 3 14:04:50 PDT 2025:
> > > > >
> > > > > And here's JDK21:
> > > > >
> > > > > [0] ~/src/httpcomponents-client # j21 ./run-example.sh
> > > > > SocketLeak
> > > > > This Java version SHOULD NOT exhibit the bug where two
> > > > > garbage
> > > > > collections are required to clean up leaked sockets.
> > > > > Leaking client; sockets should be in ESTABLISHED
> > > > > Tue Jun 3 14:06:25 PDT 2025:
> > > > > Tue Jun 3 14:06:26 PDT 2025: 3 (ESTABLISHED)
> > > > > Tue Jun 3 14:06:28 PDT 2025: 3 (ESTABLISHED)
> > > > > Tue Jun 3 14:06:29 PDT 2025: 3 (ESTABLISHED)
> > > > > Running garbage collector; no sockets should be listed
> > > > > Tue Jun 3 14:06:30 PDT 2025:
> > > > > Tue Jun 3 14:06:31 PDT 2025:
> > > > >
> > > > > I tested this on both macOS and Linux.
> > > > >
> > > > > On Fri, May 30, 2025 at 5:14 AM Oleg Kalnichevski
> > > > > <ol...@apache.org> wrote:
> > > > > >
> > > > > > On Fri, 2025-05-30 at 11:30 +0200, Oleg Kalnichevski wrote:
> > > > > > > On Thu, 2025-05-29 at 22:54 -0700, Ryan Schmitt wrote:
> > > > > > > > It's the `autoClose` setting. It used to be `true`:
> > > > > > > >
> > > > > > > > https://github.com/apache/httpcomponents-client/blob/d8f702fb4d44c746bb0edf00643aa7139cb8bdf7/httpclient5/src/main/java/org/apache/hc/client5/http/ssl/SSLConnectionSocketFactory.java#L274
> > > > > > > >
> > > > > > > > Now it's `false`:
> > > > > > > >
> > > > > > > > https://github.com/apache/httpcomponents-client/blob/ffe4a05f4faf3706a41ff660bcb3474c6b5101a3/httpclient5/src/main/java/org/apache/hc/client5/http/ssl/AbstractClientTlsStrategy.java#L208
> > > > > > > >
> > > > > > > > This was deliberately changed in this commit:
> > > > > > > >
> > > > > > > > https://github.com/apache/httpcomponents-client/commit/ee0a102104d03a8d1cfe18e571d179c41242182c
> > > > > > > >
> > > > > > >
> > > > > > > This change was introduced in response to HTTPCLIENT-2328
> > > > > > > [1].
> > > > > > > I will
> > > > > > > review the code for accidental mistakes. Conceptually the
> > > > > > > `autoClose`
> > > > > > > setting should be merely a convenience.
> > > > > > >
> > > > > > > Oleg
> > > > > > >
> > > > > > >
> > > > > > > [1] https://issues.apache.org/jira/browse/HTTPCLIENT-2328
> > > > > > >
> > > > > >
> > > > > > I think this change in core is more relevant [1]. At the
> > > > > > moment I
> > > > > > cannot see anything obviously wrong with it.
> > > > > >
> > > > > > Would you be able to share (contribute) a reproducer for the
> > > > > > defect?
> > > > > >
> > > > > > Oleg
> > > > > >
> > > > > > [1]
> > > > > > https://github.com/apache/httpcomponents-core/commit/1c95b2994f22b91bc7f54cb66ed84a8e94111a61
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > On Thu, May 29, 2025 at 6:08 PM Ryan Schmitt
> > > > > > > > <rs...@apache.org>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > I've been debugging a load test regression, which
> > > > > > > > > turned
> > > > > > > > > out to
> > > > > > > > > be
> > > > > > > > > caused by a client instance leak. The test started
> > > > > > > > > failing
> > > > > > > > > on
> > > > > > > > > 5.4.4
> > > > > > > > > due to running out of file descriptors. What I realized
> > > > > > > > > is
> > > > > > > > > that
> > > > > > > > > leaked
> > > > > > > > > connection pools are eventually cleaned up by garbage
> > > > > > > > > collection,
> > > > > > > > > which causes the sockets' file descriptors to be
> > > > > > > > > released.
> > > > > > > > > What
> > > > > > > > > has
> > > > > > > > > changed in 5.4.4 is that this process now takes TWO
> > > > > > > > > rounds
> > > > > > > > > of
> > > > > > > > > garbage
> > > > > > > > > collection. This can be seen in the `lsof` output,
> > > > > > > > > which
> > > > > > > > > shows
> > > > > > > > > where
> > > > > > > > > in the TCP state machine the socket is:
> > > > > > > > >
> > > > > > > > > 5.2: ESTABLISHED -(gc)-> (gone)
> > > > > > > > > 5.4: ESTABLISHED -(gc)-> CLOSE_WAIT -(gc)-> (gone)
> > > > > > > > >
> > > > > > > > > Does anyone know what might have changed that would
> > > > > > > > > cause
> > > > > > > > > this?
> > > > > > > > > I'm
> > > > > > > > > specifically asking about the synchronous client (I
> > > > > > > > > haven't
> > > > > > > > > tested
> > > > > > > > > the
> > > > > > > > > async client).
> > > > > > > >
> > > > > > > > ---------------------------------------------------------
> > > > > > > > ----
> > > > > > > > ------
> > > > > > > > --
> > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > > > > > > For additional commands, e-mail: dev-help@hc.apache.org
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > -----------------------------------------------------------
> > > > > > > ----
> > > > > > > ------
> > > > > > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > > > > > For additional commands, e-mail: dev-help@hc.apache.org
> > > > > > >
> > > > > >
> > > > > >
> > > > > > -------------------------------------------------------------
> > > > > > ----
> > > > > > ----
> > > > > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > > > > For additional commands, e-mail: dev-help@hc.apache.org
> > > > > >
> > > >
> > > > -----------------------------------------------------------------
> > > > ----
> > > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail: dev-help@hc.apache.org
> > > >
> > >
> > >
> > > -------------------------------------------------------------------
> > > --
> > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: dev-help@hc.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > For additional commands, e-mail: dev-help@hc.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: Change in GC behavior since 5.2.1?
Posted by Gary Gregory <ga...@gmail.com>.
I think it is worth noting that GC behavior can not only differ between
Java versions but can also be affected by JVM command line parameters. So
explaing GC precisely as it relates to our code in an application is very
difficult.
Gary
On Thu, Jun 5, 2025, 08:35 Oleg Kalnichevski <ol...@apache.org> wrote:
> Hi Ryan
>
> The reproducer stopped working for me (I use Ubuntu Linux) but this no
> longer matters. I think there is a reasonable explanation as to why it
> now may take two GC cycles to fully reclaim file descriptors in case of
> a connection pool going out of scope without being properly shut down.
>
> I think the first GC cycle releases connections and their SSLSockets.
> However SSLSockets with autoClose set to false no longer close out the
> socket they have been layered on top of. With some JREs it takes
> another GC cycle to release the underlying Socket objects and system
> resources allocated by them. This also explains why the issue appears
> to be JRE version specific.
>
> The best solution of course is to close HttpClient prior to letting it
> ho out of scope. That would be preferred. Alternatively we could add
> #finalize to managed connection classes
>
>
> https://github.com/ok2c/httpcomponents-client/commit/9cd760ee282f284824a6ce86621427babf6331fc
>
> Oleg
>
>
>
> On Wed, 2025-06-04 at 10:52 -0700, Ryan Schmitt wrote:
> > I've updated the reproducer to use a BasicHttpClientResponseHandler.
> > The behavior is the same as before.
> >
> > Unrelated question, but do the HttpClientResponseHandler-variant
> > `#execute` methods support response streaming? All the
> > implementations
> > I've looked at buffer the response entity in its entirety.
> >
> > On Wed, Jun 4, 2025 at 5:47 AM Oleg Kalnichevski <ol...@apache.org>
> > wrote:
> > >
> > > Hi Ryan
> > >
> > > I am not sure understand what exactly you are trying to do here? Do
> > > you
> > > intentionally use a deprecated method _and_ fail to correctly
> > > return
> > > connections back to the pool? Is this the scenario you are trying
> > > to
> > > test?
> > >
> > > Oleg
> > >
> > >
> > > On Tue, 2025-06-03 at 14:15 -0700, Ryan Schmitt wrote:
> > > > Furthermore, I can confirm that reverting ee0a10210 fixes the
> > > > issue.
> > > > No httpcore5 changes are required.
> > > >
> > > > On Tue, Jun 3, 2025 at 2:10 PM Ryan Schmitt <rs...@apache.org>
> > > > wrote:
> > > > >
> > > > > Here's a reproducer:
> > > > >
> > > > > https://github.com/rschmitt/httpcomponents-client/tree/gc-repro
> > > > >
> > > > > The bug is exhibited on JDK11 and JDK21, not on JDK8 or JDK21.
> > > > > I
> > > > > didn't test with any non-LTS releases. Here's JDK17:
> > > > >
> > > > > [0] ~/src/httpcomponents-client # j17 ./run-example.sh
> > > > > SocketLeak
> > > > > This Java version SHOULD exhibit the bug where two garbage
> > > > > collections are required to clean up leaked sockets.
> > > > > Leaking client; sockets should be in ESTABLISHED
> > > > > Tue Jun 3 14:04:40 PDT 2025:
> > > > > Tue Jun 3 14:04:41 PDT 2025: 2 (ESTABLISHED)
> > > > > Tue Jun 3 14:04:42 PDT 2025: 3 (ESTABLISHED)
> > > > > Tue Jun 3 14:04:44 PDT 2025: 3 (ESTABLISHED)
> > > > > Running garbage collector; sockets should be in CLOSE_WAIT
> > > > > Tue Jun 3 14:04:45 PDT 2025: 3 (CLOSE_WAIT)
> > > > > Tue Jun 3 14:04:46 PDT 2025: 3 (CLOSE_WAIT)
> > > > > Tue Jun 3 14:04:47 PDT 2025: 3 (CLOSE_WAIT)
> > > > > Running garbage collector again; no sockets should be
> > > > > listed
> > > > > Tue Jun 3 14:04:48 PDT 2025:
> > > > > Tue Jun 3 14:04:50 PDT 2025:
> > > > >
> > > > > And here's JDK21:
> > > > >
> > > > > [0] ~/src/httpcomponents-client # j21 ./run-example.sh
> > > > > SocketLeak
> > > > > This Java version SHOULD NOT exhibit the bug where two
> > > > > garbage
> > > > > collections are required to clean up leaked sockets.
> > > > > Leaking client; sockets should be in ESTABLISHED
> > > > > Tue Jun 3 14:06:25 PDT 2025:
> > > > > Tue Jun 3 14:06:26 PDT 2025: 3 (ESTABLISHED)
> > > > > Tue Jun 3 14:06:28 PDT 2025: 3 (ESTABLISHED)
> > > > > Tue Jun 3 14:06:29 PDT 2025: 3 (ESTABLISHED)
> > > > > Running garbage collector; no sockets should be listed
> > > > > Tue Jun 3 14:06:30 PDT 2025:
> > > > > Tue Jun 3 14:06:31 PDT 2025:
> > > > >
> > > > > I tested this on both macOS and Linux.
> > > > >
> > > > > On Fri, May 30, 2025 at 5:14 AM Oleg Kalnichevski
> > > > > <ol...@apache.org> wrote:
> > > > > >
> > > > > > On Fri, 2025-05-30 at 11:30 +0200, Oleg Kalnichevski wrote:
> > > > > > > On Thu, 2025-05-29 at 22:54 -0700, Ryan Schmitt wrote:
> > > > > > > > It's the `autoClose` setting. It used to be `true`:
> > > > > > > >
> > > > > > > >
> https://github.com/apache/httpcomponents-client/blob/d8f702fb4d44c746bb0edf00643aa7139cb8bdf7/httpclient5/src/main/java/org/apache/hc/client5/http/ssl/SSLConnectionSocketFactory.java#L274
> > > > > > > >
> > > > > > > > Now it's `false`:
> > > > > > > >
> > > > > > > >
> https://github.com/apache/httpcomponents-client/blob/ffe4a05f4faf3706a41ff660bcb3474c6b5101a3/httpclient5/src/main/java/org/apache/hc/client5/http/ssl/AbstractClientTlsStrategy.java#L208
> > > > > > > >
> > > > > > > > This was deliberately changed in this commit:
> > > > > > > >
> > > > > > > >
> https://github.com/apache/httpcomponents-client/commit/ee0a102104d03a8d1cfe18e571d179c41242182c
> > > > > > > >
> > > > > > >
> > > > > > > This change was introduced in response to HTTPCLIENT-2328
> > > > > > > [1].
> > > > > > > I will
> > > > > > > review the code for accidental mistakes. Conceptually the
> > > > > > > `autoClose`
> > > > > > > setting should be merely a convenience.
> > > > > > >
> > > > > > > Oleg
> > > > > > >
> > > > > > >
> > > > > > > [1] https://issues.apache.org/jira/browse/HTTPCLIENT-2328
> > > > > > >
> > > > > >
> > > > > > I think this change in core is more relevant [1]. At the
> > > > > > moment I
> > > > > > cannot see anything obviously wrong with it.
> > > > > >
> > > > > > Would you be able to share (contribute) a reproducer for the
> > > > > > defect?
> > > > > >
> > > > > > Oleg
> > > > > >
> > > > > > [1]
> > > > > >
> https://github.com/apache/httpcomponents-core/commit/1c95b2994f22b91bc7f54cb66ed84a8e94111a61
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > On Thu, May 29, 2025 at 6:08 PM Ryan Schmitt
> > > > > > > > <rs...@apache.org>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > I've been debugging a load test regression, which
> > > > > > > > > turned
> > > > > > > > > out to
> > > > > > > > > be
> > > > > > > > > caused by a client instance leak. The test started
> > > > > > > > > failing
> > > > > > > > > on
> > > > > > > > > 5.4.4
> > > > > > > > > due to running out of file descriptors. What I realized
> > > > > > > > > is
> > > > > > > > > that
> > > > > > > > > leaked
> > > > > > > > > connection pools are eventually cleaned up by garbage
> > > > > > > > > collection,
> > > > > > > > > which causes the sockets' file descriptors to be
> > > > > > > > > released.
> > > > > > > > > What
> > > > > > > > > has
> > > > > > > > > changed in 5.4.4 is that this process now takes TWO
> > > > > > > > > rounds
> > > > > > > > > of
> > > > > > > > > garbage
> > > > > > > > > collection. This can be seen in the `lsof` output,
> > > > > > > > > which
> > > > > > > > > shows
> > > > > > > > > where
> > > > > > > > > in the TCP state machine the socket is:
> > > > > > > > >
> > > > > > > > > 5.2: ESTABLISHED -(gc)-> (gone)
> > > > > > > > > 5.4: ESTABLISHED -(gc)-> CLOSE_WAIT -(gc)-> (gone)
> > > > > > > > >
> > > > > > > > > Does anyone know what might have changed that would
> > > > > > > > > cause
> > > > > > > > > this?
> > > > > > > > > I'm
> > > > > > > > > specifically asking about the synchronous client (I
> > > > > > > > > haven't
> > > > > > > > > tested
> > > > > > > > > the
> > > > > > > > > async client).
> > > > > > > >
> > > > > > > > ---------------------------------------------------------
> > > > > > > > ----
> > > > > > > > ------
> > > > > > > > --
> > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > > > > > > For additional commands, e-mail: dev-help@hc.apache.org
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > -----------------------------------------------------------
> > > > > > > ----
> > > > > > > ------
> > > > > > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > > > > > For additional commands, e-mail: dev-help@hc.apache.org
> > > > > > >
> > > > > >
> > > > > >
> > > > > > -------------------------------------------------------------
> > > > > > ----
> > > > > > ----
> > > > > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > > > > For additional commands, e-mail: dev-help@hc.apache.org
> > > > > >
> > > >
> > > > -----------------------------------------------------------------
> > > > ----
> > > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail: dev-help@hc.apache.org
> > > >
> > >
> > >
> > > -------------------------------------------------------------------
> > > --
> > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: dev-help@hc.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > For additional commands, e-mail: dev-help@hc.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
>
Re: Change in GC behavior since 5.2.1?
Posted by Oleg Kalnichevski <ol...@apache.org>.
Hi Ryan
The reproducer stopped working for me (I use Ubuntu Linux) but this no
longer matters. I think there is a reasonable explanation as to why it
now may take two GC cycles to fully reclaim file descriptors in case of
a connection pool going out of scope without being properly shut down.
I think the first GC cycle releases connections and their SSLSockets.
However SSLSockets with autoClose set to false no longer close out the
socket they have been layered on top of. With some JREs it takes
another GC cycle to release the underlying Socket objects and system
resources allocated by them. This also explains why the issue appears
to be JRE version specific.
The best solution of course is to close HttpClient prior to letting it
ho out of scope. That would be preferred. Alternatively we could add
#finalize to managed connection classes
https://github.com/ok2c/httpcomponents-client/commit/9cd760ee282f284824a6ce86621427babf6331fc
Oleg
On Wed, 2025-06-04 at 10:52 -0700, Ryan Schmitt wrote:
> I've updated the reproducer to use a BasicHttpClientResponseHandler.
> The behavior is the same as before.
>
> Unrelated question, but do the HttpClientResponseHandler-variant
> `#execute` methods support response streaming? All the
> implementations
> I've looked at buffer the response entity in its entirety.
>
> On Wed, Jun 4, 2025 at 5:47 AM Oleg Kalnichevski <ol...@apache.org>
> wrote:
> >
> > Hi Ryan
> >
> > I am not sure understand what exactly you are trying to do here? Do
> > you
> > intentionally use a deprecated method _and_ fail to correctly
> > return
> > connections back to the pool? Is this the scenario you are trying
> > to
> > test?
> >
> > Oleg
> >
> >
> > On Tue, 2025-06-03 at 14:15 -0700, Ryan Schmitt wrote:
> > > Furthermore, I can confirm that reverting ee0a10210 fixes the
> > > issue.
> > > No httpcore5 changes are required.
> > >
> > > On Tue, Jun 3, 2025 at 2:10 PM Ryan Schmitt <rs...@apache.org>
> > > wrote:
> > > >
> > > > Here's a reproducer:
> > > >
> > > > https://github.com/rschmitt/httpcomponents-client/tree/gc-repro
> > > >
> > > > The bug is exhibited on JDK11 and JDK21, not on JDK8 or JDK21.
> > > > I
> > > > didn't test with any non-LTS releases. Here's JDK17:
> > > >
> > > > [0] ~/src/httpcomponents-client # j17 ./run-example.sh
> > > > SocketLeak
> > > > This Java version SHOULD exhibit the bug where two garbage
> > > > collections are required to clean up leaked sockets.
> > > > Leaking client; sockets should be in ESTABLISHED
> > > > Tue Jun 3 14:04:40 PDT 2025:
> > > > Tue Jun 3 14:04:41 PDT 2025: 2 (ESTABLISHED)
> > > > Tue Jun 3 14:04:42 PDT 2025: 3 (ESTABLISHED)
> > > > Tue Jun 3 14:04:44 PDT 2025: 3 (ESTABLISHED)
> > > > Running garbage collector; sockets should be in CLOSE_WAIT
> > > > Tue Jun 3 14:04:45 PDT 2025: 3 (CLOSE_WAIT)
> > > > Tue Jun 3 14:04:46 PDT 2025: 3 (CLOSE_WAIT)
> > > > Tue Jun 3 14:04:47 PDT 2025: 3 (CLOSE_WAIT)
> > > > Running garbage collector again; no sockets should be
> > > > listed
> > > > Tue Jun 3 14:04:48 PDT 2025:
> > > > Tue Jun 3 14:04:50 PDT 2025:
> > > >
> > > > And here's JDK21:
> > > >
> > > > [0] ~/src/httpcomponents-client # j21 ./run-example.sh
> > > > SocketLeak
> > > > This Java version SHOULD NOT exhibit the bug where two
> > > > garbage
> > > > collections are required to clean up leaked sockets.
> > > > Leaking client; sockets should be in ESTABLISHED
> > > > Tue Jun 3 14:06:25 PDT 2025:
> > > > Tue Jun 3 14:06:26 PDT 2025: 3 (ESTABLISHED)
> > > > Tue Jun 3 14:06:28 PDT 2025: 3 (ESTABLISHED)
> > > > Tue Jun 3 14:06:29 PDT 2025: 3 (ESTABLISHED)
> > > > Running garbage collector; no sockets should be listed
> > > > Tue Jun 3 14:06:30 PDT 2025:
> > > > Tue Jun 3 14:06:31 PDT 2025:
> > > >
> > > > I tested this on both macOS and Linux.
> > > >
> > > > On Fri, May 30, 2025 at 5:14 AM Oleg Kalnichevski
> > > > <ol...@apache.org> wrote:
> > > > >
> > > > > On Fri, 2025-05-30 at 11:30 +0200, Oleg Kalnichevski wrote:
> > > > > > On Thu, 2025-05-29 at 22:54 -0700, Ryan Schmitt wrote:
> > > > > > > It's the `autoClose` setting. It used to be `true`:
> > > > > > >
> > > > > > > https://github.com/apache/httpcomponents-client/blob/d8f702fb4d44c746bb0edf00643aa7139cb8bdf7/httpclient5/src/main/java/org/apache/hc/client5/http/ssl/SSLConnectionSocketFactory.java#L274
> > > > > > >
> > > > > > > Now it's `false`:
> > > > > > >
> > > > > > > https://github.com/apache/httpcomponents-client/blob/ffe4a05f4faf3706a41ff660bcb3474c6b5101a3/httpclient5/src/main/java/org/apache/hc/client5/http/ssl/AbstractClientTlsStrategy.java#L208
> > > > > > >
> > > > > > > This was deliberately changed in this commit:
> > > > > > >
> > > > > > > https://github.com/apache/httpcomponents-client/commit/ee0a102104d03a8d1cfe18e571d179c41242182c
> > > > > > >
> > > > > >
> > > > > > This change was introduced in response to HTTPCLIENT-2328
> > > > > > [1].
> > > > > > I will
> > > > > > review the code for accidental mistakes. Conceptually the
> > > > > > `autoClose`
> > > > > > setting should be merely a convenience.
> > > > > >
> > > > > > Oleg
> > > > > >
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/browse/HTTPCLIENT-2328
> > > > > >
> > > > >
> > > > > I think this change in core is more relevant [1]. At the
> > > > > moment I
> > > > > cannot see anything obviously wrong with it.
> > > > >
> > > > > Would you be able to share (contribute) a reproducer for the
> > > > > defect?
> > > > >
> > > > > Oleg
> > > > >
> > > > > [1]
> > > > > https://github.com/apache/httpcomponents-core/commit/1c95b2994f22b91bc7f54cb66ed84a8e94111a61
> > > > >
> > > > > >
> > > > > >
> > > > > > > On Thu, May 29, 2025 at 6:08 PM Ryan Schmitt
> > > > > > > <rs...@apache.org>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > I've been debugging a load test regression, which
> > > > > > > > turned
> > > > > > > > out to
> > > > > > > > be
> > > > > > > > caused by a client instance leak. The test started
> > > > > > > > failing
> > > > > > > > on
> > > > > > > > 5.4.4
> > > > > > > > due to running out of file descriptors. What I realized
> > > > > > > > is
> > > > > > > > that
> > > > > > > > leaked
> > > > > > > > connection pools are eventually cleaned up by garbage
> > > > > > > > collection,
> > > > > > > > which causes the sockets' file descriptors to be
> > > > > > > > released.
> > > > > > > > What
> > > > > > > > has
> > > > > > > > changed in 5.4.4 is that this process now takes TWO
> > > > > > > > rounds
> > > > > > > > of
> > > > > > > > garbage
> > > > > > > > collection. This can be seen in the `lsof` output,
> > > > > > > > which
> > > > > > > > shows
> > > > > > > > where
> > > > > > > > in the TCP state machine the socket is:
> > > > > > > >
> > > > > > > > 5.2: ESTABLISHED -(gc)-> (gone)
> > > > > > > > 5.4: ESTABLISHED -(gc)-> CLOSE_WAIT -(gc)-> (gone)
> > > > > > > >
> > > > > > > > Does anyone know what might have changed that would
> > > > > > > > cause
> > > > > > > > this?
> > > > > > > > I'm
> > > > > > > > specifically asking about the synchronous client (I
> > > > > > > > haven't
> > > > > > > > tested
> > > > > > > > the
> > > > > > > > async client).
> > > > > > >
> > > > > > > ---------------------------------------------------------
> > > > > > > ----
> > > > > > > ------
> > > > > > > --
> > > > > > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > > > > > For additional commands, e-mail: dev-help@hc.apache.org
> > > > > > >
> > > > > >
> > > > > >
> > > > > > -----------------------------------------------------------
> > > > > > ----
> > > > > > ------
> > > > > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > > > > For additional commands, e-mail: dev-help@hc.apache.org
> > > > > >
> > > > >
> > > > >
> > > > > -------------------------------------------------------------
> > > > > ----
> > > > > ----
> > > > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > > > For additional commands, e-mail: dev-help@hc.apache.org
> > > > >
> > >
> > > -----------------------------------------------------------------
> > > ----
> > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: dev-help@hc.apache.org
> > >
> >
> >
> > -------------------------------------------------------------------
> > --
> > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > For additional commands, e-mail: dev-help@hc.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: Change in GC behavior since 5.2.1?
Posted by Ryan Schmitt <rs...@apache.org>.
I've updated the reproducer to use a BasicHttpClientResponseHandler.
The behavior is the same as before.
Unrelated question, but do the HttpClientResponseHandler-variant
`#execute` methods support response streaming? All the implementations
I've looked at buffer the response entity in its entirety.
On Wed, Jun 4, 2025 at 5:47 AM Oleg Kalnichevski <ol...@apache.org> wrote:
>
> Hi Ryan
>
> I am not sure understand what exactly you are trying to do here? Do you
> intentionally use a deprecated method _and_ fail to correctly return
> connections back to the pool? Is this the scenario you are trying to
> test?
>
> Oleg
>
>
> On Tue, 2025-06-03 at 14:15 -0700, Ryan Schmitt wrote:
> > Furthermore, I can confirm that reverting ee0a10210 fixes the issue.
> > No httpcore5 changes are required.
> >
> > On Tue, Jun 3, 2025 at 2:10 PM Ryan Schmitt <rs...@apache.org>
> > wrote:
> > >
> > > Here's a reproducer:
> > >
> > > https://github.com/rschmitt/httpcomponents-client/tree/gc-repro
> > >
> > > The bug is exhibited on JDK11 and JDK21, not on JDK8 or JDK21. I
> > > didn't test with any non-LTS releases. Here's JDK17:
> > >
> > > [0] ~/src/httpcomponents-client # j17 ./run-example.sh
> > > SocketLeak
> > > This Java version SHOULD exhibit the bug where two garbage
> > > collections are required to clean up leaked sockets.
> > > Leaking client; sockets should be in ESTABLISHED
> > > Tue Jun 3 14:04:40 PDT 2025:
> > > Tue Jun 3 14:04:41 PDT 2025: 2 (ESTABLISHED)
> > > Tue Jun 3 14:04:42 PDT 2025: 3 (ESTABLISHED)
> > > Tue Jun 3 14:04:44 PDT 2025: 3 (ESTABLISHED)
> > > Running garbage collector; sockets should be in CLOSE_WAIT
> > > Tue Jun 3 14:04:45 PDT 2025: 3 (CLOSE_WAIT)
> > > Tue Jun 3 14:04:46 PDT 2025: 3 (CLOSE_WAIT)
> > > Tue Jun 3 14:04:47 PDT 2025: 3 (CLOSE_WAIT)
> > > Running garbage collector again; no sockets should be listed
> > > Tue Jun 3 14:04:48 PDT 2025:
> > > Tue Jun 3 14:04:50 PDT 2025:
> > >
> > > And here's JDK21:
> > >
> > > [0] ~/src/httpcomponents-client # j21 ./run-example.sh
> > > SocketLeak
> > > This Java version SHOULD NOT exhibit the bug where two garbage
> > > collections are required to clean up leaked sockets.
> > > Leaking client; sockets should be in ESTABLISHED
> > > Tue Jun 3 14:06:25 PDT 2025:
> > > Tue Jun 3 14:06:26 PDT 2025: 3 (ESTABLISHED)
> > > Tue Jun 3 14:06:28 PDT 2025: 3 (ESTABLISHED)
> > > Tue Jun 3 14:06:29 PDT 2025: 3 (ESTABLISHED)
> > > Running garbage collector; no sockets should be listed
> > > Tue Jun 3 14:06:30 PDT 2025:
> > > Tue Jun 3 14:06:31 PDT 2025:
> > >
> > > I tested this on both macOS and Linux.
> > >
> > > On Fri, May 30, 2025 at 5:14 AM Oleg Kalnichevski
> > > <ol...@apache.org> wrote:
> > > >
> > > > On Fri, 2025-05-30 at 11:30 +0200, Oleg Kalnichevski wrote:
> > > > > On Thu, 2025-05-29 at 22:54 -0700, Ryan Schmitt wrote:
> > > > > > It's the `autoClose` setting. It used to be `true`:
> > > > > >
> > > > > > https://github.com/apache/httpcomponents-client/blob/d8f702fb4d44c746bb0edf00643aa7139cb8bdf7/httpclient5/src/main/java/org/apache/hc/client5/http/ssl/SSLConnectionSocketFactory.java#L274
> > > > > >
> > > > > > Now it's `false`:
> > > > > >
> > > > > > https://github.com/apache/httpcomponents-client/blob/ffe4a05f4faf3706a41ff660bcb3474c6b5101a3/httpclient5/src/main/java/org/apache/hc/client5/http/ssl/AbstractClientTlsStrategy.java#L208
> > > > > >
> > > > > > This was deliberately changed in this commit:
> > > > > >
> > > > > > https://github.com/apache/httpcomponents-client/commit/ee0a102104d03a8d1cfe18e571d179c41242182c
> > > > > >
> > > > >
> > > > > This change was introduced in response to HTTPCLIENT-2328 [1].
> > > > > I will
> > > > > review the code for accidental mistakes. Conceptually the
> > > > > `autoClose`
> > > > > setting should be merely a convenience.
> > > > >
> > > > > Oleg
> > > > >
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/HTTPCLIENT-2328
> > > > >
> > > >
> > > > I think this change in core is more relevant [1]. At the moment I
> > > > cannot see anything obviously wrong with it.
> > > >
> > > > Would you be able to share (contribute) a reproducer for the
> > > > defect?
> > > >
> > > > Oleg
> > > >
> > > > [1]
> > > > https://github.com/apache/httpcomponents-core/commit/1c95b2994f22b91bc7f54cb66ed84a8e94111a61
> > > >
> > > > >
> > > > >
> > > > > > On Thu, May 29, 2025 at 6:08 PM Ryan Schmitt
> > > > > > <rs...@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > I've been debugging a load test regression, which turned
> > > > > > > out to
> > > > > > > be
> > > > > > > caused by a client instance leak. The test started failing
> > > > > > > on
> > > > > > > 5.4.4
> > > > > > > due to running out of file descriptors. What I realized is
> > > > > > > that
> > > > > > > leaked
> > > > > > > connection pools are eventually cleaned up by garbage
> > > > > > > collection,
> > > > > > > which causes the sockets' file descriptors to be released.
> > > > > > > What
> > > > > > > has
> > > > > > > changed in 5.4.4 is that this process now takes TWO rounds
> > > > > > > of
> > > > > > > garbage
> > > > > > > collection. This can be seen in the `lsof` output, which
> > > > > > > shows
> > > > > > > where
> > > > > > > in the TCP state machine the socket is:
> > > > > > >
> > > > > > > 5.2: ESTABLISHED -(gc)-> (gone)
> > > > > > > 5.4: ESTABLISHED -(gc)-> CLOSE_WAIT -(gc)-> (gone)
> > > > > > >
> > > > > > > Does anyone know what might have changed that would cause
> > > > > > > this?
> > > > > > > I'm
> > > > > > > specifically asking about the synchronous client (I haven't
> > > > > > > tested
> > > > > > > the
> > > > > > > async client).
> > > > > >
> > > > > > -------------------------------------------------------------
> > > > > > ------
> > > > > > --
> > > > > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > > > > For additional commands, e-mail: dev-help@hc.apache.org
> > > > > >
> > > > >
> > > > >
> > > > > ---------------------------------------------------------------
> > > > > ------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > > > For additional commands, e-mail: dev-help@hc.apache.org
> > > > >
> > > >
> > > >
> > > > -----------------------------------------------------------------
> > > > ----
> > > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail: dev-help@hc.apache.org
> > > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > For additional commands, e-mail: dev-help@hc.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: Change in GC behavior since 5.2.1?
Posted by Oleg Kalnichevski <ol...@apache.org>.
Hi Ryan
I am not sure understand what exactly you are trying to do here? Do you
intentionally use a deprecated method _and_ fail to correctly return
connections back to the pool? Is this the scenario you are trying to
test?
Oleg
On Tue, 2025-06-03 at 14:15 -0700, Ryan Schmitt wrote:
> Furthermore, I can confirm that reverting ee0a10210 fixes the issue.
> No httpcore5 changes are required.
>
> On Tue, Jun 3, 2025 at 2:10 PM Ryan Schmitt <rs...@apache.org>
> wrote:
> >
> > Here's a reproducer:
> >
> > https://github.com/rschmitt/httpcomponents-client/tree/gc-repro
> >
> > The bug is exhibited on JDK11 and JDK21, not on JDK8 or JDK21. I
> > didn't test with any non-LTS releases. Here's JDK17:
> >
> > [0] ~/src/httpcomponents-client # j17 ./run-example.sh
> > SocketLeak
> > This Java version SHOULD exhibit the bug where two garbage
> > collections are required to clean up leaked sockets.
> > Leaking client; sockets should be in ESTABLISHED
> > Tue Jun 3 14:04:40 PDT 2025:
> > Tue Jun 3 14:04:41 PDT 2025: 2 (ESTABLISHED)
> > Tue Jun 3 14:04:42 PDT 2025: 3 (ESTABLISHED)
> > Tue Jun 3 14:04:44 PDT 2025: 3 (ESTABLISHED)
> > Running garbage collector; sockets should be in CLOSE_WAIT
> > Tue Jun 3 14:04:45 PDT 2025: 3 (CLOSE_WAIT)
> > Tue Jun 3 14:04:46 PDT 2025: 3 (CLOSE_WAIT)
> > Tue Jun 3 14:04:47 PDT 2025: 3 (CLOSE_WAIT)
> > Running garbage collector again; no sockets should be listed
> > Tue Jun 3 14:04:48 PDT 2025:
> > Tue Jun 3 14:04:50 PDT 2025:
> >
> > And here's JDK21:
> >
> > [0] ~/src/httpcomponents-client # j21 ./run-example.sh
> > SocketLeak
> > This Java version SHOULD NOT exhibit the bug where two garbage
> > collections are required to clean up leaked sockets.
> > Leaking client; sockets should be in ESTABLISHED
> > Tue Jun 3 14:06:25 PDT 2025:
> > Tue Jun 3 14:06:26 PDT 2025: 3 (ESTABLISHED)
> > Tue Jun 3 14:06:28 PDT 2025: 3 (ESTABLISHED)
> > Tue Jun 3 14:06:29 PDT 2025: 3 (ESTABLISHED)
> > Running garbage collector; no sockets should be listed
> > Tue Jun 3 14:06:30 PDT 2025:
> > Tue Jun 3 14:06:31 PDT 2025:
> >
> > I tested this on both macOS and Linux.
> >
> > On Fri, May 30, 2025 at 5:14 AM Oleg Kalnichevski
> > <ol...@apache.org> wrote:
> > >
> > > On Fri, 2025-05-30 at 11:30 +0200, Oleg Kalnichevski wrote:
> > > > On Thu, 2025-05-29 at 22:54 -0700, Ryan Schmitt wrote:
> > > > > It's the `autoClose` setting. It used to be `true`:
> > > > >
> > > > > https://github.com/apache/httpcomponents-client/blob/d8f702fb4d44c746bb0edf00643aa7139cb8bdf7/httpclient5/src/main/java/org/apache/hc/client5/http/ssl/SSLConnectionSocketFactory.java#L274
> > > > >
> > > > > Now it's `false`:
> > > > >
> > > > > https://github.com/apache/httpcomponents-client/blob/ffe4a05f4faf3706a41ff660bcb3474c6b5101a3/httpclient5/src/main/java/org/apache/hc/client5/http/ssl/AbstractClientTlsStrategy.java#L208
> > > > >
> > > > > This was deliberately changed in this commit:
> > > > >
> > > > > https://github.com/apache/httpcomponents-client/commit/ee0a102104d03a8d1cfe18e571d179c41242182c
> > > > >
> > > >
> > > > This change was introduced in response to HTTPCLIENT-2328 [1].
> > > > I will
> > > > review the code for accidental mistakes. Conceptually the
> > > > `autoClose`
> > > > setting should be merely a convenience.
> > > >
> > > > Oleg
> > > >
> > > >
> > > > [1] https://issues.apache.org/jira/browse/HTTPCLIENT-2328
> > > >
> > >
> > > I think this change in core is more relevant [1]. At the moment I
> > > cannot see anything obviously wrong with it.
> > >
> > > Would you be able to share (contribute) a reproducer for the
> > > defect?
> > >
> > > Oleg
> > >
> > > [1]
> > > https://github.com/apache/httpcomponents-core/commit/1c95b2994f22b91bc7f54cb66ed84a8e94111a61
> > >
> > > >
> > > >
> > > > > On Thu, May 29, 2025 at 6:08 PM Ryan Schmitt
> > > > > <rs...@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > I've been debugging a load test regression, which turned
> > > > > > out to
> > > > > > be
> > > > > > caused by a client instance leak. The test started failing
> > > > > > on
> > > > > > 5.4.4
> > > > > > due to running out of file descriptors. What I realized is
> > > > > > that
> > > > > > leaked
> > > > > > connection pools are eventually cleaned up by garbage
> > > > > > collection,
> > > > > > which causes the sockets' file descriptors to be released.
> > > > > > What
> > > > > > has
> > > > > > changed in 5.4.4 is that this process now takes TWO rounds
> > > > > > of
> > > > > > garbage
> > > > > > collection. This can be seen in the `lsof` output, which
> > > > > > shows
> > > > > > where
> > > > > > in the TCP state machine the socket is:
> > > > > >
> > > > > > 5.2: ESTABLISHED -(gc)-> (gone)
> > > > > > 5.4: ESTABLISHED -(gc)-> CLOSE_WAIT -(gc)-> (gone)
> > > > > >
> > > > > > Does anyone know what might have changed that would cause
> > > > > > this?
> > > > > > I'm
> > > > > > specifically asking about the synchronous client (I haven't
> > > > > > tested
> > > > > > the
> > > > > > async client).
> > > > >
> > > > > -------------------------------------------------------------
> > > > > ------
> > > > > --
> > > > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > > > For additional commands, e-mail: dev-help@hc.apache.org
> > > > >
> > > >
> > > >
> > > > ---------------------------------------------------------------
> > > > ------
> > > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail: dev-help@hc.apache.org
> > > >
> > >
> > >
> > > -----------------------------------------------------------------
> > > ----
> > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: dev-help@hc.apache.org
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: Change in GC behavior since 5.2.1?
Posted by Ryan Schmitt <rs...@apache.org>.
Furthermore, I can confirm that reverting ee0a10210 fixes the issue.
No httpcore5 changes are required.
On Tue, Jun 3, 2025 at 2:10 PM Ryan Schmitt <rs...@apache.org> wrote:
>
> Here's a reproducer:
>
> https://github.com/rschmitt/httpcomponents-client/tree/gc-repro
>
> The bug is exhibited on JDK11 and JDK21, not on JDK8 or JDK21. I
> didn't test with any non-LTS releases. Here's JDK17:
>
> [0] ~/src/httpcomponents-client # j17 ./run-example.sh SocketLeak
> This Java version SHOULD exhibit the bug where two garbage
> collections are required to clean up leaked sockets.
> Leaking client; sockets should be in ESTABLISHED
> Tue Jun 3 14:04:40 PDT 2025:
> Tue Jun 3 14:04:41 PDT 2025: 2 (ESTABLISHED)
> Tue Jun 3 14:04:42 PDT 2025: 3 (ESTABLISHED)
> Tue Jun 3 14:04:44 PDT 2025: 3 (ESTABLISHED)
> Running garbage collector; sockets should be in CLOSE_WAIT
> Tue Jun 3 14:04:45 PDT 2025: 3 (CLOSE_WAIT)
> Tue Jun 3 14:04:46 PDT 2025: 3 (CLOSE_WAIT)
> Tue Jun 3 14:04:47 PDT 2025: 3 (CLOSE_WAIT)
> Running garbage collector again; no sockets should be listed
> Tue Jun 3 14:04:48 PDT 2025:
> Tue Jun 3 14:04:50 PDT 2025:
>
> And here's JDK21:
>
> [0] ~/src/httpcomponents-client # j21 ./run-example.sh SocketLeak
> This Java version SHOULD NOT exhibit the bug where two garbage
> collections are required to clean up leaked sockets.
> Leaking client; sockets should be in ESTABLISHED
> Tue Jun 3 14:06:25 PDT 2025:
> Tue Jun 3 14:06:26 PDT 2025: 3 (ESTABLISHED)
> Tue Jun 3 14:06:28 PDT 2025: 3 (ESTABLISHED)
> Tue Jun 3 14:06:29 PDT 2025: 3 (ESTABLISHED)
> Running garbage collector; no sockets should be listed
> Tue Jun 3 14:06:30 PDT 2025:
> Tue Jun 3 14:06:31 PDT 2025:
>
> I tested this on both macOS and Linux.
>
> On Fri, May 30, 2025 at 5:14 AM Oleg Kalnichevski <ol...@apache.org> wrote:
> >
> > On Fri, 2025-05-30 at 11:30 +0200, Oleg Kalnichevski wrote:
> > > On Thu, 2025-05-29 at 22:54 -0700, Ryan Schmitt wrote:
> > > > It's the `autoClose` setting. It used to be `true`:
> > > >
> > > > https://github.com/apache/httpcomponents-client/blob/d8f702fb4d44c746bb0edf00643aa7139cb8bdf7/httpclient5/src/main/java/org/apache/hc/client5/http/ssl/SSLConnectionSocketFactory.java#L274
> > > >
> > > > Now it's `false`:
> > > >
> > > > https://github.com/apache/httpcomponents-client/blob/ffe4a05f4faf3706a41ff660bcb3474c6b5101a3/httpclient5/src/main/java/org/apache/hc/client5/http/ssl/AbstractClientTlsStrategy.java#L208
> > > >
> > > > This was deliberately changed in this commit:
> > > >
> > > > https://github.com/apache/httpcomponents-client/commit/ee0a102104d03a8d1cfe18e571d179c41242182c
> > > >
> > >
> > > This change was introduced in response to HTTPCLIENT-2328 [1]. I will
> > > review the code for accidental mistakes. Conceptually the `autoClose`
> > > setting should be merely a convenience.
> > >
> > > Oleg
> > >
> > >
> > > [1] https://issues.apache.org/jira/browse/HTTPCLIENT-2328
> > >
> >
> > I think this change in core is more relevant [1]. At the moment I
> > cannot see anything obviously wrong with it.
> >
> > Would you be able to share (contribute) a reproducer for the defect?
> >
> > Oleg
> >
> > [1]
> > https://github.com/apache/httpcomponents-core/commit/1c95b2994f22b91bc7f54cb66ed84a8e94111a61
> >
> > >
> > >
> > > > On Thu, May 29, 2025 at 6:08 PM Ryan Schmitt <rs...@apache.org>
> > > > wrote:
> > > > >
> > > > > I've been debugging a load test regression, which turned out to
> > > > > be
> > > > > caused by a client instance leak. The test started failing on
> > > > > 5.4.4
> > > > > due to running out of file descriptors. What I realized is that
> > > > > leaked
> > > > > connection pools are eventually cleaned up by garbage collection,
> > > > > which causes the sockets' file descriptors to be released. What
> > > > > has
> > > > > changed in 5.4.4 is that this process now takes TWO rounds of
> > > > > garbage
> > > > > collection. This can be seen in the `lsof` output, which shows
> > > > > where
> > > > > in the TCP state machine the socket is:
> > > > >
> > > > > 5.2: ESTABLISHED -(gc)-> (gone)
> > > > > 5.4: ESTABLISHED -(gc)-> CLOSE_WAIT -(gc)-> (gone)
> > > > >
> > > > > Does anyone know what might have changed that would cause this?
> > > > > I'm
> > > > > specifically asking about the synchronous client (I haven't
> > > > > tested
> > > > > the
> > > > > async client).
> > > >
> > > > -------------------------------------------------------------------
> > > > --
> > > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail: dev-help@hc.apache.org
> > > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: dev-help@hc.apache.org
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > For additional commands, e-mail: dev-help@hc.apache.org
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: Change in GC behavior since 5.2.1?
Posted by Ryan Schmitt <rs...@apache.org>.
Here's a reproducer:
https://github.com/rschmitt/httpcomponents-client/tree/gc-repro
The bug is exhibited on JDK11 and JDK21, not on JDK8 or JDK21. I
didn't test with any non-LTS releases. Here's JDK17:
[0] ~/src/httpcomponents-client # j17 ./run-example.sh SocketLeak
This Java version SHOULD exhibit the bug where two garbage
collections are required to clean up leaked sockets.
Leaking client; sockets should be in ESTABLISHED
Tue Jun 3 14:04:40 PDT 2025:
Tue Jun 3 14:04:41 PDT 2025: 2 (ESTABLISHED)
Tue Jun 3 14:04:42 PDT 2025: 3 (ESTABLISHED)
Tue Jun 3 14:04:44 PDT 2025: 3 (ESTABLISHED)
Running garbage collector; sockets should be in CLOSE_WAIT
Tue Jun 3 14:04:45 PDT 2025: 3 (CLOSE_WAIT)
Tue Jun 3 14:04:46 PDT 2025: 3 (CLOSE_WAIT)
Tue Jun 3 14:04:47 PDT 2025: 3 (CLOSE_WAIT)
Running garbage collector again; no sockets should be listed
Tue Jun 3 14:04:48 PDT 2025:
Tue Jun 3 14:04:50 PDT 2025:
And here's JDK21:
[0] ~/src/httpcomponents-client # j21 ./run-example.sh SocketLeak
This Java version SHOULD NOT exhibit the bug where two garbage
collections are required to clean up leaked sockets.
Leaking client; sockets should be in ESTABLISHED
Tue Jun 3 14:06:25 PDT 2025:
Tue Jun 3 14:06:26 PDT 2025: 3 (ESTABLISHED)
Tue Jun 3 14:06:28 PDT 2025: 3 (ESTABLISHED)
Tue Jun 3 14:06:29 PDT 2025: 3 (ESTABLISHED)
Running garbage collector; no sockets should be listed
Tue Jun 3 14:06:30 PDT 2025:
Tue Jun 3 14:06:31 PDT 2025:
I tested this on both macOS and Linux.
On Fri, May 30, 2025 at 5:14 AM Oleg Kalnichevski <ol...@apache.org> wrote:
>
> On Fri, 2025-05-30 at 11:30 +0200, Oleg Kalnichevski wrote:
> > On Thu, 2025-05-29 at 22:54 -0700, Ryan Schmitt wrote:
> > > It's the `autoClose` setting. It used to be `true`:
> > >
> > > https://github.com/apache/httpcomponents-client/blob/d8f702fb4d44c746bb0edf00643aa7139cb8bdf7/httpclient5/src/main/java/org/apache/hc/client5/http/ssl/SSLConnectionSocketFactory.java#L274
> > >
> > > Now it's `false`:
> > >
> > > https://github.com/apache/httpcomponents-client/blob/ffe4a05f4faf3706a41ff660bcb3474c6b5101a3/httpclient5/src/main/java/org/apache/hc/client5/http/ssl/AbstractClientTlsStrategy.java#L208
> > >
> > > This was deliberately changed in this commit:
> > >
> > > https://github.com/apache/httpcomponents-client/commit/ee0a102104d03a8d1cfe18e571d179c41242182c
> > >
> >
> > This change was introduced in response to HTTPCLIENT-2328 [1]. I will
> > review the code for accidental mistakes. Conceptually the `autoClose`
> > setting should be merely a convenience.
> >
> > Oleg
> >
> >
> > [1] https://issues.apache.org/jira/browse/HTTPCLIENT-2328
> >
>
> I think this change in core is more relevant [1]. At the moment I
> cannot see anything obviously wrong with it.
>
> Would you be able to share (contribute) a reproducer for the defect?
>
> Oleg
>
> [1]
> https://github.com/apache/httpcomponents-core/commit/1c95b2994f22b91bc7f54cb66ed84a8e94111a61
>
> >
> >
> > > On Thu, May 29, 2025 at 6:08 PM Ryan Schmitt <rs...@apache.org>
> > > wrote:
> > > >
> > > > I've been debugging a load test regression, which turned out to
> > > > be
> > > > caused by a client instance leak. The test started failing on
> > > > 5.4.4
> > > > due to running out of file descriptors. What I realized is that
> > > > leaked
> > > > connection pools are eventually cleaned up by garbage collection,
> > > > which causes the sockets' file descriptors to be released. What
> > > > has
> > > > changed in 5.4.4 is that this process now takes TWO rounds of
> > > > garbage
> > > > collection. This can be seen in the `lsof` output, which shows
> > > > where
> > > > in the TCP state machine the socket is:
> > > >
> > > > 5.2: ESTABLISHED -(gc)-> (gone)
> > > > 5.4: ESTABLISHED -(gc)-> CLOSE_WAIT -(gc)-> (gone)
> > > >
> > > > Does anyone know what might have changed that would cause this?
> > > > I'm
> > > > specifically asking about the synchronous client (I haven't
> > > > tested
> > > > the
> > > > async client).
> > >
> > > -------------------------------------------------------------------
> > > --
> > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: dev-help@hc.apache.org
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > For additional commands, e-mail: dev-help@hc.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: Change in GC behavior since 5.2.1?
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Fri, 2025-05-30 at 11:30 +0200, Oleg Kalnichevski wrote:
> On Thu, 2025-05-29 at 22:54 -0700, Ryan Schmitt wrote:
> > It's the `autoClose` setting. It used to be `true`:
> >
> > https://github.com/apache/httpcomponents-client/blob/d8f702fb4d44c746bb0edf00643aa7139cb8bdf7/httpclient5/src/main/java/org/apache/hc/client5/http/ssl/SSLConnectionSocketFactory.java#L274
> >
> > Now it's `false`:
> >
> > https://github.com/apache/httpcomponents-client/blob/ffe4a05f4faf3706a41ff660bcb3474c6b5101a3/httpclient5/src/main/java/org/apache/hc/client5/http/ssl/AbstractClientTlsStrategy.java#L208
> >
> > This was deliberately changed in this commit:
> >
> > https://github.com/apache/httpcomponents-client/commit/ee0a102104d03a8d1cfe18e571d179c41242182c
> >
>
> This change was introduced in response to HTTPCLIENT-2328 [1]. I will
> review the code for accidental mistakes. Conceptually the `autoClose`
> setting should be merely a convenience.
>
> Oleg
>
>
> [1] https://issues.apache.org/jira/browse/HTTPCLIENT-2328
>
I think this change in core is more relevant [1]. At the moment I
cannot see anything obviously wrong with it.
Would you be able to share (contribute) a reproducer for the defect?
Oleg
[1]
https://github.com/apache/httpcomponents-core/commit/1c95b2994f22b91bc7f54cb66ed84a8e94111a61
>
>
> > On Thu, May 29, 2025 at 6:08 PM Ryan Schmitt <rs...@apache.org>
> > wrote:
> > >
> > > I've been debugging a load test regression, which turned out to
> > > be
> > > caused by a client instance leak. The test started failing on
> > > 5.4.4
> > > due to running out of file descriptors. What I realized is that
> > > leaked
> > > connection pools are eventually cleaned up by garbage collection,
> > > which causes the sockets' file descriptors to be released. What
> > > has
> > > changed in 5.4.4 is that this process now takes TWO rounds of
> > > garbage
> > > collection. This can be seen in the `lsof` output, which shows
> > > where
> > > in the TCP state machine the socket is:
> > >
> > > 5.2: ESTABLISHED -(gc)-> (gone)
> > > 5.4: ESTABLISHED -(gc)-> CLOSE_WAIT -(gc)-> (gone)
> > >
> > > Does anyone know what might have changed that would cause this?
> > > I'm
> > > specifically asking about the synchronous client (I haven't
> > > tested
> > > the
> > > async client).
> >
> > -------------------------------------------------------------------
> > --
> > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > For additional commands, e-mail: dev-help@hc.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: Change in GC behavior since 5.2.1?
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Thu, 2025-05-29 at 22:54 -0700, Ryan Schmitt wrote:
> It's the `autoClose` setting. It used to be `true`:
>
> https://github.com/apache/httpcomponents-client/blob/d8f702fb4d44c746bb0edf00643aa7139cb8bdf7/httpclient5/src/main/java/org/apache/hc/client5/http/ssl/SSLConnectionSocketFactory.java#L274
>
> Now it's `false`:
>
> https://github.com/apache/httpcomponents-client/blob/ffe4a05f4faf3706a41ff660bcb3474c6b5101a3/httpclient5/src/main/java/org/apache/hc/client5/http/ssl/AbstractClientTlsStrategy.java#L208
>
> This was deliberately changed in this commit:
>
> https://github.com/apache/httpcomponents-client/commit/ee0a102104d03a8d1cfe18e571d179c41242182c
>
This change was introduced in response to HTTPCLIENT-2328 [1]. I will
review the code for accidental mistakes. Conceptually the `autoClose`
setting should be merely a convenience.
Oleg
[1] https://issues.apache.org/jira/browse/HTTPCLIENT-2328
> On Thu, May 29, 2025 at 6:08 PM Ryan Schmitt <rs...@apache.org>
> wrote:
> >
> > I've been debugging a load test regression, which turned out to be
> > caused by a client instance leak. The test started failing on 5.4.4
> > due to running out of file descriptors. What I realized is that
> > leaked
> > connection pools are eventually cleaned up by garbage collection,
> > which causes the sockets' file descriptors to be released. What has
> > changed in 5.4.4 is that this process now takes TWO rounds of
> > garbage
> > collection. This can be seen in the `lsof` output, which shows
> > where
> > in the TCP state machine the socket is:
> >
> > 5.2: ESTABLISHED -(gc)-> (gone)
> > 5.4: ESTABLISHED -(gc)-> CLOSE_WAIT -(gc)-> (gone)
> >
> > Does anyone know what might have changed that would cause this? I'm
> > specifically asking about the synchronous client (I haven't tested
> > the
> > async client).
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: Change in GC behavior since 5.2.1?
Posted by Ryan Schmitt <rs...@apache.org>.
It's the `autoClose` setting. It used to be `true`:
https://github.com/apache/httpcomponents-client/blob/d8f702fb4d44c746bb0edf00643aa7139cb8bdf7/httpclient5/src/main/java/org/apache/hc/client5/http/ssl/SSLConnectionSocketFactory.java#L274
Now it's `false`:
https://github.com/apache/httpcomponents-client/blob/ffe4a05f4faf3706a41ff660bcb3474c6b5101a3/httpclient5/src/main/java/org/apache/hc/client5/http/ssl/AbstractClientTlsStrategy.java#L208
This was deliberately changed in this commit:
https://github.com/apache/httpcomponents-client/commit/ee0a102104d03a8d1cfe18e571d179c41242182c
On Thu, May 29, 2025 at 6:08 PM Ryan Schmitt <rs...@apache.org> wrote:
>
> I've been debugging a load test regression, which turned out to be
> caused by a client instance leak. The test started failing on 5.4.4
> due to running out of file descriptors. What I realized is that leaked
> connection pools are eventually cleaned up by garbage collection,
> which causes the sockets' file descriptors to be released. What has
> changed in 5.4.4 is that this process now takes TWO rounds of garbage
> collection. This can be seen in the `lsof` output, which shows where
> in the TCP state machine the socket is:
>
> 5.2: ESTABLISHED -(gc)-> (gone)
> 5.4: ESTABLISHED -(gc)-> CLOSE_WAIT -(gc)-> (gone)
>
> Does anyone know what might have changed that would cause this? I'm
> specifically asking about the synchronous client (I haven't tested the
> async client).
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org