You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hc.apache.org by Oleg Kalnichevski <ol...@apache.org> on 2025/12/18 12:35:42 UTC
Address shadowing; was Re: [VOTE] Release HttpClient 5.6 based on RC1
Hi Ryan
Thank you so much for getting to the bottom of this problem. I, like
Arturo, have never seen anything of this sort on my Linus box. I have
been using various Debian versions for a long time.
Can this be by any chance MacOS specific?
Oleg
On Wed, 2025-12-17 at 18:27 -0800, Ryan Schmitt wrote:
> I ran this test using `@RepeatedTest(1000)` (handy dandy!) and
> managed to
> reproduce many different failures, all using JDK11 and macOS 26.2
> (aarch64). In Wireshark I'm seeing variations on two themes: (1) the
> server
> doesn't respond to the Client Hello, (2) the server responds to the
> Client
> Hello with a plaintext HTTP response (I saw both 404 Not Found and
> 501 Not
> Implemented).
>
> I think what's going on is a bug in how the test server uses
> ephemeral port
> bindings. From `netstat -nap tcp` and `lsof -P -i tcp` output, I can
> see
> that IntelliJ is listening on port 60677 for connections from its
> `compile-server`, with a persistent TCP connection established;
> however,
> while running the tests, the I saw the test server successfully bind
> to
> this port:
>
> tcp46 0 0 *.60677 *.*
> LISTEN
> <-- test server
> tcp4 0 0 127.0.0.1.60677 *.*
> LISTEN
> <-- intellij
>
> Similarly, on another test run I saw port reuse occur in such a way
> that
> the client tried to connect to the `jetbrains-toolbox` process
> (tcp4 127.0.0.1.49496), which returned plaintext `404 Not Found`
> resulting
> in:
>
> java.util.concurrent.ExecutionException: javax.net.ssl.SSLException:
> Unrecognized SSL message, plaintext connection?
>
> The reason this happens is that our test server binds to the wildcard
> address (IPv4 `0.0.0.0` or IPv6 `::`), not a specific loopback
> address.
> Crucially, other sockets can still bind to that same port, as long as
> they
> do so at a specific address; the resulting socket will effectively
> "shadow"
> the 0.0.0.0 socket for that particular address. I tested this on my
> machine
> and it is indeed very cool and very legal:
>
> tcp4 0 0 127.0.0.2.12345 *.*
> LISTEN
> tcp4 0 0 127.0.0.1.12345 *.*
> LISTEN
> tcp6 0 0 ::1.12345 *.*
> LISTEN
> tcp46 0 0 *.12345 *.*
> LISTEN
>
> It even works when setting `-Djava.net.preferIPv4Stack=true`, which
> prevents Java from creating a dual-stack (`tcp46`) socket:
>
> tcp4 0 0 127.0.0.2.12345 *.*
> LISTEN
> tcp4 0 0 127.0.0.1.12345 *.*
> LISTEN
> tcp4 0 0 *.12345 *.*
> LISTEN
>
> I've been seeing occasional test failures on macOS for years with
> absolutely no pattern to them, and I think this is the root cause.
>
> The good news is that this isn't a regression or a release blocker.
> The bad
> news is that we have to fix a bunch of test code in both core and
> client.
> Any test that binds a server to `new InetSocketAddress(0)` (which
> implicitly binds to the wildcard address) is liable to randomly fail
> due to
> address shadowing. Our test servers should _always_ explicitly bind
> to
> `InetAddress.getLoopbackAddress()` or a hardcoded loopback address.
>
> On Wed, Dec 17, 2025 at 7:52 AM Gary Gregory <ga...@gmail.com>
> wrote:
>
> > I've seen the same random failure again (I don't know how many
> > build
> > loops ran), still on Oracle Java 1.8.0_471:
> >
> > [INFO] Running
> > org.apache.hc.client5.testing.async.HttpIntegrationTests$RedirectsH
> > 2Tls
> > [ERROR] Tests run: 20, Failures: 0, Errors: 1, Skipped: 0, Time
> > elapsed: 19.97 s <<< FAILURE! -- in
> > org.apache.hc.client5.testing.async.HttpIntegrationTests$RedirectsH
> > 2Tls
> > [ERROR]
> > org.apache.hc.client5.testing.async.HttpIntegrationTests$RedirectsH
> > 2Tls.testRelativeRedirect
> > -- Time elapsed: 19.80 s <<< ERROR!
> > java.util.concurrent.ExecutionException:
> > org.apache.hc.core5.http2.impl.nio.ProtocolNegotiationException:
> > Unexpected input
> > at
> > org.apache.hc.core5.concurrent.BasicFuture.getResult(BasicFuture.ja
> > va:84)
> > at
> > org.apache.hc.core5.concurrent.BasicFuture.get(BasicFuture.java:99)
> > at
> > org.apache.hc.client5.testing.async.AbstractHttpAsyncRedirectsTest.
> > testRelativeRedirect(AbstractHttpAsyncRedirectsTest.java:442)
> > at java.lang.reflect.Method.invoke(Method.java:498)
> > at java.util.ArrayList.forEach(ArrayList.java:1259)
> > at java.util.ArrayList.forEach(ArrayList.java:1259)
> > at java.util.ArrayList.forEach(ArrayList.java:1259)
> > Caused by:
> > org.apache.hc.core5.http2.impl.nio.ProtocolNegotiationException:
> > Unexpected input
> > at
> > org.apache.hc.core5.http2.impl.nio.ClientH2PrefaceHandler.inputRead
> > y(ClientH2PrefaceHandler.java:165)
> > at
> > org.apache.hc.core5.reactor.ssl.SSLIOSession.decryptData(SSLIOSessi
> > on.java:643)
> > at
> > org.apache.hc.core5.reactor.ssl.SSLIOSession.access$200(SSLIOSessio
> > n.java:74)
> > at
> > org.apache.hc.core5.reactor.ssl.SSLIOSession$1.inputReady(SSLIOSess
> > ion.java:204)
> > at
> > org.apache.hc.core5.reactor.InternalDataChannel.onIOEvent(InternalD
> > ataChannel.java:139)
> > at
> > org.apache.hc.core5.reactor.InternalChannel.handleIOEvent(InternalC
> > hannel.java:51)
> > at
> > org.apache.hc.core5.reactor.SingleCoreIOReactor.processEvents(Singl
> > eCoreIOReactor.java:193)
> > at
> > org.apache.hc.core5.reactor.SingleCoreIOReactor.doExecute(SingleCor
> > eIOReactor.java:140)
> > at
> > org.apache.hc.core5.reactor.AbstractSingleCoreIOReactor.execute(Abs
> > tractSingleCoreIOReactor.java:92)
> > at
> > org.apache.hc.core5.reactor.IOReactorWorker.run(IOReactorWorker.jav
> > a:44)
> > at java.lang.Thread.run(Thread.java:750)
> >
> > Gary
> >
> > On Tue, Dec 16, 2025 at 5:54 PM Gary Gregory
> > <ga...@gmail.com>
> > wrote:
> > >
> > > +1
> > >
> > > Note that I got a random failure on Java 11:
> > >
> > > [INFO] Running
> > org.apache.hc.client5.testing.async.HttpIntegrationTests$Authentica
> > tionH2Tls
> > > [ERROR] Tests run: 12, Failures: 0, Errors: 1, Skipped: 0, Time
> > > elapsed:
> > 20.44 s <<< FAILURE! -- in
> > org.apache.hc.client5.testing.async.HttpIntegrationTests$Authentica
> > tionH2Tls
> > > [ERROR]
> > org.apache.hc.client5.testing.async.HttpIntegrationTests$Authentica
> > tionH2Tls.testBasicAuthenticationFailure
> > -- Time elapsed: 20.21 s <<< ERROR!
> > > java.util.concurrent.ExecutionException:
> > org.apache.hc.core5.http2.impl.nio.ProtocolNegotiationException:
> > Unexpected
> > input
> > > at
> > org.apache.hc.core5.concurrent.BasicFuture.getResult(BasicFuture.ja
> > va:84)
> > > at
> > > org.apache.hc.core5.concurrent.BasicFuture.get(BasicFuture.java:9
> > > 9)
> > > at
> > org.apache.hc.client5.testing.async.AbstractHttpAsyncClientAuthenti
> > cationTest.testBasicAuthenticationFailure(AbstractHttpAsyncClientAu
> > thenticationTest.java:142)
> > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > > at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
> > > at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
> > > at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
> > > Caused by:
> > org.apache.hc.core5.http2.impl.nio.ProtocolNegotiationException:
> > Unexpected
> > input
> > > at
> > org.apache.hc.core5.http2.impl.nio.ClientH2PrefaceHandler.inputRead
> > y(ClientH2PrefaceHandler.java:165)
> > > at
> > org.apache.hc.core5.reactor.ssl.SSLIOSession.decryptData(SSLIOSessi
> > on.java:643)
> > > at
> > org.apache.hc.core5.reactor.ssl.SSLIOSession.access$200(SSLIOSessio
> > n.java:74)
> > > at
> > org.apache.hc.core5.reactor.ssl.SSLIOSession$1.inputReady(SSLIOSess
> > ion.java:204)
> > > at
> > org.apache.hc.core5.reactor.InternalDataChannel.onIOEvent(InternalD
> > ataChannel.java:139)
> > > at
> > org.apache.hc.core5.reactor.InternalChannel.handleIOEvent(InternalC
> > hannel.java:51)
> > > at
> > org.apache.hc.core5.reactor.SingleCoreIOReactor.processEvents(Singl
> > eCoreIOReactor.java:193)
> > > at
> > org.apache.hc.core5.reactor.SingleCoreIOReactor.doExecute(SingleCor
> > eIOReactor.java:140)
> > > at
> > org.apache.hc.core5.reactor.AbstractSingleCoreIOReactor.execute(Abs
> > tractSingleCoreIOReactor.java:92)
> > > at
> > org.apache.hc.core5.reactor.IOReactorWorker.run(IOReactorWorker.jav
> > a:44)
> > > at java.base/java.lang.Thread.run(Thread.java:829)
> > >
> > > I tested the source zip file.
> > > - ASC OK
> > > - SHA512 OK
> > > - Building using the default Maven goal: mvn -P 'use-toolchains'
> > > - Java LTS versions tested: 25, 21, 17, 11, 8:
> > >
> > > java version "1.8.0_471"
> > > Java(TM) SE Runtime Environment (build 1.8.0_471-b09)
> > > Java HotSpot(TM) 64-Bit Server VM (build 25.471-b09, mixed mode)
> > >
> > > openjdk version "11.0.29" 2025-10-21
> > > OpenJDK Runtime Environment Homebrew (build 11.0.29+0)
> > > OpenJDK 64-Bit Server VM Homebrew (build 11.0.29+0, mixed mode)
> > >
> > > openjdk version "17.0.17" 2025-10-21
> > > OpenJDK Runtime Environment Homebrew (build 17.0.17+0)
> > > OpenJDK 64-Bit Server VM Homebrew (build 17.0.17+0, mixed mode,
> > > sharing)
> > >
> > > openjdk version "21.0.9" 2025-10-21
> > > OpenJDK Runtime Environment Homebrew (build 21.0.9)
> > > OpenJDK 64-Bit Server VM Homebrew (build 21.0.9, mixed mode,
> > > sharing)
> > >
> > > openjdk version "25.0.1" 2025-10-21
> > > OpenJDK Runtime Environment Homebrew (build 25.0.1)
> > > OpenJDK 64-Bit Server VM Homebrew (build 25.0.1, mixed mode,
> > > sharing)
> > >
> > > And:
> > >
> > > Apache Maven 3.9.11 (3e54c93a704957b63ee3494413a2b544fd3d825b)
> > > Maven home: /opt/homebrew/Cellar/maven/3.9.11/libexec
> > > Java version: 25.0.1, vendor: Homebrew, runtime:
> > > /opt/homebrew/Cellar/openjdk/25.0.1/libexec/openjdk.jdk/Contents/
> > > Home
> > > Default locale: en_US, platform encoding: UTF-8
> > > OS name: "mac os x", version: "26.2", arch: "aarch64", family:
> > > "mac"
> > >
> > > Darwin ****-MacBook-Pro.local 25.2.0 Darwin Kernel Version
> > > 25.2.0: Tue
> > > Nov 18 21:09:56 PST 2025; root:xnu-
> > > 12377.61.12~1/RELEASE_ARM64_T6041
> > > arm64
> > >
> > > Docker version 28.4.0, build d8eb465
> > >
> > >
> > > On Tue, Dec 16, 2025 at 3:22 PM Oleg Kalnichevski
> > > <ol...@apache.org>
> > wrote:
> > > >
> > > > Please vote on releasing these packages as HttpClient 5.6.
> > > > The vote is open for the at least 72 hours, and only votes from
> > > > HttpComponents PMC members are binding. The vote passes if at
> > > > least
> > > > three binding +1 votes are cast and there are more +1 than -1
> > > > votes.
> > > >
> > > > Release notes:
> > > >
> > https://dist.apache.org/repos/dist/dev/httpcomponents/httpclient-5.6-RC1/RELEASE_NOTES-5.6.x.txt
> > > >
> > > > Maven artefacts:
> > > >
> > https://repository.apache.org/content/repositories/orgapachehttpcomponents-1200/org/apache/httpcomponents/client5/
> > > >
> > > > Git Tag: 5.6-RC1
> > > > https://github.com/apache/httpcomponents-client/tree/5.6-RC1
> > > >
> > > > Packages:
> > > >
> > https://dist.apache.org/repos/dist/dev/httpcomponents/httpclient-5.6-RC1
> > > > revision 81440
> > > >
> > > > Hashes:
> > > >
> > 391ade1e3a2c725fe8ebf2c8a35f127d2824de85a77fca8317b3638b04edc29f17c
> > d174fa78e32435b8c222fe8dc35128d9cf88229dd2f3c10f6719650ba504c
> > *httpcomponents-client-5.6-bin.zip httpcomponents-client-5.6-
> > bin.zip
> > > >
> > 102f183ce805450a731556dcac9ab46eeb0cb06aff77ed027f7c96be5ac92bd7f53
> > 05c7d25d24bb0898d874700252e45f151a0419e9406b0f11f26937f45ee2c
> > *httpcomponents-client-5.6-bin.tar.gz httpcomponents-client-5.6-
> > bin.tar.gz
> > > >
> > c7209f76a23a68387a74a789bf91f8ed67beec7eea2e33b9e09b429ba1e05b4a432
> > 1677a79cfad2a75bccf195a1bc760ce459a631d7b366a2a838f2018f4323a
> > *httpcomponents-client-5.6-src.zip httpcomponents-client-5.6-
> > src.zip
> > > >
> > afe142f43aca91bac37aad46214f6a1ce37bdc050dac81f07397c25b6a4109c658e
> > 57fb27d6b060da17ca28adbea5b2012621a22991f0edfbe176f53c8d1ce40
> > *httpcomponents-client-5.6-src.tar.gz httpcomponents-client-5.6-
> > src.tar.gz
> > > >
> > > > Keys:
> > > > https://www.apache.org/dist/httpcomponents/httpclient/KEYS
> > > >
> > > >
> > -------------------------------------------------------------------
> > -------
> > > > Vote: HttpClient 5.6 release
> > > > [ ] +1 Release the packages as HttpClient 5.6.
> > > > [ ] -1 I am against releasing the packages (must include a
> > > > reason).
> > > >
> > > > ---------------------------------------------------------------
> > > > ------
> > > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail: dev-help@hc.apache.org
> > > >
> >
> > -------------------------------------------------------------------
> > --
> > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > For additional commands, e-mail: dev-help@hc.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org