[PR] [improve][misc] Upgrade Conscrypt to 2.6.1 to add aarch64 native support [pulsar]

["lhotari (via GitHub)" <gi...@apache.org>]

lhotari opened a new pull request, #26314:
URL: https://github.com/apache/pulsar/pull/26314

   ### Motivation
   
   `org.conscrypt:conscrypt-openjdk-uber` has been pinned at 2.5.2, which ships natives for only four platforms and **none for aarch64**:
   
   ```
   2.5.2: conscrypt_openjdk_jni-windows-x86.dll
          conscrypt_openjdk_jni-windows-x86_64.dll
          libconscrypt_openjdk_jni-linux-x86_64.so
          libconscrypt_openjdk_jni-osx-x86_64.dylib
   ```
   
   This matters more than it looks, because Conscrypt is a **shipped default**, not an opt-in: `webServiceTlsProvider=Conscrypt` in `conf/broker.conf`, `conf/proxy.conf` and `conf/standalone.conf`, and `tlsProvider=Conscrypt` in `conf/websocket.conf`.
   
   On ARM64 that default has never been honoured. `Conscrypt.checkAvailability()` throws `UnsatisfiedLinkError`, the provider is never registered, and `SecurityUtility.resolveProvider` silently substitutes the JDK default:
   
   ```java
   provider = Security.getProvider(providerName);   // null - Conscrypt never registered
   if (provider == null) {
       provider = SSLContext.getDefault().getProvider();
   }
   ```
   
   The fallback is logged at `DEBUG` only, so ARM64 operators get the JDK JSSE provider while their configuration says Conscrypt, with no indication anything was substituted.
   
   Conscrypt 2.6.0 adds `linux-aarch_64` and `osx-aarch_64` natives, which fixes this. Two secondary reasons to move now:
   
   - **Jetty already expects 2.6.0.** `jetty-project:12.1.12` declares `<conscrypt.version>2.6.0</conscrypt.version>`, so our catalog pin was *downgrading* Jetty's dependency to 2.5.2. After this change it upgrades it to 2.6.1 instead.
   - **The `google/conscrypt#1015` workaround becomes dead code.** Conscrypt 2.6.0 includes google/conscrypt#1060, which fixes that bug upstream.
   
   ### Modifications
   
   **1. Bump the version catalog** — `conscrypt = "2.5.2"` → `"2.6.1"`, plus the jar name in both `LICENSE.bin.txt` files (server and shell use different naming conventions, both updated).
   
   **2. Remove the obsolete `google/conscrypt#1015` workaround.** Pulsar sets a process-global CN-tolerant hostname verifier via `Conscrypt.setDefaultHostnameVerifier(...)`, but under 2.5.2 `TrustManagerImpl` ignored it, so Pulsar had to copy it onto every `TrustManager` instance by hand. google/conscrypt#1060 added the missing fallback. Comparing `TrustManagerImpl.getHttpsVerifier()` — the sole consumer of that field — across the two jars:
   
   | | 2.5.2 | 2.6.1 |
   |---|---|---|
   | `getHttpsVerifier()` | `hostnameVerifier` → `Platform.getDefaultHostnameVerifier()` | `hostnameVerifier` → **`defaultHostnameVerifier`** → `Platform.getDefaultHostnameVerifier()` |
   
   2.5.2 never reads the static `defaultHostnameVerifier` field; 2.6.1 does. So `processConscryptTrustManager(s)` is removed from both `SecurityUtility` (the wired PIP-337 path) and `JcaProviders` (the PIP-478 path), along with the two call sites in `KeyStoreSSLContext` and `JdkSslContexts`. The `setDefaultHostnameVerifier` call itself is **kept** — that is what installs the verifier; only the redundant per-instance propagation is gone.
   
   Net: **+18 / −103**.
   
   ### Verifying this change
   
   - [x] Make sure that the change passes the CI checks.
   
   This change is already covered by existing tests — 138 TLS tests in `pulsar-common` (`JcaProvidersTest`, `JdkSslContextsTest`, `JcaProviderPinningTest`, `DefaultPulsarSslFactoryTest`, `FileBasedTlsFactoryTest`, keystore-TLS) and 37 in `pulsar-broker-common`, all green before and after the removal. `./gradlew sanityCheck` is green across all 582 tasks, which also confirms nothing outside `pulsar-common` referenced the removed `public` method.
   
   `./gradlew checkBinaryLicense` passes for both distributions. As a negative control, reverting just the server `LICENSE.bin.txt` line to 2.5.2 fails as expected — which confirms both that the check is real and that 2.6.1 is the version actually resolved and bundled:
   
   ```
   org.conscrypt-conscrypt-openjdk-uber-2.6.1.jar unaccounted for in LICENSE
   org.conscrypt-conscrypt-openjdk-uber-2.5.2.jar mentioned in LICENSE, but not bundled
   ```
   
   The API surface was verified rather than assumed: Pulsar reaches Conscrypt **reflectively**, so the compiler cannot catch breakage. All reflectively-referenced members still resolve in 2.6.1 with identical signatures (`checkAvailability`, `wrapHostnameVerifier`, `set`/`getDefaultHostnameVerifier`, `set`/`getHostnameVerifier`, plus `OpenSSLProvider` and `TrustManagerImpl`).
   
   The `#1015` fix was also confirmed at runtime — same JVM, only the jar swapped, workaround deliberately not applied:
   
   ```
   conscrypt 2.5.2 -> EFFECTIVE getHttpsVerifier() = org.conscrypt.OkHostnameVerifier@32e6e9c3
                      default verifier honoured WITHOUT workaround = false
   conscrypt 2.6.1 -> EFFECTIVE getHttpsVerifier() = PULSAR_CUSTOM_VERIFIER
                      default verifier honoured WITHOUT workaround = true
   ```
   
   Native loading was smoke-tested in Pulsar's actual container base images (load + `SSLContext` + TLSv1.3):
   
   | Image | 2.5.2 | 2.6.1 |
   |---|---|---|
   | Alpine arm64 (musl + `gcompat`) | `UnsatisfiedLinkError` | OK |
   | Alpine x86_64 (musl + `gcompat`) | OK | OK |
   | Wolfi arm64 (glibc 2.43) | not run (no aarch64 native in 2.5.2) | OK |
   | Ubuntu 20.04 x86_64 (glibc 2.31) | OK | `GLIBC_2.35' not found` |
   
   ### Notes for reviewers
   
   Three consequences of the upgrade that are worth an explicit decision:
   
   **1. glibc 2.35 floor on `linux-x86_64` (tarball installs only).** The 2.6.1 `linux-x86_64` native references `_dl_find_object@GLIBC_2.35` (2.5.2 topped out at 2.14). On glibc < 2.35 hosts — Ubuntu 20.04, Debian 11, RHEL/CentOS 8, Amazon Linux 2 — Conscrypt now fails to load and TLS falls back to the JDK provider. This is a **silent performance degradation, not an outage**: the fallback shown in the Motivation section keeps TLS working, and the failure is logged at `DEBUG`. Worth a release note.
   
   Both shipped Docker images are unaffected: Wolfi has glibc 2.43, and the Alpine image already installs `gcompat` with `LD_PRELOAD=/lib/libgcompat.so.0`, under which the native loads fine on both architectures (verified above). The `linux-aarch_64` native tops out at `GLIBC_2.34`, so the floor is x86_64-only.
   
   **2. Hostname verification becomes more lenient in code paths Pulsar does not own.** This follows from the upgrade itself, not from the workaround removal. `setDefaultHostnameVerifier` is process-global. Under 2.5.2 that global was inert, so Pulsar's CN-tolerant `TlsHostnameVerifier` only reached the `TrustManager`s Pulsar explicitly post-processed. Under 2.6.1 *every* Conscrypt `TrustManagerImpl` in the JVM falls back to it — including ones created by Jetty, connectors or offloaders, which previously got Conscrypt's stricter SAN-only `OkHostnameVerifier`. Removing the workaround is behaviour-neutral here; keeping it would not prevent the widening. Flagging it because it runs opposite to PIP-478's move to SAN-only verification, so it may deserve a follow-up.
   
   **3. Smaller packaging notes.** The uber jar grows ~9.7 MB → ~14 MB (`docker/pulsar/build-scripts/remove-unnecessary-native-binaries.sh` prunes netty-tcnative and RocksDB but not Conscrypt — could be a follow-up). 32-bit Windows (`windows-x86`) loses Conscrypt, as 2.6.0 dropped that native. 2.5.2 was a signed jar and 2.6.1 is not. The bytecode floor rises from Java 7 to Java 8, well under Pulsar's minimum. No new license obligations: neither jar bundles a `META-INF/LICENSE` or `NOTICE`, and BoringSSL was already statically embedded in 2.5.2.
   
   ### Does this pull request potentially affect one of the following parts:
   
   *If the box was checked, please highlight the changes*
   
   - [x] Dependencies (add or upgrade a dependency)
   - [ ] The public API
   - [ ] The schema
   - [ ] The default values of configurations
   - [ ] The threading model
   - [ ] The binary protocol
   - [ ] The REST endpoints
   - [ ] The admin CLI options
   - [ ] The metrics
   - [x] Anything that affects deployment
   
   **Dependencies:** `org.conscrypt:conscrypt-openjdk-uber` 2.5.2 → 2.6.1.
   
   **Deployment:** ARM64 deployments start actually using Conscrypt where they previously fell back to the JDK provider, and tarball installs on glibc < 2.35 now fall back where they previously used Conscrypt. See *Notes for reviewers*.
   
   **Public API:** `SecurityUtility.processConscryptTrustManagers` was `public` but annotated `@InterfaceAudience.Private`, so its removal is not a public-API break under Pulsar's compatibility policy. `sanityCheck` confirms it had no callers outside `pulsar-common`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [improve][misc] Upgrade Conscrypt to 2.6.1 to add aarch64 native support [pulsar]

["lhotari (via GitHub)" <gi...@apache.org>]

lhotari merged PR #26314:
URL: https://github.com/apache/pulsar/pull/26314


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org