You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by david kjerrumgaard <dk...@gmail.com> on 2026/07/06 17:40:16 UTC

[DISCUSS] PIP-489: FIPS 140-3 compliance mode for Apache Pulsar

Hi Pulsar community,

I'd like to start the discussion on PIP-489: FIPS 140-3 compliance mode for
Apache Pulsar. PIP PR: https://github.com/apache/pulsar/pull/26155

Summary:

Pulsar is already FIPS-capable at the JCA layer — SecurityUtility resolves
the BouncyCastle vs. BC-FIPS provider reflectively — butthere is no
supported FIPS deployment path. A full audit of master found the gaps this
PIP addresses:

1. The BC/BC-FIPS swap lost its packaging story in the Gradle migration;
the server distribution bundles non-FIPS BC and explicitlyexcludes bc-fips.

2. The broker binary listener never wires the existing tlsProvider setting,
so PEM TLS is hardwired to Netty's default, which is


   -   The BC/BC-FIPS swap lost its packaging story in the Gradle
   migration; the server distribution bundles non-FIPS BC and explicitly
   excludes bc-fips.
   - The broker binary listener never wires the existing tlsProvider
   setting, so PEM TLS is hardwired to Netty's default, which is BoringSSL
   (netty-tcnative is an unconditional dependency). The web/websocket/worker
   listeners have the same gap; the proxy wires it correctly.
   - Conscrypt (non-FIPS) is the shipped webServiceTlsProvider default.
   - Several security paths use non-approved algorithms: SHA-1-based
   RSA-OAEP and ECIES in MessageCryptoBc (ECIES is absent from BC-FIPS
   entirely, and the class compile-imports org.bouncycastle.jce.* so it
   fails to load on a FIPS classpath), MD5-crypt/DES-crypt in
   AuthenticationProviderBasic, and a non-HMAC SHA-512 construction in
   SaslRoleTokenSigner.
   - The existing bcfips test module only validates the JCA provider swap,
   not the TLS transport, and there is no FIPS documentation.


The proposal defines FIPS mode as a deployment profile, not a fork: TLS
provider wiring for every listener, a FIPS distribution variant (or
documented jar-swap with first-class Gradle wiring), a fipsMode fail-fast
startup validator, a metadata-negotiated migration to RSA_OAEP_SHA256
key wrapping (with ECDH+KDF+AES-KW replacing ECIES in a later phase),
SHA-2-crypt support for Basic auth, a dual-verify HMAC migration for the
SASL role-token signer that survives rolling upgrades, a name-stable fix
for the SHA-1-derived Kubernetes resource names, and a real FIPS TLS
integration test group. Implementation is phased; each phase is
independently shippable, and all shipped defaults are unchanged.

Three companion quick-win PRs are already open and intentionally outside
the PIP: #26152 (MD5 NAR/archive checksums to SHA-256), #26153 (SHA-1
reader subscription names to SHA-256), #26154 (client TLS/crypto helper
fixes for FIPS-restricted JVMs).

Two open questions flagged in the document for this thread:

   - Whether the phase-2 FIPS distribution variant should be published to
   dist.apache.org or provided as a build recipe only (release-management
   question).
   - Whether to eventually revisit the Conscrypt web-TLS default once JDK
   TLS performance is re-benchmarked on modern JVMs (deferred in this PIP).


 Looking forward to your feedback.

  Thanks,
  David Kjerrumgaard

Re: [DISCUSS] PIP-489: FIPS 140-3 compliance mode for Apache Pulsar

Posted by Lari Hotari <lh...@apache.org>.
Hi David,

Thanks for putting this together. A supported FIPS 140-3 path is a
real need for regulated Pulsar deployments, and I think it's a good
idea.

I wanted to flag some overlap so we can keep the efforts aligned.
PIP-478 has been in progress and, besides the asynchronous
authentication interfaces on the client side, it covers the TLS
transport aspects of Pulsar 5.0 on both the client and the server
side. The dependency between the auth and the TLS parts is explained
in the PIP document (PIP-478:
https://github.com/apache/pulsar/pull/25890, discussion thread:
https://lists.apache.org/thread/s9n9jksr9vqgn9o982zmnnkcxdcncy3f).

On the TLS transport side there are more concerns than configuring the
JCA provider. Stricter security compliance -- for example FIPS 140-3
Level 3 -- requires an HSM: the key material is stored in a hardware
device and kept outside the application, so it never crosses into the
Pulsar process. This can be achieved with the PulsarTlsFactory plugin
interface introduced in PIP-478: a plugin implementation can integrate
with a security solution that meets such requirements (for instance
building the TLS context against an HSM via a PKCS#11 token, so the
private key never leaves the device). For the simpler FIPS 140-3 Level
1 case, PIP-478 also lets you configure the JCA provider directly (for
example a FIPS-validated provider such as BC-FIPS on the JDK TLS
engine), and it wires the engine/provider selection through every
server component and the client.

PIP-478 also adds a PulsarHttpClient API. The Pulsar client today uses
several HTTP clients -- for example authentication plugins such as
OAuth2 that call a token endpoint -- which currently don't share a
centralized configuration. The PulsarHttpClient API is added for
authentication plugin implementations so that the TLS configuration of
those HTTP clients can be controlled by the PulsarTlsFactory when
there are special requirements, or handled through the Pulsar v5
client's TlsPolicy configuration when there are none. This keeps the
TLS transport (and FIPS) configuration consistent across the client's
outbound TLS, not only the binary protocol and the web/admin
listeners.

Given that overlap, I'd suggest that PIP-489 builds upon PIP-478 for
the TLS transport aspects rather than defining a separate TLS
transport configuration path. That would let PIP-489 concentrate on
the broader FIPS-compliance concerns that PIP-478 intentionally leaves
out of scope -- FIPS-approved algorithms in message encryption and
authentication, the FIPS distribution/packaging, and a fail-fast
FIPS-mode validation switch -- while reusing the TLS transport
foundation.

On the packaging point specifically: I don't think the Gradle
migration actually lost anything essential there. BC and BC-FIPS can't
co-exist on the classpath, so switching between them is really a
matter of replacing the non-FIPS BC jars with the FIPS ones. The old
bouncy-castle/bcfips module wasn't especially useful for that swap in
practice -- a client can simply exclude the non-FIPS BC dependencies
and add the FIPS ones, and on the server side it can be handled by
keeping the non-FIPS and FIPS jars in separate directories and
choosing which one to put on the classpath based on configuration. So
the FIPS packaging story seems fairly tractable without reintroducing
a dedicated swappable module.

Happy to coordinate on this.

Thanks,
Lari


On Mon, 6 Jul 2026 at 20:40, david kjerrumgaard <dk...@gmail.com> wrote:
>
> Hi Pulsar community,
>
> I'd like to start the discussion on PIP-489: FIPS 140-3 compliance mode for
> Apache Pulsar. PIP PR: https://github.com/apache/pulsar/pull/26155
>
> Summary:
>
> Pulsar is already FIPS-capable at the JCA layer — SecurityUtility resolves
> the BouncyCastle vs. BC-FIPS provider reflectively — butthere is no
> supported FIPS deployment path. A full audit of master found the gaps this
> PIP addresses:
>
> 1. The BC/BC-FIPS swap lost its packaging story in the Gradle migration;
> the server distribution bundles non-FIPS BC and explicitlyexcludes bc-fips.
>
> 2. The broker binary listener never wires the existing tlsProvider setting,
> so PEM TLS is hardwired to Netty's default, which is
>
>
>    -   The BC/BC-FIPS swap lost its packaging story in the Gradle
>    migration; the server distribution bundles non-FIPS BC and explicitly
>    excludes bc-fips.
>    - The broker binary listener never wires the existing tlsProvider
>    setting, so PEM TLS is hardwired to Netty's default, which is BoringSSL
>    (netty-tcnative is an unconditional dependency). The web/websocket/worker
>    listeners have the same gap; the proxy wires it correctly.
>    - Conscrypt (non-FIPS) is the shipped webServiceTlsProvider default.
>    - Several security paths use non-approved algorithms: SHA-1-based
>    RSA-OAEP and ECIES in MessageCryptoBc (ECIES is absent from BC-FIPS
>    entirely, and the class compile-imports org.bouncycastle.jce.* so it
>    fails to load on a FIPS classpath), MD5-crypt/DES-crypt in
>    AuthenticationProviderBasic, and a non-HMAC SHA-512 construction in
>    SaslRoleTokenSigner.
>    - The existing bcfips test module only validates the JCA provider swap,
>    not the TLS transport, and there is no FIPS documentation.
>
>
> The proposal defines FIPS mode as a deployment profile, not a fork: TLS
> provider wiring for every listener, a FIPS distribution variant (or
> documented jar-swap with first-class Gradle wiring), a fipsMode fail-fast
> startup validator, a metadata-negotiated migration to RSA_OAEP_SHA256
> key wrapping (with ECDH+KDF+AES-KW replacing ECIES in a later phase),
> SHA-2-crypt support for Basic auth, a dual-verify HMAC migration for the
> SASL role-token signer that survives rolling upgrades, a name-stable fix
> for the SHA-1-derived Kubernetes resource names, and a real FIPS TLS
> integration test group. Implementation is phased; each phase is
> independently shippable, and all shipped defaults are unchanged.
>
> Three companion quick-win PRs are already open and intentionally outside
> the PIP: #26152 (MD5 NAR/archive checksums to SHA-256), #26153 (SHA-1
> reader subscription names to SHA-256), #26154 (client TLS/crypto helper
> fixes for FIPS-restricted JVMs).
>
> Two open questions flagged in the document for this thread:
>
>    - Whether the phase-2 FIPS distribution variant should be published to
>    dist.apache.org or provided as a build recipe only (release-management
>    question).
>    - Whether to eventually revisit the Conscrypt web-TLS default once JDK
>    TLS performance is re-benchmarked on modern JVMs (deferred in this PIP).
>
>
>  Looking forward to your feedback.
>
>   Thanks,
>   David Kjerrumgaard

Re: [DISCUSS] PIP-489: FIPS 140-3 compliance mode for Apache Pulsar

Posted by Lari Hotari <lh...@apache.org>.
Hi David,

One more bit of history that may help scope PIP-489: FIPS / BouncyCastle-FIPS support in Pulsar has been partial and handled case by case in the past, rather than as a coherent, tested capability -- which is a big part of why a proper end-to-end FIPS design is worth doing.

Two concrete examples:

1. Pulsar has never actually used BouncyCastle's JSSE (TLS) provider. A pulsar-site correction (https://github.com/apache/pulsar-site/pull/974) removed a docs statement that implied otherwise, noting "Pulsar does not implement bouncy castle jsse; there is no dependency on bc jsse in Pulsar." So even with the BC-FIPS jars on the classpath, the TLS transport did not route through a BouncyCastle/FIPS TLS provider -- the FIPS story for the transport itself was never wired.

2. On the message-encryption side, https://github.com/apache/pulsar/pull/23122 switched AES-GCM in MessageCryptoBc from the BouncyCastle provider to SunJCE for performance. In reviewing it (https://github.com/apache/pulsar/pull/23122#pullrequestreview-2218133922) I raised the FIPS concern -- when the FIPS library is enabled, SunJCE should not become the default -- and the finding was that the existing code already ignored the presence of BouncyCastleFipsProvider. In other words FIPS compliance was already a separate, not-fully-handled concern rather than something the crypto paths consistently respected.

The takeaway is that FIPS support in Pulsar has been incomplete and piecemeal: the TLS transport was never routed through a FIPS provider, and the crypto paths didn't consistently honor the FIPS provider when it was present. That's exactly the gap worth closing, and it splits naturally: PIP-478 covers the TLS-transport slice (a configurable engine plus any JCA provider via TlsPolicy.jcaProvider, and the PulsarTlsFactory plugin for HSM-backed / FIPS 140-3 Level 3 cases), and PIP-489 can cover the broader FIPS-mode concerns -- FIPS-approved algorithms in message encryption and authentication, packaging, and a fail-fast validation switch -- so the crypto paths consistently respect FIPS rather than silently falling back to non-validated providers.

Thanks,

Lari

On 2026/07/06 17:40:16 david kjerrumgaard wrote:
> Hi Pulsar community,
> 
> I'd like to start the discussion on PIP-489: FIPS 140-3 compliance mode for
> Apache Pulsar. PIP PR: https://github.com/apache/pulsar/pull/26155
> 
> Summary:
> 
> Pulsar is already FIPS-capable at the JCA layer — SecurityUtility resolves
> the BouncyCastle vs. BC-FIPS provider reflectively — butthere is no
> supported FIPS deployment path. A full audit of master found the gaps this
> PIP addresses:
> 
> 1. The BC/BC-FIPS swap lost its packaging story in the Gradle migration;
> the server distribution bundles non-FIPS BC and explicitlyexcludes bc-fips.
> 
> 2. The broker binary listener never wires the existing tlsProvider setting,
> so PEM TLS is hardwired to Netty's default, which is
> 
> 
>    -   The BC/BC-FIPS swap lost its packaging story in the Gradle
>    migration; the server distribution bundles non-FIPS BC and explicitly
>    excludes bc-fips.
>    - The broker binary listener never wires the existing tlsProvider
>    setting, so PEM TLS is hardwired to Netty's default, which is BoringSSL
>    (netty-tcnative is an unconditional dependency). The web/websocket/worker
>    listeners have the same gap; the proxy wires it correctly.
>    - Conscrypt (non-FIPS) is the shipped webServiceTlsProvider default.
>    - Several security paths use non-approved algorithms: SHA-1-based
>    RSA-OAEP and ECIES in MessageCryptoBc (ECIES is absent from BC-FIPS
>    entirely, and the class compile-imports org.bouncycastle.jce.* so it
>    fails to load on a FIPS classpath), MD5-crypt/DES-crypt in
>    AuthenticationProviderBasic, and a non-HMAC SHA-512 construction in
>    SaslRoleTokenSigner.
>    - The existing bcfips test module only validates the JCA provider swap,
>    not the TLS transport, and there is no FIPS documentation.
> 
> 
> The proposal defines FIPS mode as a deployment profile, not a fork: TLS
> provider wiring for every listener, a FIPS distribution variant (or
> documented jar-swap with first-class Gradle wiring), a fipsMode fail-fast
> startup validator, a metadata-negotiated migration to RSA_OAEP_SHA256
> key wrapping (with ECDH+KDF+AES-KW replacing ECIES in a later phase),
> SHA-2-crypt support for Basic auth, a dual-verify HMAC migration for the
> SASL role-token signer that survives rolling upgrades, a name-stable fix
> for the SHA-1-derived Kubernetes resource names, and a real FIPS TLS
> integration test group. Implementation is phased; each phase is
> independently shippable, and all shipped defaults are unchanged.
> 
> Three companion quick-win PRs are already open and intentionally outside
> the PIP: #26152 (MD5 NAR/archive checksums to SHA-256), #26153 (SHA-1
> reader subscription names to SHA-256), #26154 (client TLS/crypto helper
> fixes for FIPS-restricted JVMs).
> 
> Two open questions flagged in the document for this thread:
> 
>    - Whether the phase-2 FIPS distribution variant should be published to
>    dist.apache.org or provided as a build recipe only (release-management
>    question).
>    - Whether to eventually revisit the Conscrypt web-TLS default once JDK
>    TLS performance is re-benchmarked on modern JVMs (deferred in this PIP).
> 
> 
>  Looking forward to your feedback.
> 
>   Thanks,
>   David Kjerrumgaard
>