[I] TLS Configuration with Public Certificates [pulsar-helm-chart]

["lukasscholze-plenasec (via GitHub)" <gi...@apache.org>]

lukasscholze-plenasec opened a new issue, #721:
URL: https://github.com/apache/pulsar-helm-chart/issues/721

   # TLS Configuration with Public Certificates from Cloudflare
   
   ## Overview
   
   I am trying to deploy Apache Pulsar using the official Helm chart with **publicly trusted TLS certificates** issued by Cloudflare via cert-manager.
   
   My preferred setup would be to let the Helm chart create and manage the certificates itself. However, the generated certificate requests automatically include Kubernetes-internal DNS names for the Pulsar components. These names are not covered by our public certificates, so the certificates cannot be issued for our intended setup.
   
   Ideally, I would like to be able to configure the DNS names/SANs used by the Helm chart when it creates the certificates, so that the certificates contain the DNS names actually used for communication between the Pulsar components.
   
   As an alternative, I tried creating the certificates separately and mounting the resulting Secrets into the Pulsar components. This exposed several additional issues with using externally managed certificates, described below.
   
   The Helm chart version I am using is **4.7.0**.
   
   ## Environment
   
   * Pulsar Helm chart: `4.7.0`
   * Kubernetes
   * cert-manager
   * Cloudflare ACME `ClusterIssuer`
   * ArgoCD
   * Namespace: `pulsar`
   * Domain: `plenasec.fyi`
   * 3 replicas per Pulsar component
   * ClusterIP services
   * No Kubernetes Ingress
   
   ## Architecture
   
   The intended architecture is:
   
   ```text
   VPN
    |
    v
   CoreDNS
    |
    +-- pulsar.plenasec.fyi
    +-- broker.pulsar.plenasec.fyi
    +-- bookie.pulsar.plenasec.fyi
    +-- zookeeper.pulsar.plenasec.fyi
    |
    v
   Pulsar ClusterIP Services
   ```
   
   All Pulsar components need to communicate using DNS names that are included in the certificates' SANs.
   
   The custom DNS names are resolved internally by CoreDNS to the corresponding Pulsar services. The names themselves are public DNS names covered by the Cloudflare certificates.
   
   ## Problems encountered
   
   ### 1. Certificate generation includes Kubernetes-internal DNS names
   
   When using the Helm chart's built-in certificate management, the generated certificate requests include Kubernetes-internal DNS names in addition to the configured names.
   
   For this setup, I need the certificates to contain the DNS names actually used for Pulsar communication, for example:
   
   ```text
   pulsar.plenasec.fyi
   broker.pulsar.plenasec.fyi
   bookie.pulsar.plenasec.fyi
   zookeeper.pulsar.plenasec.fyi
   ```
   
   The publicly issued certificates cannot be issued for the automatically generated Kubernetes-internal names.
   
   Is there a supported way to configure the Helm chart's certificate generation so that I can control the DNS names/SANs included in the generated certificates?
   
   ### 2. External certificates still require issuer configuration
   
   As an alternative, I tried managing the certificates separately and using `createCert: false`.
   
   The Helm chart's `_certs.tpl` appears to require issuer configuration even when certificate creation is disabled.
   
   I encountered:
   
   ```text
   certs.issuers.ca.secretName is required when TLS is enabled and certs.internal_issuer.enabled is false
   ```
   
   I worked around this by modifying `_certs.tpl` so that it does not fail when no issuer is configured.
   
   ### 3. CA volume is always generated
   
   The component templates appear to create a `ca` volume referencing:
   
   ```yaml
   pulsar.certs.issuers.ca.secretName
   ```
   
   If this value is empty, Kubernetes rejects the generated pod specification:
   
   ```text
   spec.template.spec.volumes[3].secret.secretName: Required value
   ```
   
   As a workaround, I configured:
   
   ```yaml
   certs:
     issuers:
       ca:
         secretName: pulsar-ca-secret
   ```
   
   This allows the pods to be created.
   
   ### 4. Zookeeper TLS configuration
   
   The remaining problem occurs with Zookeeper.
   
   The certificates created by cert-manager provide:
   
   ```text
   tls.crt
   tls.key
   ca.crt
   ```
   
   I also tried enabling:
   
   ```yaml
   additionalOutputFormats:
     - type: CombinedPEM
   ```
   
   which creates:
   
   ```text
   tls-combined.pem
   ```
   
   However, the generated Zookeeper configuration uses:
   
   ```text
   -Dzookeeper.ssl.keyStore.location=/pulsar/certs/zookeeper/tls-combined.pem
   ```
   
   Zookeeper then fails to start with:
   
   ```text
   java.security.spec.InvalidKeySpecException: Inappropriate key specification: Unable to decode key
   ```
   
   This appears to be a mismatch between the certificate format generated by cert-manager and the format expected by the Zookeeper TLS configuration generated by the Helm chart.
   
   ## Question
   
   The preferred solution for my use case would be to continue using the Helm chart's built-in certificate management, but have the ability to control the DNS names/SANs included in the generated certificates.
   
   If that is not currently supported, I would like to understand the intended way to use externally managed certificates with the chart.
   
   In particular:
   
   1. Can the DNS names/SANs generated by the Helm chart's certificate resources be configured so that Kubernetes-internal DNS names are not automatically added?
   
   2. If external certificates are used with `createCert: false`, is there a supported way to do so without requiring an issuer configuration?
   
   3. Can the `ca` volume be made optional when using externally managed certificates?
   
   4. What certificate/keystore format is expected by the Zookeeper TLS configuration?
   
      Should externally managed certificates be provided as:
   
      ```text
      tls.crt
      tls.key
      ```
   
      separately, or does Zookeeper require a combined PEM/PKCS12/JKS keystore?
   
   5. If `tls-combined.pem` is the intended format, could the documentation clarify how the certificate and private key are expected to be provided?
   
   ## Current workaround
   
   I currently create the certificates separately using cert-manager and manage them through a separate ArgoCD application.
   
   I have also modified `_certs.tpl` to allow TLS configuration without an internal issuer and configured an existing CA Secret.
   
   This gets the deployment past the certificate and pod-generation issues, but Zookeeper still fails during startup because of the TLS keystore configuration.
   
   The main goal is to use the Helm chart's certificate management with configurable DNS names. The external-certificate setup is currently a workaround for the limitations encountered with the built-in certificate generation.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] TLS Configuration with Public Certificates [pulsar-helm-chart]

["lhotari (via GitHub)" <gi...@apache.org>]

lhotari closed issue #721: TLS Configuration with Public Certificates
URL: https://github.com/apache/pulsar-helm-chart/issues/721


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org