You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "merlimat (via GitHub)" <gi...@apache.org> on 2026/10/12 21:44:52 UTC
[PR] Restrict network access to Oxia and support TLS between Oxia processes [pulsar-helm-chart]
merlimat opened a new pull request, #727:
URL: https://github.com/apache/pulsar-helm-chart/pull/727
### Motivation
When the chart deploys Oxia as the metadata store, Oxia's cluster-internal endpoints are more exposed than they need to be:
- The `<release>-pulsar-oxia` ClusterIP Service publishes the servers' internal endpoint (6649) and their metrics endpoint, not just the public endpoint. The internal endpoint serves the RPCs the Oxia servers use for replication and the coordinator uses to manage the shards, and it has no authentication by default. Nothing in the chart uses the extra ports through this Service: the Pulsar components connect to the public endpoint through the headless `<release>-pulsar-oxia-svc` Service, and the PodMonitors scrape the pods directly.
- There is no NetworkPolicy, so any pod in the cluster can reach the internal endpoints of the servers and the coordinator, and the coordinator's admin endpoint (6651), through pod IPs or the headless Service.
- `tls.oxia.enabled` has been in `values.yaml` since Oxia support was added (#544), but no template uses it.
For example, with Oxia enabled, any pod in any namespace can open a connection to `<release>-pulsar-oxia:6649` or `<release>-pulsar-oxia-server-0.<release>-pulsar-oxia-svc:6649` and call the internal RPCs.
### Modifications
- `oxia-server-service-public.yaml`: the `<release>-pulsar-oxia` Service only exposes the `public` port. The headless Service is unchanged.
- New `oxia-networkpolicy.yaml`, controlled by `oxia.networkPolicy.enabled` (default `true`). It selects the Oxia server and coordinator pods and allows:
- the public endpoint from anywhere (`oxia.networkPolicy.public.from` can restrict it),
- the server and coordinator metrics endpoints from anywhere (`oxia.networkPolicy.metrics.from`),
- the server internal, coordinator internal and coordinator admin (6651) endpoints only from the Oxia pods of the release, plus `oxia.networkPolicy.internal.from`.
The Pulsar components wait for Oxia with `nslookup` and connect to the public endpoint, and the probes are exec probes, so the policy doesn't affect them. It needs a CNI plugin that enforces NetworkPolicy.
- `tls.oxia` now takes effect, together with `tls.enabled`:
- `tls-certs-internal.yaml` creates the `<release>-tls-oxia` Certificate through `pulsar.cert.template`, for the server pods and the headless Service. `createCert: false`, `dnsNames`, `ipAddresses`, `secretAnnotations` and every `tls.common.sanMode` work as for the other components.
- The servers and the coordinator mount it at `/pulsar/certs/oxia`, and the CA at `/pulsar/certs/ca`.
- The servers get `--internal-tls-*` for their internal endpoint and `--peer-tls-*` for replication. The coordinator gets `--peer-tls-*` for its calls to the servers.
- `tls.oxia.clientAuth` adds `--internal-tls-client-auth` (mTLS).
- The servers share one certificate, so the client side verifies it against the headless Service name (`--peer-tls-server-name`). The certificate contains that name in every SAN mode.
- The public endpoint stays plaintext, because the Oxia Java client in Pulsar only trusts the JVM's default CA certificates, so it can't verify a certificate from the chart's CA.
- `oxia health` only speaks plaintext, so with TLS on, the server probes use the public endpoint, which serves the same health service.
- README: a new "Oxia network access and TLS" section.
- CI:
- a new "Oxia TLS" scenario (`.ci/clusters/values-oxia-tls.yaml`, with mTLS),
- Oxia added to `.ci/test-certificate-san-modes.sh` for all three SAN modes,
- the NetworkPolicy `from` lists and Oxia mTLS added to `.ci/templates-all-values.yaml`.
Upgrade notes:
- Anything outside the release that connects to the Oxia internal or admin endpoints, or that uses the internal or metrics ports through `<release>-pulsar-oxia`, needs to be added to `oxia.networkPolicy.internal.from` or switched to the headless Service. NetworkPolicies are additive, so a release whose Oxia pods are already selected by stricter policies gets this policy's allowed sources added. Set `oxia.networkPolicy.enabled: false` to keep the previous behavior.
- Turning `tls.oxia` on or off in an existing release restarts the Oxia pods. Servers on the new setting can't replicate with servers on the old one until all of them have restarted.
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
- `helm lint`.
- `helm template` diffed against `master`:
- Defaults (ZooKeeper) and the all-values render are unchanged.
- With Oxia, only the trimmed Service and the new NetworkPolicy differ.
- With `tls.oxia`, the Oxia StatefulSet and Deployment add only the TLS flags, the cert mounts and the probe port, and there is a new Certificate.
- `.ci/test-certificate-san-modes.sh` passes. A negative control with a wrong Oxia service name fails.
- I haven't run this chart on a cluster yet. The new "Oxia TLS" scenario covers a fresh install with mTLS, and the existing Oxia scenario now runs with the NetworkPolicy. The same NetworkPolicy and internal-endpoint mTLS setup was tested on kind with Oxia 0.16.11 in the equivalent oxia-db/helm-charts change (oxia-db/helm-charts#15): the cluster forms, and from other namespaces 6648 and metrics stay reachable while 6649 and 6651 are blocked. Plaintext connections, and TLS connections without a client certificate, are rejected on 6649.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Restrict network access to Oxia and support TLS between Oxia processes [pulsar-helm-chart]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #727:
URL: https://github.com/apache/pulsar-helm-chart/pull/727#discussion_r4258497170
##########
charts/pulsar/templates/oxia-networkpolicy.yaml:
##########
@@ -0,0 +1,77 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+{{- if and .Values.components.oxia .Values.oxia.networkPolicy.enabled }}
+{{- $oxiaPods := list (printf "%s-server" .Values.oxia.component) (printf "%s-coordinator" .Values.oxia.component) }}
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+ name: {{ template "pulsar.fullname" . }}-{{ .Values.oxia.component }}
+ namespace: {{ template "pulsar.namespace" . }}
+ labels:
+ {{- include "pulsar.standardLabels" . | nindent 4 }}
+ component: {{ .Values.oxia.component }}
+spec:
+ podSelector:
+ matchLabels:
+ {{- include "pulsar.matchLabels" . | nindent 6 }}
+ matchExpressions:
+ - key: component
+ operator: In
+ values:
+ {{- toYaml $oxiaPods | nindent 10 }}
+ policyTypes:
+ - Ingress
+ ingress:
+ # Client API, used by the Pulsar components
+ - ports:
+ - port: {{ .Values.oxia.server.ports.public }}
+ {{- with .Values.oxia.networkPolicy.public.from }}
+ from:
+ {{- toYaml . | nindent 8 }}
+ {{- end }}
Review Comment:
**[QUALITY] a non-empty `public.from` silently cuts off the release's own Pulsar components**
The internal rule always includes the release's Oxia pods, but the public rule is exactly `oxia.networkPolicy.public.from` when it is set. The broker, proxy, bookkeeper, toolset, function worker and the init jobs all connect to the public port, so a user who follows the README hint and sets `public.from` to, say, a monitoring or client namespace selector will lose metadata store access for the whole release, and the cause (a NetworkPolicy on the Oxia pods) is not obvious from the symptoms.
The empty-means-everyone semantics are fine, but could `values.yaml` and the README say explicitly that a non-empty `public.from` must also list the release's own Pulsar pods (for example a `podSelector` on `app: pulsar` and `release: <release>`)? Alternatively the template could append that selector to the user's list, the way it does for the internal rule.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Restrict network access to Oxia and support TLS between Oxia processes [pulsar-helm-chart]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on code in PR #727:
URL: https://github.com/apache/pulsar-helm-chart/pull/727#discussion_r4260497231
##########
charts/pulsar/templates/oxia-networkpolicy.yaml:
##########
@@ -0,0 +1,77 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+{{- if and .Values.components.oxia .Values.oxia.networkPolicy.enabled }}
+{{- $oxiaPods := list (printf "%s-server" .Values.oxia.component) (printf "%s-coordinator" .Values.oxia.component) }}
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+ name: {{ template "pulsar.fullname" . }}-{{ .Values.oxia.component }}
+ namespace: {{ template "pulsar.namespace" . }}
+ labels:
+ {{- include "pulsar.standardLabels" . | nindent 4 }}
+ component: {{ .Values.oxia.component }}
+spec:
+ podSelector:
+ matchLabels:
+ {{- include "pulsar.matchLabels" . | nindent 6 }}
+ matchExpressions:
+ - key: component
+ operator: In
+ values:
+ {{- toYaml $oxiaPods | nindent 10 }}
+ policyTypes:
+ - Ingress
+ ingress:
+ # Client API, used by the Pulsar components
+ - ports:
+ - port: {{ .Values.oxia.server.ports.public }}
+ {{- with .Values.oxia.networkPolicy.public.from }}
+ from:
+ {{- toYaml . | nindent 8 }}
+ {{- end }}
Review Comment:
Changed in 411ea36: when `oxia.networkPolicy.public.from` is set, the public rule now always includes a `podSelector` on the release's pods (`app` + `release`), like the internal rule, so the brokers, bookies, proxies, toolset and init jobs keep access. `values.yaml` and the README say so. An empty list still means anyone.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Restrict network access to Oxia and support TLS between Oxia processes [pulsar-helm-chart]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat merged PR #727:
URL: https://github.com/apache/pulsar-helm-chart/pull/727
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org