You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by lh...@apache.org on 2026/06/24 12:14:10 UTC
(pulsar-helm-chart) branch master updated (626bcec -> bcdbeef)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar-helm-chart.git
from 626bcec Add one node oxia settings
new fce2efd Add example of faster disk cleanup
new 0b15145 remove values-testing.yaml
new d28440f Remove tuning for oxia in values-one-node.yaml
new 3d7ae32 Add example for disabling fsync
new bcdbeef Examples: update README for fsync/disk-cleanup examples and testing removal
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
examples/README.md | 5 +-
.../values-disable-fsync.yaml | 18 +++----
examples/values-faster-disk-cleanup.yaml | 61 ++++++++++++++++++++++
examples/values-one-node.yaml | 4 --
examples/values-testing.yaml | 42 ---------------
5 files changed, 72 insertions(+), 58 deletions(-)
copy .ci/clusters/values-zk-tls.yaml => examples/values-disable-fsync.yaml (75%)
create mode 100644 examples/values-faster-disk-cleanup.yaml
delete mode 100644 examples/values-testing.yaml
(pulsar-helm-chart) 01/05: Add example of faster disk cleanup
Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar-helm-chart.git
commit fce2efd7f4d7a732216d624b3d8bcfde9de998d1
Author: Lari Hotari <lh...@apache.org>
AuthorDate: Wed Jun 24 14:26:08 2026 +0300
Add example of faster disk cleanup
---
examples/README.md | 1 +
examples/values-faster-disk-cleanup.yaml | 61 ++++++++++++++++++++++++++++++++
2 files changed, 62 insertions(+)
diff --git a/examples/README.md b/examples/README.md
index fe1b330..0a75220 100644
--- a/examples/README.md
+++ b/examples/README.md
@@ -147,6 +147,7 @@ in examples that deploy a broker.
| [`values-no-persistence.yaml`](values-no-persistence.yaml) | **Ephemeral.** Deploys stateful components with `emptyDir` instead of PVCs. All data is lost on pod restart / cluster shutdown. Sets `autoSkipNonRecoverableData` so BookKeeper tolerates the lost state. For throwaway testing/CI only. |
| [`values-bookkeeper-aws.yaml`](values-bookkeeper-aws.yaml) | A 3-bookie cluster using AWS EBS (`gp2`) `PersistentVolumeClaims` for the BookKeeper journal and ledgers. Monitoring stack disabled. |
| [`values-zookeeper-aws.yaml`](values-zookeeper-aws.yaml) | A configuration store running only ZooKeeper backed by AWS EBS (`gp2`) volumes, including the `externalZookeeperServerList` option for building a ZooKeeper cluster that spans namespaces/clusters. |
+| [`values-faster-disk-cleanup.yaml`](values-faster-disk-cleanup.yaml) | Tune BookKeeper and the broker to reclaim disk space faster: more frequent BookKeeper compaction/garbage collection with a smaller journal, and faster broker managed-ledger rollover so closed ledgers are trimmed sooner. Handy for space-constrained or test clusters. These are the same cleanup settings bundled into `values-testing.yaml`. |
### Security (TLS and authentication)
diff --git a/examples/values-faster-disk-cleanup.yaml b/examples/values-faster-disk-cleanup.yaml
new file mode 100644
index 0000000..0b17a51
--- /dev/null
+++ b/examples/values-faster-disk-cleanup.yaml
@@ -0,0 +1,61 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# Faster disk cleanup settings for BookKeeper and the Pulsar broker.
+# Useful for testing scenarios where disk space is limited and you
+# want to reclaim disk space as quickly as possible.
+# These values are tuned for testing and are not optimal for production use.
+
+bookkeeper:
+ configData:
+ # roll the journal file more frequently
+ journalMaxSizeMB: "256"
+ # don't keep any journal backups, since we want to clean up disk space as fast as possible
+ journalMaxBackups: "0" # default 5
+ # configure the entry log size limit
+ logSizeLimit: "536870912" # 512MB, default 1073741824 (1024MB)
+ # configure the data compaction (bookie entry log compaction and gc) settings
+ # https://bookkeeper.apache.org/docs/reference/config#garbage-collection-settings
+ # https://bookkeeper.apache.org/docs/reference/config#entry-log-compaction-settings
+ minorCompactionThreshold: "0.2" # default 0.2 (use default)
+ minorCompactionInterval: "300" # default 3600 seconds (5 minutes vs default 1 hour)
+ majorCompactionThreshold: "0.8" # default 0.8 (use default)
+ majorCompactionInterval: "900" # default 86400 seconds (15 minutes vs default 1 day)
+ gcWaitTime: "60000" # default 600000 milli-seconds (1 minute vs default 10 minutes)
+ isForceGCAllowWhenNoSpace: "true" # default false
+ # disk utilization configuration
+ # https://bookkeeper.apache.org/docs/reference/config#disk-utilization
+ # Make sure that diskUsageLwmThreshold <= diskUsageWarnThreshold <= diskUsageThreshold
+ diskUsageLwmThreshold: "0.85" # default 0.95 (inherits diskUsageThreshold); GC keeps running until usage drops below this
+ diskUsageWarnThreshold: "0.9" # default 0.90 (use default)
+ diskUsageThreshold: "0.95" # default 0.95 (use default); bookie goes read-only above this
+ diskCheckInterval: "5000" # value is in milliseconds; check every 5s vs default 10s (10000)
+
+broker:
+ configData:
+ # more aggressive disk cleanup
+ managedLedgerMinLedgerRolloverTimeMinutes: "1"
+ managedLedgerMaxLedgerRolloverTimeMinutes: "5"
+ managedLedgerMaxSizePerLedgerMbytes: "256"
+ managedLedgerCursorRolloverTimeInSeconds: "900"
+ # inactive topic deletion is already enabled by default
+ # (brokerDeleteInactiveTopicsEnabled=true, mode delete_when_no_subscriptions, checked every 60s).
+ # Uncomment to set how long a topic may stay inactive before deletion;
+ # use a SMALL value for faster cleanup (default falls back to brokerDeleteInactiveTopicsFrequencySeconds=60).
+ # brokerDeleteInactiveTopicsMaxInactiveDurationSeconds: "60"
\ No newline at end of file
(pulsar-helm-chart) 02/05: remove values-testing.yaml
Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar-helm-chart.git
commit 0b15145e369cb57aa0d7409c65a6a3eae9fd6982
Author: Lari Hotari <lh...@apache.org>
AuthorDate: Wed Jun 24 14:58:32 2026 +0300
remove values-testing.yaml
---
examples/values-testing.yaml | 42 ------------------------------------------
1 file changed, 42 deletions(-)
diff --git a/examples/values-testing.yaml b/examples/values-testing.yaml
deleted file mode 100644
index 99d37b9..0000000
--- a/examples/values-testing.yaml
+++ /dev/null
@@ -1,42 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-
-# disabled AntiAffinity
-affinity:
- anti_affinity: false
-
-bookkeeper:
- configData:
- # more aggressive disk cleanup
- journalMaxSizeMB: "256"
- majorCompactionInterval: "600"
- minorCompactionInterval: "300"
- compactionRateByEntries: "5000"
- gcWaitTime: "60000"
-
-broker:
- configData:
- # more aggressive disk cleanup
- managedLedgerMinLedgerRolloverTimeMinutes: "1"
- managedLedgerMaxLedgerRolloverTimeMinutes: "5"
- # configure deletion of inactive topics
- brokerDeleteInactiveTopicsMaxInactiveDurationSeconds: "86400"
-
-proxy:
- replicaCount: 1
(pulsar-helm-chart) 04/05: Add example for disabling fsync
Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar-helm-chart.git
commit 3d7ae323ea7ba37f8195097902000d18c7f5af4d
Author: Lari Hotari <lh...@apache.org>
AuthorDate: Wed Jun 24 15:09:04 2026 +0300
Add example for disabling fsync
---
examples/values-disable-fsync.yaml | 28 ++++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
diff --git a/examples/values-disable-fsync.yaml b/examples/values-disable-fsync.yaml
new file mode 100644
index 0000000..b3d3ea7
--- /dev/null
+++ b/examples/values-disable-fsync.yaml
@@ -0,0 +1,28 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+bookkeeper:
+ configData:
+ # disable fsync for testing purposes (not recommended for production)
+ journalSyncData: "false"
+
+zookeeper:
+ configData:
+ # disable fsync for testing purposes (not recommended for production)
+ forceSync: "false"
\ No newline at end of file
(pulsar-helm-chart) 05/05: Examples: update README for fsync/disk-cleanup examples and testing removal
Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar-helm-chart.git
commit bcdbeef817520c0c68aa0b34d4689cbc169d2e00
Author: Lari Hotari <lh...@apache.org>
AuthorDate: Wed Jun 24 15:12:33 2026 +0300
Examples: update README for fsync/disk-cleanup examples and testing removal
- Add a row for the new values-disable-fsync.yaml example.
- Drop the values-testing.yaml row (the file was removed) and the now-dead
cross-reference to it from the faster-disk-cleanup row.
- Expand the values-faster-disk-cleanup.yaml description to match the broader
set of BookKeeper GC/compaction, disk-utilization and journal settings.
- Note the single-node Oxia sizing now included in values-one-node.yaml.
---
examples/README.md | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/examples/README.md b/examples/README.md
index 0a75220..1ac8b15 100644
--- a/examples/README.md
+++ b/examples/README.md
@@ -127,7 +127,7 @@ in examples that deploy a broker.
| File | Description |
| ---- | ----------- |
-| [`values-one-node.yaml`](values-one-node.yaml) | Minimal distributed cluster with a single replica of each component (ZooKeeper, BookKeeper, broker, proxy). Auto-recovery and anti-affinity are disabled and the managed-ledger ensemble/quorum sizes are set to `1` to fit a single bookie. Persistence is kept enabled. Good for local development on a single node. |
+| [`values-one-node.yaml`](values-one-node.yaml) | Minimal distributed cluster with a single replica of each component (ZooKeeper, BookKeeper, broker, proxy). Auto-recovery and anti-affinity are disabled and the managed-ledger ensemble/quorum sizes are set to `1` to fit a single bookie. Single-node Oxia sizing is included as well (applied if Oxia is enabled as the metadata store). Persistence is kept enabled. Good for local development on a single node. |
| [`values-standalone.yaml`](values-standalone.yaml) | Runs all of Pulsar in a single standalone process. The distributed components (ZooKeeper, BookKeeper, broker, auto-recovery) are automatically suppressed. Smallest footprint; ideal for development and smoke tests. Includes a commented-out snippet for fronting standalone with the Pulsar proxy. |
| [`values-minikube.yaml`](values-minikube.yaml) | Single-replica cluster tuned for [Minikube](https://minikube.sigs.k8s.io/): anti-affinity off, BookKeeper memory caches minimized, the Dekaf UI enabled. Persistence is kept enabled so data survives pod restarts. |
| [`values-local-cluster.yaml`](values-local-cluster.yaml) | A Pulsar cluster (`metadataPrefix: /cluster1`) that attaches to a **separate** configuration store. Pair it with `values-cs.yaml`, which deploys that configuration store. Disables the bundled monitoring stack and the Dekaf UI is enabled. |
@@ -147,7 +147,8 @@ in examples that deploy a broker.
| [`values-no-persistence.yaml`](values-no-persistence.yaml) | **Ephemeral.** Deploys stateful components with `emptyDir` instead of PVCs. All data is lost on pod restart / cluster shutdown. Sets `autoSkipNonRecoverableData` so BookKeeper tolerates the lost state. For throwaway testing/CI only. |
| [`values-bookkeeper-aws.yaml`](values-bookkeeper-aws.yaml) | A 3-bookie cluster using AWS EBS (`gp2`) `PersistentVolumeClaims` for the BookKeeper journal and ledgers. Monitoring stack disabled. |
| [`values-zookeeper-aws.yaml`](values-zookeeper-aws.yaml) | A configuration store running only ZooKeeper backed by AWS EBS (`gp2`) volumes, including the `externalZookeeperServerList` option for building a ZooKeeper cluster that spans namespaces/clusters. |
-| [`values-faster-disk-cleanup.yaml`](values-faster-disk-cleanup.yaml) | Tune BookKeeper and the broker to reclaim disk space faster: more frequent BookKeeper compaction/garbage collection with a smaller journal, and faster broker managed-ledger rollover so closed ledgers are trimmed sooner. Handy for space-constrained or test clusters. These are the same cleanup settings bundled into `values-testing.yaml`. |
+| [`values-faster-disk-cleanup.yaml`](values-faster-disk-cleanup.yaml) | Tune BookKeeper and the broker to reclaim disk space as fast as possible: frequent BookKeeper entry-log compaction and garbage collection, lower disk-utilization GC thresholds, a smaller rollover-heavy journal with no backups, and faster broker managed-ledger rollover so closed ledgers are trimmed sooner. For space-constrained test clusters; the values are not tuned for production. |
+| [`values-disable-fsync.yaml`](values-disable-fsync.yaml) | Disable fsync on the BookKeeper journal (`journalSyncData: false`) and ZooKeeper (`forceSync: false`) for faster writes in tests. Trades durability for speed — data written just before a crash or power loss can be lost — so it is **not for production**. |
### Security (TLS and authentication)
@@ -169,4 +170,3 @@ in examples that deploy a broker.
| File | Description |
| ---- | ----------- |
| [`values-init-containers.yaml`](values-init-containers.yaml) | Demonstrates attaching custom `initContainers` to each component (ZooKeeper, BookKeeper, auto-recovery, broker, proxy, toolset). |
-| [`values-testing.yaml`](values-testing.yaml) | Settings handy for test environments: anti-affinity off, aggressive BookKeeper/broker disk cleanup, faster ledger rollover and inactive-topic deletion. Not for production. |
(pulsar-helm-chart) 03/05: Remove tuning for oxia in values-one-node.yaml
Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar-helm-chart.git
commit d28440fb5ac7b5d1358e68042fffba1d595f1d3d
Author: Lari Hotari <lh...@apache.org>
AuthorDate: Wed Jun 24 14:58:47 2026 +0300
Remove tuning for oxia in values-one-node.yaml
---
examples/values-one-node.yaml | 4 ----
1 file changed, 4 deletions(-)
diff --git a/examples/values-one-node.yaml b/examples/values-one-node.yaml
index 9751436..c1efb27 100644
--- a/examples/values-one-node.yaml
+++ b/examples/values-one-node.yaml
@@ -30,10 +30,6 @@ oxia:
replicationFactor: 1
server:
replicas: 1
- cpuLimit: 333m
- memoryLimit: 200Mi
- dbCacheSizeMb: 100
- storageSize: 1Gi
zookeeper:
replicaCount: 1