You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by ti...@apache.org on 2023/09/07 08:02:14 UTC
[pulsar-site] branch main updated: Fix script and previous content (#701)
This is an automated email from the ASF dual-hosted git repository.
tison pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git
The following commit(s) were added to refs/heads/main by this push:
new e42133a1154 Fix script and previous content (#701)
e42133a1154 is described below
commit e42133a11543ce6ba26258d47679ac56cefc5de5
Author: tison <wa...@gmail.com>
AuthorDate: Thu Sep 7 16:02:08 2023 +0800
Fix script and previous content (#701)
Signed-off-by: tison <wa...@gmail.com>
---
static/reference/3.0.x/cli.md | 10 +-
static/reference/3.1.x/client/README.md | 28 +
static/reference/{next => 3.1.x}/config/README.md | 35 +-
.../config/reference-configuration-bookkeeper.md | 583 +++++++++++++++++++++
.../config/reference-configuration-log4j-shell.md | 37 ++
.../3.1.x/config/reference-configuration-log4j.md | 80 +++
.../config/reference-configuration-zookeeper.md | 83 +++
static/reference/3.1.x/pulsar-admin/README.md | 16 +
static/reference/3.1.x/pulsar-client/README.md | 12 +
static/reference/3.1.x/pulsar-perf/README.md | 12 +
.../reference/3.1.x/pulsar-shell/pulsar-shell.md | 83 ++-
static/reference/3.1.x/pulsar/README.md | 1 +
static/reference/next/cli.md | 10 +-
static/reference/next/client/README.md | 14 +-
static/reference/next/config/README.md | 35 +-
static/reference/next/pulsar-admin/README.md | 14 +-
static/reference/next/pulsar-client/README.md | 12 +-
static/reference/next/pulsar-perf/README.md | 12 +-
static/reference/next/pulsar-shell/pulsar-shell.md | 83 ++-
tools/pytools/bin/reference-doc-generator.py | 19 +-
20 files changed, 1013 insertions(+), 166 deletions(-)
diff --git a/static/reference/3.0.x/cli.md b/static/reference/3.0.x/cli.md
index e79763044aa..520d14e5b03 100644
--- a/static/reference/3.0.x/cli.md
+++ b/static/reference/3.0.x/cli.md
@@ -1,9 +1,3 @@
-Pulsar offers several command-line tools that help you manage Pulsar and BookKeeper, produce and consume messages, administer Pulsar entities, test Pulsar, and more.
+# Welcome
-|Goal|Tool
-|---|---
-Manage Pulsar| - `pulsar` <br/><br/> - [`pulsar-daemon`](/next/pulsar-daemon/pulsar-daemon) <br/><br/> - [`pulsar-shell`](/next/pulsar-shell/pulsar-shell)
-Produce and consume messages | [`pulsar-client`](/next/pulsar-client/pulsar-client)
-Administer Pulsar entities | `pulsar-admin`
-Test Pulsar | [`pulsar-perf`](/next/pulsar-perf/pulsar-perf)
-Manage BookKeeper | [`bookkeper`](/next/bookkeeper/bookkeeper)
\ No newline at end of file
+> Welcome to Pulsar CLI Tools Docs.
diff --git a/static/reference/3.1.x/client/README.md b/static/reference/3.1.x/client/README.md
new file mode 100644
index 00000000000..71df588c6f8
--- /dev/null
+++ b/static/reference/3.1.x/client/README.md
@@ -0,0 +1,28 @@
+## Client Configurations
+
+If you create a Java client, producer, consumer, or reader, you can use the `loadConf` configurations.
+
+- [Client configurations](/@pulsar:version_reference@/client/client-configuration-client)
+- [Producer configurations](/@pulsar:version_reference@/client/client-configuration-producer)
+- [Consumer configurations](/@pulsar:version_reference@/client/client-configuration-consumer)
+- [Reader configurations](/@pulsar:version_reference@/client/client-configuration-reader)
+
+### Set client memory allocator configs using Java system properties
+
+You can set the client memory allocator configurations through Java properties.<br />
+
+| Property | Type | <div style="width: 110pt"> Description </div> [...]
+|-----------------------------------------|--------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| `pulsar.allocator.pooled` | String | If set to `true`, the client uses a direct memory pool. <br /> If set to `false`, the client uses a heap memory without pool. [...]
+| `pulsar.allocator.exit_on_oom` | String | Whether to exit the JVM when OOM happens. [...]
+| `pulsar.allocator.leak_detection` | String | The leak detection policy for Pulsar bytebuf allocator. <li> **Disabled**: No leak detection and no overhead. </li> <li> **Simple**: Instruments 1% of the allocated buffer to track for leaks. </li> <li> **Advanced**: Instruments 1% of the allocated buffer to track for leaks, reporting stack traces of places where the buffer is used. </li> <li> **Paranoid**: Instruments 100% of the allocated buffer to track for leaks, reporting stack tr [...]
+| `pulsar.allocator.out_of_memory_policy` | String | When an OOM occurs, the client throws an exception or fallbacks to heap. [...]
+
+**Example**
+
+```conf
+Dpulsar.allocator.pooled=true
+Dpulsar.allocator.exit_on_oom=false
+Dpulsar.allocator.leak_detection=Disabled
+Dpulsar.allocator.out_of_memory_policy=ThrowException
+```
diff --git a/static/reference/next/config/README.md b/static/reference/3.1.x/config/README.md
similarity index 76%
copy from static/reference/next/config/README.md
copy to static/reference/3.1.x/config/README.md
index 807c89e69b6..02de3826186 100644
--- a/static/reference/next/config/README.md
+++ b/static/reference/3.1.x/config/README.md
@@ -1,6 +1,7 @@
## Pulsar configurations
-You can manage Pulsar configurations through configuration files in the [`conf`](https://github.com/apache/pulsar/tree/master/conf) directory of a Pulsar installation.
+You can manage Pulsar configurations through configuration files in
+the [`conf`](https://github.com/apache/pulsar/tree/master/conf) directory of a Pulsar installation.
- [BookKeeper](/@pulsar:version_reference@/config/reference-configuration-bookkeeper)
- [Broker](/@pulsar:version_reference@/config/reference-configuration-broker)
@@ -14,7 +15,8 @@ You can manage Pulsar configurations through configuration files in the [`conf`]
### Override client configurations
-If you want to override the configurations of clients internal to brokers, websockets, and proxies, you can use the following prefix.
+If you want to override the configurations of clients internal to brokers, websockets, and proxies, you can use the
+following prefix.
| Prefix | Description |
|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
@@ -23,21 +25,23 @@ If you want to override the configurations of clients internal to brokers, webso
> Notes:
> * This override feature only applies to Pulsar 2.10.1 and later versions.
-> * When running the function worker within the broker, you have to configure those clients by using the `functions_worker.yml` file. These prefixed configurations do not apply to any of those clients.
+> * When running the function worker within the broker, you have to configure those clients by using
+ the `functions_worker.yml` file. These prefixed configurations do not apply to any of those clients.
### Set specific configurations using Java system property
You can set specific configurations through Java properties.
-| **Property** | **Description** |
-| ------------------ | ------------------------------------------------------------ |
+| **Property** | **Description** |
+|--------------------|---------------------------------------------------------------------------------------------------------------|
| pulsar.enableUring | Use `io_uring` instead of `epoll` as network IO mode. `-Dpulsar.enableUring=1` means this feature is enabled. |
## Broker load balancing configurations
-Below is a brief summary of configurations for broker load balancing.
+Below is a brief summary of configurations for broker load balancing.
-For detailed descriptions of each configuration, see [Broker load balancing | Configurations](pathname:///reference/#/@pulsar:version_reference@/config/reference-configuration-broker).
+For detailed descriptions of each configuration,
+see [Broker load balancing | Configurations](pathname:///reference/#/@pulsar:version_reference@/config/reference-configuration-broker).
> Note
> Configurations with an asterisk (*) are only available in the extensible load balancer.
@@ -53,14 +57,13 @@ For detailed descriptions of each configuration, see [Broker load balancing | Co
- loadBalancerMemoryResourceWeight
- loadBalancerDirectMemoryResourceWeight
- loadBalancerHistoryResourcePercentage
-
+
### TopK bundle load data
- loadBalancerReportUpdateMinIntervalMillis
- statsUpdateFrequencyInSecs
- loadBalancerMaxNumberOfBundlesInBundleLoadReport*
-
### Bundle-broker assignment
- loadBalancerAverageResourceUsageDifferenceThresholdPercentage
@@ -76,11 +79,13 @@ For detailed descriptions of each configuration, see [Broker load balancing | Co
- loadBalancerSheddingIntervalMinutes
- loadBalancerSheddingGracePeriodMinutes
- loadBalancerBrokerOverloadedThresholdPercentage
-- loadBalancerLoadSheddingStrategy. The default value is [org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder](https://github.com/apache/pulsar/blob/782e91fe327efe2c9c9107d6c679c2837d43935b/conf/broker.conf#L1324). Available values are:
- - `org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder`
- - `org.apache.pulsar.broker.loadbalance.impl.UniformLoadShedder`
- - `org.apache.pulsar.broker.loadbalance.impl.OverloadShedder`
- - `org.apache.pulsar.broker.loadbalance.extensions.scheduler.TransferShedder`
+- loadBalancerLoadSheddingStrategy. The default value
+ is [org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder](https://github.com/apache/pulsar/blob/782e91fe327efe2c9c9107d6c679c2837d43935b/conf/broker.conf#L1324).
+ Available values are:
+ - `org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder`
+ - `org.apache.pulsar.broker.loadbalance.impl.UniformLoadShedder`
+ - `org.apache.pulsar.broker.loadbalance.impl.OverloadShedder`
+ - `org.apache.pulsar.broker.loadbalance.extensions.scheduler.TransferShedder`
- loadBalancerTransferEnabled*
- loadBalancerBrokerLoadTargetStd*
- loadBalancerSheddingConditionHitCountThreshold*
@@ -100,4 +105,4 @@ For detailed descriptions of each configuration, see [Broker load balancing | Co
- loadBalancerNamespaceMaximumBundles
- loadBalancerSplitIntervalMinutes*
- loadBalancerNamespaceBundleSplitConditionHitCountThreshold*
-- loadBalancerMaxNumberOfBundlesToSplitPerCycle*
\ No newline at end of file
+- loadBalancerMaxNumberOfBundlesToSplitPerCycle*
diff --git a/static/reference/3.1.x/config/reference-configuration-bookkeeper.md b/static/reference/3.1.x/config/reference-configuration-bookkeeper.md
new file mode 100644
index 00000000000..28e85dcaf6c
--- /dev/null
+++ b/static/reference/3.1.x/config/reference-configuration-bookkeeper.md
@@ -0,0 +1,583 @@
+# BookKeeper
+
+BookKeeper is a replicated log storage system that Pulsar uses for durable storage of all messages.
+
+### bookiePort
+
+The port on which the bookie server listens.
+
+**Default**: 3181
+
+### allowLoopback
+
+Whether the bookie is allowed to use a loopback interface as its primary interface (that is the interface used to establish its identity). By default, loopback interfaces are not allowed to work as the primary interface. Using a loopback interface as the primary interface usually indicates a configuration error. For example, it’s fairly common in some VPS setups to not configure a hostname or to have the hostname resolve to `127.0.0.1`. If this is the case, then all bookies in the cluste [...]
+
+**Default**: false
+
+### listeningInterface
+
+The network interface on which the bookie listens. By default, the bookie listens on all interfaces.
+
+**Default**: eth0
+
+### advertisedAddress
+
+Configure a specific hostname or IP address that the bookie should use to advertise itself to clients. By default, the bookie advertises either its own IP address or hostname according to the `listeningInterface` and `useHostNameAsBookieID` settings.
+
+**Default**: N/A
+
+### allowMultipleDirsUnderSameDiskPartition
+
+Configure the bookie to enable/disable multiple ledger/index/journal directories in the same filesystem disk partition.
+
+**Default**: false
+
+### minUsableSizeForIndexFileCreation
+
+The minimum safe usable size available in index directory for bookie to create index files while replaying journal at the time of bookie starts in Readonly Mode (in bytes).
+
+**Default**: 1073741824
+
+### journalDirectory
+
+The directory where BookKeeper outputs its write-ahead log (WAL).
+
+**Default**: data/bookkeeper/journal
+
+### journalDirectories
+
+Directories that BookKeeper outputs its write ahead log. Multiple directories are available, being separated by `,`. For example: `journalDirectories=/tmp/bk-journal1,/tmp/bk-journal2`. If `journalDirectories` is set, the bookies skip `journalDirectory` and use this setting directory.
+
+**Default**: /tmp/bk-journal
+
+### ledgerDirectories
+
+The directory where BookKeeper outputs ledger snapshots. This could define multiple directories to store snapshots separated by `,`, for example `ledgerDirectories=/tmp/bk1-data,/tmp/bk2-data`. Ideally, ledger dirs and the journal dir are each in a different device, which reduces the contention between random I/O and sequential write. It is possible to run with a single disk, but performance will be significantly lower.
+
+**Default**: data/bookkeeper/ledgers
+
+### ledgerManagerType
+
+The type of ledger manager used to manage how ledgers are stored, managed, and garbage collected. See [BookKeeper Internals](https://bookkeeper.apache.org/docs/next/getting-started/concepts) for more info.
+
+**Default**: hierarchical
+
+### zkLedgersRootPath
+
+The root ZooKeeper path used to store ledger metadata. This parameter is used by the ZooKeeper-based ledger manager as a root znode to store all ledgers.
+
+**Default**: /ledgers
+
+### ledgerStorageClass
+
+Ledger storage implementation class
+
+**Default**: org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage
+
+### entryLogFilePreallocationEnabled
+
+Enable or disable entry logger preallocation
+
+**Default**: true
+
+### logSizeLimit
+
+Max file size of the entry logger, in bytes. A new entry log file will be created when the old one reaches the file size limitation.
+
+**Default**: 1073741824
+
+### minorCompactionThreshold
+
+Threshold of minor compaction. Entry log files whose remaining size percentage reaches below this threshold will be compacted in a minor compaction. If set to less than zero, the minor compaction is disabled.
+
+**Default**: 0.2
+
+### minorCompactionInterval
+
+Time interval to run minor compaction, in seconds. If set to less than zero, the minor compaction is disabled. Note: should be greater than gcWaitTime.
+
+**Default**: 3600
+
+### majorCompactionThreshold
+
+The threshold of major compaction. Entry log files whose remaining size percentage reaches below this threshold will be compacted in a major compaction. Those entry log files whose remaining size percentage is still higher than the threshold will never be compacted. If set to less than zero, the major compaction is disabled.
+
+**Default**: 0.5
+
+### majorCompactionInterval
+
+The time interval to run major compaction, in seconds. If set to less than zero, the major compaction is disabled. Note: should be greater than gcWaitTime.
+
+**Default**: 86400
+
+### readOnlyModeEnabled
+
+If `readOnlyModeEnabled=true`, then on all full ledger disks, bookie will be converted to read-only mode and serve only read requests. Otherwise the bookie will be shutdown.
+
+**Default**: true
+
+### forceReadOnlyBookie
+
+Whether the bookie is force started in read only mode.
+
+**Default**: false
+
+### persistBookieStatusEnabled
+
+Persist the bookie status locally on the disks. So the bookies can keep their status upon restarts.
+
+**Default**: false
+
+### compactionMaxOutstandingRequests
+
+Sets the maximum number of entries that can be compacted without flushing. When compacting, the entries are written to the entrylog and the new offsets are cached in memory. Once the entrylog is flushed the index is updated with the new offsets. This parameter controls the number of entries added to the entrylog before a flush is forced. A higher value for this parameter means more memory will be used for offsets. Each offset consists of 3 longs. This parameter should not be modified unl [...]
+
+**Default**: 100000
+
+### compactionRate
+
+The rate at which compaction will read entries, in adds per second.
+
+**Default**: 1000
+
+### isThrottleByBytes
+
+Throttle compaction by bytes or by entries.
+
+**Default**: false
+
+### compactionRateByEntries
+
+The rate at which compaction will read entries, in adds per second.
+
+**Default**: 1000
+
+### compactionRateByBytes
+
+Set the rate at which compaction reads entries. The unit is bytes added per second.
+
+**Default**: 1000000
+
+### journalMaxSizeMB
+
+Max file size of journal file, in megabytes. A new journal file will be created when the old one reaches the file size limitation.
+
+**Default**: 2048
+
+### journalMaxBackups
+
+The max number of old journal files to keep. Keeping a number of old journal files would help data recovery in special cases.
+
+**Default**: 5
+
+### journalPreAllocSizeMB
+
+How space to pre-allocate at a time in the journal.
+
+**Default**: 16
+
+### journalWriteBufferSizeKB
+
+The of the write buffers used for the journal.
+
+**Default**: 64
+
+### journalRemoveFromPageCache
+
+Whether pages should be removed from the page cache after force write.
+
+**Default**: true
+
+### journalAdaptiveGroupWrites
+
+Whether to group journal force writes, which optimizes group commit for higher throughput.
+
+**Default**: true
+
+### journalMaxGroupWaitMSec
+
+The maximum latency to impose on a journal write to achieve grouping.
+
+**Default**: 1
+
+### journalAlignmentSize
+
+All the journal writes and commits should be aligned to given size
+
+**Default**: 4096
+
+### journalBufferedWritesThreshold
+
+Maximum writes to buffer to achieve grouping
+
+**Default**: 524288
+
+### journalFlushWhenQueueEmpty
+
+If we should flush the journal when journal queue is empty
+
+**Default**: false
+
+### numJournalCallbackThreads
+
+The number of threads that should handle journal callbacks
+
+**Default**: 8
+
+### openLedgerRereplicationGracePeriod
+
+The grace period, in milliseconds, that the replication worker waits before fencing and replicating a ledger fragment that's still being written to upon bookie failure.
+
+**Default**: 30000
+
+### rereplicationEntryBatchSize
+
+The number of max entries to keep in fragment for re-replication
+
+**Default**: 100
+
+### autoRecoveryDaemonEnabled
+
+Whether the bookie itself can start auto-recovery service.
+
+**Default**: true
+
+### lostBookieRecoveryDelay
+
+How long to wait, in seconds, before starting auto recovery of a lost bookie.
+
+**Default**: 0
+
+### gcWaitTime
+
+How long the interval to trigger next garbage collection, in milliseconds. Since garbage collection is running in background, too frequent gc will heart performance. It is better to give a higher number of gc interval if there is enough disk capacity.
+
+**Default**: 900000
+
+### gcOverreplicatedLedgerWaitTime
+
+How long the interval to trigger next garbage collection of overreplicated ledgers, in milliseconds. This should not be run very frequently since we read the metadata for all the ledgers on the bookie from zk.
+
+**Default**: 86400000
+
+### flushInterval
+
+How long the interval to flush ledger index pages to disk, in milliseconds. Flushing index files will introduce much random disk I/O. If separating journal dir and ledger dirs each on different devices, flushing would not affect performance. But if putting journal dir and ledger dirs on same device, performance degrade significantly on too frequent flushing. You can consider increment flush interval to get better performance, but you need to pay more time on bookie server restart after failure.
+
+**Default**: 60000
+
+### bookieDeathWatchInterval
+
+Interval to watch whether bookie is dead or not, in milliseconds
+
+**Default**: 1000
+
+### allowStorageExpansion
+
+Allow the bookie storage to expand. Newly added ledger and index dirs must be empty.
+
+**Default**: false
+
+### zkServers
+
+A list of one of more servers on which zookeeper is running. The server list can be comma separated values, for example: zkServers=zk1:2181,zk2:2181,zk3:2181.
+
+**Default**: localhost:2181
+
+### zkTimeout
+
+ZooKeeper client session timeout in milliseconds Bookie server will exit if it received SESSION_EXPIRED because it was partitioned off from ZooKeeper for more than the session timeout JVM garbage collection, disk I/O will cause SESSION_EXPIRED. Increment this value could help avoiding this issue
+
+**Default**: 30000
+
+### zkRetryBackoffStartMs
+
+The start time that the Zookeeper client backoff retries in milliseconds.
+
+**Default**: 1000
+
+### zkRetryBackoffMaxMs
+
+The maximum time that the Zookeeper client backoff retries in milliseconds.
+
+**Default**: 10000
+
+### zkEnableSecurity
+
+Set ACLs on every node written on ZooKeeper, allowing users to read and write BookKeeper metadata stored on ZooKeeper. In order to make ACLs work you need to setup ZooKeeper JAAS authentication. All the bookies and Client need to share the same user, and this is usually done using Kerberos authentication. See ZooKeeper documentation.
+
+**Default**: false
+
+### httpServerEnabled
+
+The flag enables/disables starting the admin http server.
+
+**Default**: false
+
+### httpServerPort
+
+The HTTP server port to listen on. By default, the value is `8080`. If you want to keep it consistent with the Prometheus stats provider, you can set it to `8000`.
+
+**Default**: 8080
+
+### httpServerClass
+
+The http server class.
+
+**Default**: org.apache.bookkeeper.http.vertx.VertxHttpServer
+
+### serverTcpNoDelay
+
+This settings is used to enabled/disabled Nagle’s algorithm, which is a means of improving the efficiency of TCP/IP networks by reducing the number of packets that need to be sent over the network. If you are sending many small messages, such that more than one can fit in a single IP packet, setting server.tcpnodelay to false to enable Nagle algorithm can provide better performance.
+
+**Default**: true
+
+### serverSockKeepalive
+
+This setting is used to send keep-alive messages on connection-oriented sockets.
+
+**Default**: true
+
+### serverTcpLinger
+
+The socket linger timeout on close. When enabled, a close or shutdown will not return until all queued messages for the socket have been successfully sent or the linger timeout has been reached. Otherwise, the call returns immediately and the closing is done in the background.
+
+**Default**: 0
+
+### byteBufAllocatorSizeMax
+
+The maximum buf size of the received ByteBuf allocator.
+
+**Default**: 1048576
+
+### nettyMaxFrameSizeBytes
+
+The maximum netty frame size in bytes. Any message received larger than this will be rejected.
+
+**Default**: 5253120
+
+### openFileLimit
+
+Max number of ledger index files could be opened in bookie server If number of ledger index files reaches this limitation, bookie server started to swap some ledgers from memory to disk. Too frequent swap will affect performance. You can tune this number to gain performance according your requirements.
+
+**Default**: 0
+
+### pageSize
+
+Size of a index page in ledger cache, in bytes A larger index page can improve performance writing page to disk, which is efficient when you have small number of ledgers and these ledgers have similar number of entries. If you have large number of ledgers and each ledger has fewer entries, smaller index page would improve memory usage.
+
+**Default**: 8192
+
+### pageLimit
+
+How many index pages provided in ledger cache If number of index pages reaches this limitation, bookie server starts to swap some ledgers from memory to disk. You can increment this value when you found swap became more frequent. But make sure pageLimit\*pageSize should not more than JVM max memory limitation, otherwise you would got OutOfMemoryException. In general, incrementing pageLimit, using smaller index page would gain better performance in lager number of ledgers with fewer entri [...]
+
+**Default**: 0
+
+### readOnlyModeEnabled
+
+If all ledger directories configured are full, then support only read requests for clients. If "readOnlyModeEnabled=true" then on all ledger disks full, bookie will be converted to read-only mode and serve only read requests. Otherwise the bookie will be shutdown. By default this will be disabled.
+
+**Default**: true
+
+### diskUsageThreshold
+
+For each ledger dir, maximum disk space which can be used. Default is 0.95f. i.e. 95% of disk can be used at most after which nothing will be written to that partition. If all ledger dir partitions are full, then bookie will turn to readonly mode if ‘readOnlyModeEnabled=true’ is set, else it will shutdown. Valid values should be in between 0 and 1 (exclusive).
+
+**Default**: 0.95
+
+### diskCheckInterval
+
+Disk check interval in milli seconds, interval to check the ledger dirs usage.
+
+**Default**: 10000
+
+### auditorPeriodicCheckInterval
+
+Interval at which the auditor will do a check of all ledgers in the cluster. By default this runs once a week. The interval is set in seconds. To disable the periodic check completely, set this to 0. Note that periodic checking will put extra load on the cluster, so it should not be run more frequently than once a day.
+
+**Default**: 604800
+
+### sortedLedgerStorageEnabled
+
+Whether sorted-ledger storage is enabled.
+
+**Default**: true
+
+### auditorPeriodicBookieCheckInterval
+
+The interval between auditor bookie checks. The auditor bookie check, checks ledger metadata to see which bookies should contain entries for each ledger. If a bookie which should contain entries is unavailable, thea the ledger containing that entry is marked for recovery. Setting this to 0 disabled the periodic check. Bookie checks will still run when a bookie fails. The interval is specified in seconds.
+
+**Default**: 86400
+
+### numAddWorkerThreads
+
+The number of threads that should handle write requests. if zero, the writes would be handled by netty threads directly.
+
+**Default**: 0
+
+### numReadWorkerThreads
+
+The number of threads that should handle read requests. if zero, the reads would be handled by netty threads directly.
+
+**Default**: 8
+
+### numHighPriorityWorkerThreads
+
+The umber of threads that should be used for high priority requests (i.e. recovery reads and adds, and fencing).
+
+**Default**: 8
+
+### maxPendingReadRequestsPerThread
+
+If read workers threads are enabled, limit the number of pending requests, to avoid the executor queue to grow indefinitely.
+
+**Default**: 2500
+
+### maxPendingAddRequestsPerThread
+
+The limited number of pending requests, which is used to avoid the executor queue to grow indefinitely when add workers threads are enabled.
+
+**Default**: 10000
+
+### isForceGCAllowWhenNoSpace
+
+Whether force compaction is allowed when the disk is full or almost full. Forcing GC could get some space back, but could also fill up the disk space more quickly. This is because new log files are created before GC, while old garbage log files are deleted after GC.
+
+**Default**: false
+
+### verifyMetadataOnGC
+
+True if the bookie should double check `readMetadata` prior to GC.
+
+**Default**: false
+
+### flushEntrylogBytes
+
+Entry log flush interval in bytes. Flushing in smaller chunks but more frequently reduces spikes in disk I/O. Flushing too frequently may also affect performance negatively.
+
+**Default**: 268435456
+
+### readBufferSizeBytes
+
+The number of bytes we should use as capacity for BufferedReadChannel.
+
+**Default**: 4096
+
+### writeBufferSizeBytes
+
+The number of bytes used as capacity for the write buffer
+
+**Default**: 65536
+
+### useHostNameAsBookieID
+
+Whether the bookie should use its hostname to register with the coordination service (e.g.: zookeeper service). When false, bookie will use its ip address for the registration.
+
+**Default**: false
+
+### bookieId
+
+If you want to custom a bookie ID or use a dynamic network address for the bookie, you can set the `bookieId`. <br /><br />Bookie advertises itself using the `bookieId` rather than the `BookieSocketAddress` (`hostname:port` or `IP:port`). If you set the `bookieId`, then the `useHostNameAsBookieID` does not take effect.<br /><br />The `bookieId` is a non-empty string that can contain ASCII digits and letters ([a-zA-Z9-0]), colons, dashes, and dots. <br /><br />For more information about ` [...]
+
+**Default**: N/A
+
+### allowEphemeralPorts
+
+Whether the bookie is allowed to use an ephemeral port (port 0) as its server port. By default, an ephemeral port is not allowed. Using an ephemeral port as the service port usually indicates a configuration error. However, in unit tests, using an ephemeral port will address port conflict problems and allow running tests in parallel.
+
+**Default**: false
+
+### enableLocalTransport
+
+Whether the bookie is allowed to listen for the BookKeeper clients executed on the local JVM.
+
+**Default**: false
+
+### disableServerSocketBind
+
+Whether the bookie is allowed to disable bind on network interfaces. This bookie will be available only to BookKeeper clients executed on the local JVM.
+
+**Default**: false
+
+### skipListArenaChunkSize
+
+The number of bytes that we should use as chunk allocation for `org.apache.bookkeeper.bookie.SkipListArena`.
+
+**Default**: 4194304
+
+### skipListArenaMaxAllocSize
+
+The maximum size that we should allocate from the skiplist arena. Allocations larger than this should be allocated directly by the VM to avoid fragmentation.
+
+**Default**: 131072
+
+### bookieAuthProviderFactoryClass
+
+The factory class name of the bookie authentication provider. If this is null, then there is no authentication.
+
+**Default**: null
+
+### statsProviderClass
+
+ [...]
+
+ [...]
+
+### prometheusStatsHttpPort
+
+ [...]
+
+ [...]
+
+### dbStorage_writeCacheMaxSizeMb
+
+Size of Write Cache. Memory is allocated from JVM direct memory. Write cache is used to buffer entries before flushing into the entry log. For good performance, it should be big enough to hold a substantial amount of entries in the flush interval.
+
+**Default**: 25% of direct memory
+
+### dbStorage_readAheadCacheMaxSizeMb
+
+Size of Read cache. Memory is allocated from JVM direct memory. This read cache is pre-filled doing read-ahead whenever a cache miss happens. By default, it is allocated to 25% of the available direct memory.
+
+**Default**: N/A
+
+### dbStorage_readAheadCacheBatchSize
+
+How many entries to pre-fill in cache after a read cache miss
+
+**Default**: 1000
+
+### dbStorage_rocksDB_blockCacheSize
+
+Size of RocksDB block-cache. For best performance, this cache should be big enough to hold a significant portion of the index database which can reach ~2GB in some cases. By default, it uses 10% of direct memory.
+
+**Default**: N/A
+
+### dbStorage_rocksDB_writeBufferSizeMB
+
+**Default**: 64
+
+### dbStorage_rocksDB_sstSizeInMB
+
+**Default**: 64
+
+### dbStorage_rocksDB_blockSize
+
+**Default**: 65536
+
+### dbStorage_rocksDB_bloomFilterBitsPerKey
+
+**Default**: 10
+
+### dbStorage_rocksDB_numLevels
+
+**Default**: -1
+
+### dbStorage_rocksDB_numFilesInLevel0
+
+**Default**: 4
+
+### dbStorage_rocksDB_maxSizeInLevel1MB
+
+**Default**: 256
diff --git a/static/reference/3.1.x/config/reference-configuration-log4j-shell.md b/static/reference/3.1.x/config/reference-configuration-log4j-shell.md
new file mode 100644
index 00000000000..43fca015c0d
--- /dev/null
+++ b/static/reference/3.1.x/config/reference-configuration-log4j-shell.md
@@ -0,0 +1,37 @@
+# Log4j shell
+
+### bookkeeper.root.logger
+
+**Default**: ERROR,CONSOLE
+
+### log4j.rootLogger
+
+**Default**: ${bookkeeper.root.logger}
+
+### log4j.appender.CONSOLE
+
+**Default**: org.apache.log4j.ConsoleAppender
+
+### log4j.appender.CONSOLE.Threshold
+
+**Default**: DEBUG
+
+### log4j.appender.CONSOLE.layout
+
+**Default**: org.apache.log4j.PatternLayout
+
+### log4j.appender.CONSOLE.layout.ConversionPattern
+
+**Default**: %d{ABSOLUTE} %-5p %m%n
+
+### log4j.logger.org.apache.zookeeper
+
+**Default**: ERROR
+
+### log4j.logger.org.apache.bookkeeper
+
+**Default**: ERROR
+
+### log4j.logger.org.apache.bookkeeper.bookie.BookieShell
+
+**Default**: INFO
diff --git a/static/reference/3.1.x/config/reference-configuration-log4j.md b/static/reference/3.1.x/config/reference-configuration-log4j.md
new file mode 100644
index 00000000000..6aa18e03a23
--- /dev/null
+++ b/static/reference/3.1.x/config/reference-configuration-log4j.md
@@ -0,0 +1,80 @@
+# Log4j
+
+You can set the log level and configuration in the [log4j2.yaml](https://github.com/apache/pulsar/blob/d557e0aa286866363bc6261dec87790c055db1b0/conf/log4j2.yaml#L155) file. The following logging configuration parameters are available.
+
+### pulsar.root.logger
+
+**Default**: WARN,CONSOLE
+
+### pulsar.log.dir
+
+**Default**: logs
+
+### pulsar.log.file
+
+**Default**: pulsar.log
+
+### log4j.rootLogger
+
+**Default**: ${pulsar.root.logger}
+
+### log4j.appender.CONSOLE
+
+**Default**: org.apache.log4j.ConsoleAppender
+
+### log4j.appender.CONSOLE.Threshold
+
+**Default**: DEBUG
+
+### log4j.appender.CONSOLE.layout
+
+**Default**: org.apache.log4j.PatternLayout
+
+### log4j.appender.CONSOLE.layout.ConversionPattern
+
+**Default**: %d{ISO8601} - %-5p - [%t:%C{1}@%L] - %m%n
+
+### log4j.appender.ROLLINGFILE
+
+**Default**: org.apache.log4j.DailyRollingFileAppender
+
+### log4j.appender.ROLLINGFILE.Threshold
+
+**Default**: DEBUG
+
+### log4j.appender.ROLLINGFILE.File
+
+**Default**: ${pulsar.log.dir}/${pulsar.log.file}
+
+### log4j.appender.ROLLINGFILE.layout
+
+**Default**: org.apache.log4j.PatternLayout
+
+### log4j.appender.ROLLINGFILE.layout.ConversionPattern
+
+**Default**: %d{ISO8601} - %-5p [%t:%C{1}@%L] - %m%n
+
+### log4j.appender.TRACEFILE
+
+**Default**: org.apache.log4j.FileAppender
+
+### log4j.appender.TRACEFILE.Threshold
+
+**Default**: TRACE
+
+### log4j.appender.TRACEFILE.File
+
+**Default**: pulsar-trace.log
+
+### log4j.appender.TRACEFILE.layout
+
+**Default**: org.apache.log4j.PatternLayout
+
+### log4j.appender.TRACEFILE.layout.ConversionPattern
+
+**Default**: %d{ISO8601} - %-5p [%t:%C{1}@%L][%x] - %m%n
+
+> Note: 'topic' in log4j2.appender is configurable.
+>
+> - If you want to append all logs to a single topic, set the same topic name.
+> - If you want to append logs to different topics, you can set different topic names.
diff --git a/static/reference/3.1.x/config/reference-configuration-zookeeper.md b/static/reference/3.1.x/config/reference-configuration-zookeeper.md
new file mode 100644
index 00000000000..0e4b6b43f54
--- /dev/null
+++ b/static/reference/3.1.x/config/reference-configuration-zookeeper.md
@@ -0,0 +1,83 @@
+# ZooKeeper
+
+ZooKeeper handles a broad range of essential configuration- and coordination-related tasks for Pulsar. The default configuration file for ZooKeeper is in the `conf/zookeeper.conf` file in your Pulsar installation. The following parameters are available:
+
+### tickTime
+
+The tick is the basic unit of time in ZooKeeper, measured in milliseconds and used to regulate things like heartbeats and timeouts. tickTime is the length of a single tick.
+
+**Default**: 2000
+
+### initLimit
+
+The maximum time, in ticks, that the leader ZooKeeper server allows follower ZooKeeper servers to successfully connect and sync. The tick time is set in milliseconds using the tickTime parameter.
+
+**Default**: 10
+
+### syncLimit
+
+The maximum time, in ticks, that a follower ZooKeeper server is allowed to sync with other ZooKeeper servers. The tick time is set in milliseconds using the tickTime parameter.
+
+**Default**: 5
+
+### dataDir
+
+The location where ZooKeeper will store in-memory database snapshots as well as the transaction log of updates to the database.
+
+**Default**: data/zookeeper
+
+### clientPort
+
+The port on which the ZooKeeper server will listen for connections.
+
+**Default**: 2181
+
+### admin.enableServer
+
+The port at which the admin listens.
+
+**Default**: true
+
+### admin.serverPort
+
+The port at which the admin listens.
+
+**Default**: 9990
+
+### autopurge.snapRetainCount
+
+In ZooKeeper, auto purge determines how many recent snapshots of the database stored in dataDir to retain within the time interval specified by autopurge.purgeInterval (while deleting the rest).
+
+**Default**: 3
+
+### autopurge.purgeInterval
+
+The time interval, in hours, by which the ZooKeeper database purge task is triggered. Setting to a non-zero number will enable auto purge; setting to 0 will disable. Read this guide before enabling auto purge.
+
+**Default**: 1
+
+### forceSync
+
+Requires updates to be synced to media of the transaction log before finishing processing the update. If this option is set to 'no', ZooKeeper will not require updates to be synced to the media. WARNING: it's not recommended to run a production ZK cluster with `forceSync` disabled.
+
+**Default**: yes
+
+### maxClientCnxns
+
+The maximum number of client connections. Increase this if you need to handle more ZooKeeper clients.
+
+**Default**: 60
+
+---
+
+In addition to the parameters above, configuring ZooKeeper for Pulsar involves adding a `server.N` line to the `conf/zookeeper.conf` file for each node in the ZooKeeper cluster, where `N` is the number of the ZooKeeper node. Here's an example of a three-node ZooKeeper cluster:
+
+```properties
+
+server.1=zk1.us-west.example.com:2888:3888
+server.2=zk2.us-west.example.com:2888:3888
+server.3=zk3.us-west.example.com:2888:3888
+
+```
+
+> We strongly recommend consulting the [ZooKeeper Administrator's Guide](https://zookeeper.apache.org/doc/current/zookeeperAdmin.html) for a more thorough and comprehensive introduction to ZooKeeper configuration
diff --git a/static/reference/3.1.x/pulsar-admin/README.md b/static/reference/3.1.x/pulsar-admin/README.md
new file mode 100644
index 00000000000..63a4c92b5ad
--- /dev/null
+++ b/static/reference/3.1.x/pulsar-admin/README.md
@@ -0,0 +1,16 @@
+`pulsar-admin` is a tool used to administer Pulsar entities.
+
+## Environment variables
+
+You can use the following environment variables to configure `pulsar-admin`.
+
+| Variable | Description | Default |
+|--------------------------|------------------------------------|------------------|
+| `PULSAR_LOG_CONF` | Log4j configuration file | conf/log4j2.yaml |
+| `PULSAR_CLIENT_CONF` | Configuration file for the client | conf/client.conf |
+| `PULSAR_EXTRA_OPTS` | Extra options passed to the JVM | N/A |
+| `PULSAR_EXTRA_CLASSPATH` | Extra paths for Pulsar's classpath | N/A |
+
+## Related topics
+
+- [Pulsar admin API](docs/admin-api-overview)
diff --git a/static/reference/3.1.x/pulsar-client/README.md b/static/reference/3.1.x/pulsar-client/README.md
new file mode 100644
index 00000000000..d25591d2d24
--- /dev/null
+++ b/static/reference/3.1.x/pulsar-client/README.md
@@ -0,0 +1,12 @@
+`pulsar-client` is a tool used to produce and consume messages.
+
+## Environment variables
+
+You can use the following environment variables to configure `pulsar-client`.
+
+| Variable | Description | Default |
+|--------------------------|------------------------------------|------------------|
+| `PULSAR_LOG_CONF` | Log4j configuration file | conf/log4j2.yaml |
+| `PULSAR_CLIENT_CONF` | Configuration file for the client | conf/client.conf |
+| `PULSAR_EXTRA_OPTS` | Extra options passed to the JVM | N/A |
+| `PULSAR_EXTRA_CLASSPATH` | Extra paths for Pulsar's classpath | N/A |
diff --git a/static/reference/3.1.x/pulsar-perf/README.md b/static/reference/3.1.x/pulsar-perf/README.md
new file mode 100644
index 00000000000..3a9ee62e2ef
--- /dev/null
+++ b/static/reference/3.1.x/pulsar-perf/README.md
@@ -0,0 +1,12 @@
+`pulsar-perf` is a tool used to test the performance of Pulsar brokers.
+
+## Environment variables
+
+You can use the following environment variables to configure `pulsar-perf`.
+
+| Variable | Description | Default |
+|--------------------------|------------------------------------|------------------|
+| `PULSAR_LOG_CONF` | Log4j configuration file | conf/log4j2.yaml |
+| `PULSAR_CLIENT_CONF` | Configuration file for the client | conf/client.conf |
+| `PULSAR_EXTRA_OPTS` | Extra options passed to the JVM | N/A |
+| `PULSAR_EXTRA_CLASSPATH` | Extra paths for Pulsar's classpath | N/A |
diff --git a/static/reference/3.1.x/pulsar-shell/pulsar-shell.md b/static/reference/3.1.x/pulsar-shell/pulsar-shell.md
index 1e50f7fe2e6..c7d6e227a09 100644
--- a/static/reference/3.1.x/pulsar-shell/pulsar-shell.md
+++ b/static/reference/3.1.x/pulsar-shell/pulsar-shell.md
@@ -1,6 +1,6 @@
# pulsar-shell
-[Pulsar shell](https://pulsar.apache.org/docs/next/administration-pulsar-shell) tool.
+[Pulsar shell](pathname:///docs/@pulsar:version@/administration-pulsar-shell) tool.
### Interactive mode
@@ -10,14 +10,13 @@ Usage
pulsar-shell
```
-| Flag | Description | Default |
-|--------------------|---------------------------------------------------------------------------|------------------|
-| `-c`, `--config` | Client configuration file. It is used as a `default` config. | conf/client.conf |
-| `--fail-on-error` | If true, the shell is interrupted when a command throws an exception. | false |
-| `-h`, `--help` | Show this help. | |
+| Flag | Description | Default |
+|-------------------|-----------------------------------------------------------------------|------------------|
+| `-c`, `--config` | Client configuration file. It is used as a `default` config. | conf/client.conf |
+| `--fail-on-error` | If true, the shell is interrupted when a command throws an exception. | false |
+| `-h`, `--help` | Show this help. | |
-
-### Non interactive mode
+### Non-interactive mode
Usage
@@ -27,22 +26,21 @@ pulsar-shell -e [COMMAND]
echo "[COMMAND]" | pulsar-shell -
```
-| Flag | Description | Default |
-|---------------------------|-----------------------------------------------------------------------------------------------------|-----------------|
-| `-c`, `--config` | Client configuration file. It is used as a `default` config. | conf/client.conf |
-| `--fail-on-error` | If true, the shell is interrupted when a command throws an exception. | false |
-| `-np`, `--no-progress` | Display raw output of the commands without the fancy progress visualization. | false |
-| `-f`, `--filename` | Input filename with a list of commands to be executed. Each command must be separated by a newline. | |
-| `-e`, `--execute-command` | Execute this command and exit. | |
-| `-` | Read commands from the standard input. | |
-| `-h`, `--help` | Show this help. | |
-
+| Flag | Description | Default |
+|---------------------------|-----------------------------------------------------------------------------------------------------|------------------|
+| `-c`, `--config` | Client configuration file. It is used as a `default` config. | conf/client.conf |
+| `--fail-on-error` | If true, the shell is interrupted when a command throws an exception. | false |
+| `-np`, `--no-progress` | Display raw output of the commands without the fancy progress visualization. | false |
+| `-f`, `--filename` | Input filename with a list of commands to be executed. Each command must be separated by a newline. | |
+| `-e`, `--execute-command` | Execute this command and exit. | |
+| `-` | Read commands from the standard input. | |
+| `-h`, `--help` | Show this help. | |
Commands
-* `admin` - See [Admin API](admin-api-overview.md)
-* `client` - See [pulsar-client](#pulsar-client)
-* `config`
+* `admin` - See [Admin API](pathname:///docs/@pulsar:version@/admin-api-overview)
+* `client` - See [pulsar-client](../pulsar-client/README.md)
+* `config`
### config
@@ -64,11 +62,11 @@ Create a new configuration.
default(localhost)> config create --file ./conf/client.conf mycluster
```
-| Flag | Description | Default |
-|----------|--------------------------|-----------------|
-| `--file` | File path of the config. | |
-| `--url` | URL of the config. | |
-| `--value` | Inline value of the config. Base64-encoded value is supported with the prefix `base64:`. | |
+| Flag | Description | Default |
+|-----------|------------------------------------------------------------------------------------------|---------|
+| `--file` | File path of the config. | |
+| `--url` | URL of the config. | |
+| `--value` | Inline value of the config. Base64-encoded value is supported with the prefix `base64:`. | |
#### clone
@@ -78,9 +76,9 @@ Create a new configuration cloning an existing one.
default(localhost)> config clone mycluster --name mycluster2
```
-| Flag | Description | Default |
-|----------|--------------------------|-----------------|
-| `--name` | Name of the new config. | |
+| Flag | Description | Default |
+|----------|-------------------------|---------|
+| `--name` | Name of the new config. | |
#### update
@@ -90,11 +88,11 @@ Update an existing configuration.
default(localhost)> config update --file ./conf/client.conf mycluster
```
-| Flag | Description | Default |
-|----------|--------------------------|-----------------|
-| `--file` | File path of the config. | |
-| `--url` | URL of the config. | |
-| `--value` | Inline value of the config. Base64-encoded value is supported with the prefix `base64:`. | |
+| Flag | Description | Default |
+|-----------|------------------------------------------------------------------------------------------|---------|
+| `--file` | File path of the config. | |
+| `--url` | URL of the config. | |
+| `--value` | Inline value of the config. Base64-encoded value is supported with the prefix `base64:`. | |
#### set-property
@@ -104,11 +102,10 @@ Set a value for a specified configuration property.
default(localhost)> config set-property -p webServiceUrl -v http://<cluster-hostname> mycluster
```
-| Flag | Description | Default |
-|--------------------|-----------------------------|-----------------|
-| `-p`, `--property` | Property name to update. | |
-| `-v`, `--value` | New value for the property. | |
-
+| Flag | Description | Default |
+|--------------------|-----------------------------|---------|
+| `-p`, `--property` | Property name to update. | |
+| `-v`, `--value` | New value for the property. | |
#### get-property
@@ -118,10 +115,9 @@ Get the value for a specified configuration property.
default(localhost)> config get-property -p webServiceUrl mycluster
```
-| Flag | Description | Default |
-|--------------------|-----------------------------|-----------------|
-| `-p`, `--property` | Property name to update. | |
-
+| Flag | Description | Default |
+|--------------------|--------------------------|---------|
+| `-p`, `--property` | Property name to update. | |
#### view
@@ -139,7 +135,6 @@ Delete a config. You can't delete a config if it's currently used.
default(localhost)> config delete mycluster
```
-
#### list
List all the configuration names.
diff --git a/static/reference/3.1.x/pulsar/README.md b/static/reference/3.1.x/pulsar/README.md
new file mode 100644
index 00000000000..bf2c5426160
--- /dev/null
+++ b/static/reference/3.1.x/pulsar/README.md
@@ -0,0 +1 @@
+> Docs for `pulsar`.
diff --git a/static/reference/next/cli.md b/static/reference/next/cli.md
index e79763044aa..520d14e5b03 100644
--- a/static/reference/next/cli.md
+++ b/static/reference/next/cli.md
@@ -1,9 +1,3 @@
-Pulsar offers several command-line tools that help you manage Pulsar and BookKeeper, produce and consume messages, administer Pulsar entities, test Pulsar, and more.
+# Welcome
-|Goal|Tool
-|---|---
-Manage Pulsar| - `pulsar` <br/><br/> - [`pulsar-daemon`](/next/pulsar-daemon/pulsar-daemon) <br/><br/> - [`pulsar-shell`](/next/pulsar-shell/pulsar-shell)
-Produce and consume messages | [`pulsar-client`](/next/pulsar-client/pulsar-client)
-Administer Pulsar entities | `pulsar-admin`
-Test Pulsar | [`pulsar-perf`](/next/pulsar-perf/pulsar-perf)
-Manage BookKeeper | [`bookkeper`](/next/bookkeeper/bookkeeper)
\ No newline at end of file
+> Welcome to Pulsar CLI Tools Docs.
diff --git a/static/reference/next/client/README.md b/static/reference/next/client/README.md
index 718f7ad84b7..71df588c6f8 100644
--- a/static/reference/next/client/README.md
+++ b/static/reference/next/client/README.md
@@ -11,12 +11,12 @@ If you create a Java client, producer, consumer, or reader, you can use the `loa
You can set the client memory allocator configurations through Java properties.<br />
-| Property | Type | <div style="width: 110pt"> Description </div> | Default | Available values |
-|---|---|---|---|---|
-|`pulsar.allocator.pooled` | String | If set to `true`, the client uses a direct memory pool. <br /> If set to `false`, the client uses a heap memory without pool. | true | <li> true </li> <li> false </li> |
-|`pulsar.allocator.exit_on_oom` | String | Whether to exit the JVM when OOM happens. | false | <li> true </li> <li> false </li>|
-|`pulsar.allocator.leak_detection` | String | The leak detection policy for Pulsar bytebuf allocator. <li> **Disabled**: No leak detection and no overhead. </li> <li> **Simple**: Instruments 1% of the allocated buffer to track for leaks. </li> <li> **Advanced**: Instruments 1% of the allocated buffer to track for leaks, reporting stack traces of places where the buffer is used. </li> <li> **Paranoid**: Instruments 100% of the allocated buffer to track for leaks, reporting stack traces of [...]
-|`pulsar.allocator.out_of_memory_policy` | String | When an OOM occurs, the client throws an exception or fallbacks to heap. | FallbackToHeap | <li> ThrowException </li> <li> FallbackToHeap </li>|
+| Property | Type | <div style="width: 110pt"> Description </div> [...]
+|-----------------------------------------|--------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| `pulsar.allocator.pooled` | String | If set to `true`, the client uses a direct memory pool. <br /> If set to `false`, the client uses a heap memory without pool. [...]
+| `pulsar.allocator.exit_on_oom` | String | Whether to exit the JVM when OOM happens. [...]
+| `pulsar.allocator.leak_detection` | String | The leak detection policy for Pulsar bytebuf allocator. <li> **Disabled**: No leak detection and no overhead. </li> <li> **Simple**: Instruments 1% of the allocated buffer to track for leaks. </li> <li> **Advanced**: Instruments 1% of the allocated buffer to track for leaks, reporting stack traces of places where the buffer is used. </li> <li> **Paranoid**: Instruments 100% of the allocated buffer to track for leaks, reporting stack tr [...]
+| `pulsar.allocator.out_of_memory_policy` | String | When an OOM occurs, the client throws an exception or fallbacks to heap. [...]
**Example**
@@ -25,4 +25,4 @@ Dpulsar.allocator.pooled=true
Dpulsar.allocator.exit_on_oom=false
Dpulsar.allocator.leak_detection=Disabled
Dpulsar.allocator.out_of_memory_policy=ThrowException
-```
\ No newline at end of file
+```
diff --git a/static/reference/next/config/README.md b/static/reference/next/config/README.md
index 807c89e69b6..02de3826186 100644
--- a/static/reference/next/config/README.md
+++ b/static/reference/next/config/README.md
@@ -1,6 +1,7 @@
## Pulsar configurations
-You can manage Pulsar configurations through configuration files in the [`conf`](https://github.com/apache/pulsar/tree/master/conf) directory of a Pulsar installation.
+You can manage Pulsar configurations through configuration files in
+the [`conf`](https://github.com/apache/pulsar/tree/master/conf) directory of a Pulsar installation.
- [BookKeeper](/@pulsar:version_reference@/config/reference-configuration-bookkeeper)
- [Broker](/@pulsar:version_reference@/config/reference-configuration-broker)
@@ -14,7 +15,8 @@ You can manage Pulsar configurations through configuration files in the [`conf`]
### Override client configurations
-If you want to override the configurations of clients internal to brokers, websockets, and proxies, you can use the following prefix.
+If you want to override the configurations of clients internal to brokers, websockets, and proxies, you can use the
+following prefix.
| Prefix | Description |
|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
@@ -23,21 +25,23 @@ If you want to override the configurations of clients internal to brokers, webso
> Notes:
> * This override feature only applies to Pulsar 2.10.1 and later versions.
-> * When running the function worker within the broker, you have to configure those clients by using the `functions_worker.yml` file. These prefixed configurations do not apply to any of those clients.
+> * When running the function worker within the broker, you have to configure those clients by using
+ the `functions_worker.yml` file. These prefixed configurations do not apply to any of those clients.
### Set specific configurations using Java system property
You can set specific configurations through Java properties.
-| **Property** | **Description** |
-| ------------------ | ------------------------------------------------------------ |
+| **Property** | **Description** |
+|--------------------|---------------------------------------------------------------------------------------------------------------|
| pulsar.enableUring | Use `io_uring` instead of `epoll` as network IO mode. `-Dpulsar.enableUring=1` means this feature is enabled. |
## Broker load balancing configurations
-Below is a brief summary of configurations for broker load balancing.
+Below is a brief summary of configurations for broker load balancing.
-For detailed descriptions of each configuration, see [Broker load balancing | Configurations](pathname:///reference/#/@pulsar:version_reference@/config/reference-configuration-broker).
+For detailed descriptions of each configuration,
+see [Broker load balancing | Configurations](pathname:///reference/#/@pulsar:version_reference@/config/reference-configuration-broker).
> Note
> Configurations with an asterisk (*) are only available in the extensible load balancer.
@@ -53,14 +57,13 @@ For detailed descriptions of each configuration, see [Broker load balancing | Co
- loadBalancerMemoryResourceWeight
- loadBalancerDirectMemoryResourceWeight
- loadBalancerHistoryResourcePercentage
-
+
### TopK bundle load data
- loadBalancerReportUpdateMinIntervalMillis
- statsUpdateFrequencyInSecs
- loadBalancerMaxNumberOfBundlesInBundleLoadReport*
-
### Bundle-broker assignment
- loadBalancerAverageResourceUsageDifferenceThresholdPercentage
@@ -76,11 +79,13 @@ For detailed descriptions of each configuration, see [Broker load balancing | Co
- loadBalancerSheddingIntervalMinutes
- loadBalancerSheddingGracePeriodMinutes
- loadBalancerBrokerOverloadedThresholdPercentage
-- loadBalancerLoadSheddingStrategy. The default value is [org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder](https://github.com/apache/pulsar/blob/782e91fe327efe2c9c9107d6c679c2837d43935b/conf/broker.conf#L1324). Available values are:
- - `org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder`
- - `org.apache.pulsar.broker.loadbalance.impl.UniformLoadShedder`
- - `org.apache.pulsar.broker.loadbalance.impl.OverloadShedder`
- - `org.apache.pulsar.broker.loadbalance.extensions.scheduler.TransferShedder`
+- loadBalancerLoadSheddingStrategy. The default value
+ is [org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder](https://github.com/apache/pulsar/blob/782e91fe327efe2c9c9107d6c679c2837d43935b/conf/broker.conf#L1324).
+ Available values are:
+ - `org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder`
+ - `org.apache.pulsar.broker.loadbalance.impl.UniformLoadShedder`
+ - `org.apache.pulsar.broker.loadbalance.impl.OverloadShedder`
+ - `org.apache.pulsar.broker.loadbalance.extensions.scheduler.TransferShedder`
- loadBalancerTransferEnabled*
- loadBalancerBrokerLoadTargetStd*
- loadBalancerSheddingConditionHitCountThreshold*
@@ -100,4 +105,4 @@ For detailed descriptions of each configuration, see [Broker load balancing | Co
- loadBalancerNamespaceMaximumBundles
- loadBalancerSplitIntervalMinutes*
- loadBalancerNamespaceBundleSplitConditionHitCountThreshold*
-- loadBalancerMaxNumberOfBundlesToSplitPerCycle*
\ No newline at end of file
+- loadBalancerMaxNumberOfBundlesToSplitPerCycle*
diff --git a/static/reference/next/pulsar-admin/README.md b/static/reference/next/pulsar-admin/README.md
index 3180c0ef14d..63a4c92b5ad 100644
--- a/static/reference/next/pulsar-admin/README.md
+++ b/static/reference/next/pulsar-admin/README.md
@@ -4,13 +4,13 @@
You can use the following environment variables to configure `pulsar-admin`.
-|Variable|Description|Default|
-|---|---|---|
-|`PULSAR_LOG_CONF`|Log4j configuration file|conf/log4j2.yaml|
-|`PULSAR_CLIENT_CONF`|Configuration file for the client|conf/client.conf|
-|`PULSAR_EXTRA_OPTS`|Extra options passed to the JVM|N/A|
-|`PULSAR_EXTRA_CLASSPATH`|Extra paths for Pulsar's classpath|N/A|
+| Variable | Description | Default |
+|--------------------------|------------------------------------|------------------|
+| `PULSAR_LOG_CONF` | Log4j configuration file | conf/log4j2.yaml |
+| `PULSAR_CLIENT_CONF` | Configuration file for the client | conf/client.conf |
+| `PULSAR_EXTRA_OPTS` | Extra options passed to the JVM | N/A |
+| `PULSAR_EXTRA_CLASSPATH` | Extra paths for Pulsar's classpath | N/A |
## Related topics
-- [Pulsar admin API](docs/admin-api-overview)
\ No newline at end of file
+- [Pulsar admin API](docs/admin-api-overview)
diff --git a/static/reference/next/pulsar-client/README.md b/static/reference/next/pulsar-client/README.md
index f875e02f284..d25591d2d24 100644
--- a/static/reference/next/pulsar-client/README.md
+++ b/static/reference/next/pulsar-client/README.md
@@ -4,9 +4,9 @@
You can use the following environment variables to configure `pulsar-client`.
-|Variable|Description|Default|
-|---|---|---|
-|`PULSAR_LOG_CONF`|Log4j configuration file|conf/log4j2.yaml|
-|`PULSAR_CLIENT_CONF`|Configuration file for the client|conf/client.conf|
-|`PULSAR_EXTRA_OPTS`|Extra options passed to the JVM|N/A|
-|`PULSAR_EXTRA_CLASSPATH`|Extra paths for Pulsar's classpath|N/A|
\ No newline at end of file
+| Variable | Description | Default |
+|--------------------------|------------------------------------|------------------|
+| `PULSAR_LOG_CONF` | Log4j configuration file | conf/log4j2.yaml |
+| `PULSAR_CLIENT_CONF` | Configuration file for the client | conf/client.conf |
+| `PULSAR_EXTRA_OPTS` | Extra options passed to the JVM | N/A |
+| `PULSAR_EXTRA_CLASSPATH` | Extra paths for Pulsar's classpath | N/A |
diff --git a/static/reference/next/pulsar-perf/README.md b/static/reference/next/pulsar-perf/README.md
index 6359464dd35..3a9ee62e2ef 100644
--- a/static/reference/next/pulsar-perf/README.md
+++ b/static/reference/next/pulsar-perf/README.md
@@ -4,9 +4,9 @@
You can use the following environment variables to configure `pulsar-perf`.
-|Variable|Description|Default|
-|---|---|---|
-|`PULSAR_LOG_CONF`|Log4j configuration file|conf/log4j2.yaml|
-|`PULSAR_CLIENT_CONF`|Configuration file for the client|conf/client.conf|
-|`PULSAR_EXTRA_OPTS`|Extra options passed to the JVM|N/A|
-|`PULSAR_EXTRA_CLASSPATH`|Extra paths for Pulsar's classpath|N/A|
\ No newline at end of file
+| Variable | Description | Default |
+|--------------------------|------------------------------------|------------------|
+| `PULSAR_LOG_CONF` | Log4j configuration file | conf/log4j2.yaml |
+| `PULSAR_CLIENT_CONF` | Configuration file for the client | conf/client.conf |
+| `PULSAR_EXTRA_OPTS` | Extra options passed to the JVM | N/A |
+| `PULSAR_EXTRA_CLASSPATH` | Extra paths for Pulsar's classpath | N/A |
diff --git a/static/reference/next/pulsar-shell/pulsar-shell.md b/static/reference/next/pulsar-shell/pulsar-shell.md
index 1e50f7fe2e6..c7d6e227a09 100644
--- a/static/reference/next/pulsar-shell/pulsar-shell.md
+++ b/static/reference/next/pulsar-shell/pulsar-shell.md
@@ -1,6 +1,6 @@
# pulsar-shell
-[Pulsar shell](https://pulsar.apache.org/docs/next/administration-pulsar-shell) tool.
+[Pulsar shell](pathname:///docs/@pulsar:version@/administration-pulsar-shell) tool.
### Interactive mode
@@ -10,14 +10,13 @@ Usage
pulsar-shell
```
-| Flag | Description | Default |
-|--------------------|---------------------------------------------------------------------------|------------------|
-| `-c`, `--config` | Client configuration file. It is used as a `default` config. | conf/client.conf |
-| `--fail-on-error` | If true, the shell is interrupted when a command throws an exception. | false |
-| `-h`, `--help` | Show this help. | |
+| Flag | Description | Default |
+|-------------------|-----------------------------------------------------------------------|------------------|
+| `-c`, `--config` | Client configuration file. It is used as a `default` config. | conf/client.conf |
+| `--fail-on-error` | If true, the shell is interrupted when a command throws an exception. | false |
+| `-h`, `--help` | Show this help. | |
-
-### Non interactive mode
+### Non-interactive mode
Usage
@@ -27,22 +26,21 @@ pulsar-shell -e [COMMAND]
echo "[COMMAND]" | pulsar-shell -
```
-| Flag | Description | Default |
-|---------------------------|-----------------------------------------------------------------------------------------------------|-----------------|
-| `-c`, `--config` | Client configuration file. It is used as a `default` config. | conf/client.conf |
-| `--fail-on-error` | If true, the shell is interrupted when a command throws an exception. | false |
-| `-np`, `--no-progress` | Display raw output of the commands without the fancy progress visualization. | false |
-| `-f`, `--filename` | Input filename with a list of commands to be executed. Each command must be separated by a newline. | |
-| `-e`, `--execute-command` | Execute this command and exit. | |
-| `-` | Read commands from the standard input. | |
-| `-h`, `--help` | Show this help. | |
-
+| Flag | Description | Default |
+|---------------------------|-----------------------------------------------------------------------------------------------------|------------------|
+| `-c`, `--config` | Client configuration file. It is used as a `default` config. | conf/client.conf |
+| `--fail-on-error` | If true, the shell is interrupted when a command throws an exception. | false |
+| `-np`, `--no-progress` | Display raw output of the commands without the fancy progress visualization. | false |
+| `-f`, `--filename` | Input filename with a list of commands to be executed. Each command must be separated by a newline. | |
+| `-e`, `--execute-command` | Execute this command and exit. | |
+| `-` | Read commands from the standard input. | |
+| `-h`, `--help` | Show this help. | |
Commands
-* `admin` - See [Admin API](admin-api-overview.md)
-* `client` - See [pulsar-client](#pulsar-client)
-* `config`
+* `admin` - See [Admin API](pathname:///docs/@pulsar:version@/admin-api-overview)
+* `client` - See [pulsar-client](../pulsar-client/README.md)
+* `config`
### config
@@ -64,11 +62,11 @@ Create a new configuration.
default(localhost)> config create --file ./conf/client.conf mycluster
```
-| Flag | Description | Default |
-|----------|--------------------------|-----------------|
-| `--file` | File path of the config. | |
-| `--url` | URL of the config. | |
-| `--value` | Inline value of the config. Base64-encoded value is supported with the prefix `base64:`. | |
+| Flag | Description | Default |
+|-----------|------------------------------------------------------------------------------------------|---------|
+| `--file` | File path of the config. | |
+| `--url` | URL of the config. | |
+| `--value` | Inline value of the config. Base64-encoded value is supported with the prefix `base64:`. | |
#### clone
@@ -78,9 +76,9 @@ Create a new configuration cloning an existing one.
default(localhost)> config clone mycluster --name mycluster2
```
-| Flag | Description | Default |
-|----------|--------------------------|-----------------|
-| `--name` | Name of the new config. | |
+| Flag | Description | Default |
+|----------|-------------------------|---------|
+| `--name` | Name of the new config. | |
#### update
@@ -90,11 +88,11 @@ Update an existing configuration.
default(localhost)> config update --file ./conf/client.conf mycluster
```
-| Flag | Description | Default |
-|----------|--------------------------|-----------------|
-| `--file` | File path of the config. | |
-| `--url` | URL of the config. | |
-| `--value` | Inline value of the config. Base64-encoded value is supported with the prefix `base64:`. | |
+| Flag | Description | Default |
+|-----------|------------------------------------------------------------------------------------------|---------|
+| `--file` | File path of the config. | |
+| `--url` | URL of the config. | |
+| `--value` | Inline value of the config. Base64-encoded value is supported with the prefix `base64:`. | |
#### set-property
@@ -104,11 +102,10 @@ Set a value for a specified configuration property.
default(localhost)> config set-property -p webServiceUrl -v http://<cluster-hostname> mycluster
```
-| Flag | Description | Default |
-|--------------------|-----------------------------|-----------------|
-| `-p`, `--property` | Property name to update. | |
-| `-v`, `--value` | New value for the property. | |
-
+| Flag | Description | Default |
+|--------------------|-----------------------------|---------|
+| `-p`, `--property` | Property name to update. | |
+| `-v`, `--value` | New value for the property. | |
#### get-property
@@ -118,10 +115,9 @@ Get the value for a specified configuration property.
default(localhost)> config get-property -p webServiceUrl mycluster
```
-| Flag | Description | Default |
-|--------------------|-----------------------------|-----------------|
-| `-p`, `--property` | Property name to update. | |
-
+| Flag | Description | Default |
+|--------------------|--------------------------|---------|
+| `-p`, `--property` | Property name to update. | |
#### view
@@ -139,7 +135,6 @@ Delete a config. You can't delete a config if it's currently used.
default(localhost)> config delete mycluster
```
-
#### list
List all the configuration names.
diff --git a/tools/pytools/bin/reference-doc-generator.py b/tools/pytools/bin/reference-doc-generator.py
index 59a0e091c41..e8293bfec14 100755
--- a/tools/pytools/bin/reference-doc-generator.py
+++ b/tools/pytools/bin/reference-doc-generator.py
@@ -18,11 +18,13 @@
# under the License.
import enum
+import shutil
from argparse import ArgumentParser, ArgumentDefaultsHelpFormatter
from pathlib import Path
import semver
+from constant import site_path
from execute import config_doc_generator, pulsar_admin_clidoc_generator, pulsar_clidoc_generator
from execute import pulsar_client_clidoc_generator, pulsar_perf_clidoc_generator
@@ -45,14 +47,19 @@ if __name__ == '__main__':
args = parser.parse_args()
master_path = Path(args.master_path)
kinds = set(args.kind)
- if Kind.all in kinds:
- kinds = {Kind.config, Kind.admin, Kind.pulsar, Kind.client, Kind.perf}
+ version = args.version
- if args.version != 'next':
- ver = semver.VersionInfo.parse(args.version)
+ if version != 'next':
+ ver = semver.VersionInfo.parse(version)
version = f"{ver.major}.{ver.minor}.x"
- else:
- version = args.version
+
+ if version != 'next' and Kind.all in kinds:
+ src = site_path() / 'static' / 'reference' / 'next'
+ dst = site_path() / 'static' / 'reference' / version
+ shutil.copytree(src, dst)
+
+ if Kind.all in kinds:
+ kinds = {Kind.config, Kind.admin, Kind.pulsar, Kind.client, Kind.perf}
for kind in kinds:
if kind == Kind.config: