You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by zi...@apache.org on 2022/07/12 07:01:40 UTC
[pulsar] branch master updated: [feature][doc] Add docs for how to configure metadata store (#16463)
This is an automated email from the ASF dual-hosted git repository.
zixuan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 59658cb6dd0 [feature][doc] Add docs for how to configure metadata store (#16463)
59658cb6dd0 is described below
commit 59658cb6dd01880b4699ca330d9c846c9fe985d0
Author: momo-jun <60...@users.noreply.github.com>
AuthorDate: Tue Jul 12 15:01:33 2022 +0800
[feature][doc] Add docs for how to configure metadata store (#16463)
* initial draft
* Fix review comments
---
site2/docs/administration-metadata-store.md | 120 ++++++++++++++++++++++++++++
site2/docs/reference-configuration.md | 3 +-
site2/website/sidebars.json | 3 +-
3 files changed, 124 insertions(+), 2 deletions(-)
diff --git a/site2/docs/administration-metadata-store.md b/site2/docs/administration-metadata-store.md
new file mode 100644
index 00000000000..a4550b1f085
--- /dev/null
+++ b/site2/docs/administration-metadata-store.md
@@ -0,0 +1,120 @@
+---
+id: administration-metadata-store
+title: Configure metadata store
+sidebar_label: "Configure metadata store"
+---
+
+Pulsar metadata store maintains all the metadata, configuration, and coordination of a Pulsar cluster, such as topic metadata, schema, broker load data, and so on.
+
+The metadata store of each Pulsar instance should contain the following two components:
+* A local metadata store ensemble (`metadataStoreUrl`) that stores cluster-specific configuration and coordination, such as which brokers are responsible for which topics as well as ownership metadata, broker load reports, and BookKeeper ledger metadata.
+* A configuration store quorum (`configurationMetadataStoreUrl`) stores configuration for clusters, tenants, namespaces, topics, and other entities that need to be globally consistent.
+
+:::note
+
+If you are using a standalone Pulsar or a single Pulsar cluster, you only need to configure one metadata store (via `metadataStoreUrl`) and it also serves as a configuration store.
+
+:::
+
+Pulsar supports the following metadata store services:
+* [Apache ZooKeeper](https://zookeeper.apache.org/)
+* [Etcd](https://etcd.io/)
+* [RocksDB](http://rocksdb.org/)
+* Local memory
+
+:::note
+
+RocksDB and local memory are only applicable to standalone Pulsar or single-node Pulsar clusters.
+
+:::
+
+## Use ZooKeeper as metadata store
+
+Pulsar metadata store can be deployed on a separate ZooKeeper cluster or deployed on an existing ZooKeeper cluster.
+
+To use ZooKeeper as the metadata store, add the following parameters to the `conf/broker.conf` or `conf/standalone.conf` file.
+
+```conf
+
+metadataStoreUrl=zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181
+configurationMetadataStoreUrl=zk:my-global-zk-1:2181,my-global-zk-2:2181,my-global-zk-3:2181
+
+```
+
+## Use etcd as metadata store
+
+To use etcd as the metadata store, add the following parameters to the `conf/broker.conf` or `conf/standalone.conf` file.
+
+```conf
+
+metadataStoreUrl=etcd:my-etcd-1:2379,my-etcd-2:2379,my-etcd-3:2379
+configurationMetadataStoreUrl=etcd:my-global-etcd-1:2379,my-global-etcd-2:2379,my-global-etcd-3:2379
+# metadataStoreConfigPath=/path/to/file
+
+```
+
+:::tip
+
+The `metadataStoreConfigPath` parameter is required when you want to use the following advanced configurations.
+
+```
+useTls=false
+tlsProvider=JDK
+tlsTrustCertsFilePath=
+tlsKeyFilePath=
+tlsCertificateFilePath=
+authority=
+```
+
+:::
+
+## Use RocksDB as metadata store
+
+To use RocksDB as the metadata store, add the following parameters to the `conf/broker.conf` or `conf/standalone.conf` file.
+
+```conf
+
+metadataStoreUrl=rocksdb://data/metadata
+# metadataStoreConfigPath=/path/to/file
+
+```
+
+:::tip
+
+The `metadataStoreConfigPath` parameter is required when you want to use advanced configurations. See [this example](https://github.com/facebook/rocksdb/blob/main/examples/rocksdb_option_file_example.ini) for more information.
+
+:::
+
+## Use local memory as metadata store
+
+To use local memory as the metadata store, add the following parameters to the `conf/broker.conf` or `conf/standalone.conf` file.
+
+```conf
+
+metadataStoreUrl=memory://local
+
+```
+
+
+## Enable batch operations on metadata store
+
+Pulsar metadata store supports batch operations and caching to meet low latency and high throughput and improve performance.
+
+To enable batch operations on the metadata store, you can configure the following parameters in the `conf/broker.conf` or `conf/standalone.conf` file.
+
+```conf
+
+# Whether we should enable metadata operations batching
+metadataStoreBatchingEnabled=true
+
+# Maximum delay to impose on batching grouping
+metadataStoreBatchingMaxDelayMillis=5
+
+# Maximum number of operations to include in a singular batch
+metadataStoreBatchingMaxOperations=1000
+
+# Maximum size of a batch
+metadataStoreBatchingMaxSizeKb=128
+
+```
+
diff --git a/site2/docs/reference-configuration.md b/site2/docs/reference-configuration.md
index a57d02995c0..f7a7fd5bda8 100644
--- a/site2/docs/reference-configuration.md
+++ b/site2/docs/reference-configuration.md
@@ -252,6 +252,7 @@ brokerServiceCompactionThresholdInBytes|If the estimated backlog size is greater
| authenticationRefreshCheckSeconds | Interval of time for checking for expired authentication credentials | 60 |
|authorizationEnabled| Enforce authorization |false|
|superUserRoles| Role names that are treated as “super-user”, meaning they will be able to do all admin operations and publish/consume from all topics ||
+| proxyRoles | Role names that are treated as "proxy roles". If the broker receives a request from a proxy role, it demands to authenticate its client role. Note that client role and proxy role cannot use the same name. | |
|brokerClientAuthenticationPlugin| Authentication settings of the broker itself. Used when the broker connects to other brokers, either in same or other clusters ||
|brokerClientAuthenticationParameters|||
|athenzDomainNames| Supported Athenz provider domain names(comma separated) for authentication ||
@@ -605,7 +606,7 @@ You can set the log level and configuration in the [log4j2.yaml](https://github
| systemTopicEnabled | Enable/Disable system topics. | false |
| topicLevelPoliciesEnabled | Enable or disable topic level policies. Topic level policies depends on the system topic. Please enable the system topic first. | false |
| topicFencingTimeoutSeconds | If a topic remains fenced for a certain time period (in seconds), it is closed forcefully. If set to 0 or a negative number, the fenced topic is not closed. | 0 |
-| proxyRoles | Role names that are treated as "proxy roles". If the broker sees a request with role as proxyRoles, it demands to see a valid original principal. | |
+| proxyRoles | Role names that are treated as "proxy roles". If the broker receives a request from a proxy role, it demands to authenticate its client role. Note that client role and proxy role cannot use the same name. | |
|authenticationEnabled| Enable authentication for the broker. |false|
|authenticationProviders| A comma-separated list of class names for authentication providers. |false|
|authorizationEnabled| Enforce authorization in brokers. |false|
diff --git a/site2/website/sidebars.json b/site2/website/sidebars.json
index be4f3b0d796..960baf32f87 100644
--- a/site2/website/sidebars.json
+++ b/site2/website/sidebars.json
@@ -230,8 +230,9 @@
"administration-stats",
"administration-load-balance",
"administration-proxy",
+ "administration-isolation",
"administration-upgrade",
- "administration-isolation"
+ "administration-metadata-store"
]
},
{