You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by ur...@apache.org on 2022/07/20 18:01:40 UTC
[pulsar-site] branch main updated: Docs sync done from apache/pulsar(#74bafe2)
This is an automated email from the ASF dual-hosted git repository.
urfree pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git
The following commit(s) were added to refs/heads/main by this push:
new 39258f1027b Docs sync done from apache/pulsar(#74bafe2)
39258f1027b is described below
commit 39258f1027bf056696569bf7e20edbb92089ede7
Author: Pulsar Site Updater <de...@pulsar.apache.org>
AuthorDate: Wed Jul 20 18:01:34 2022 +0000
Docs sync done from apache/pulsar(#74bafe2)
---
.../docs/administration-isolation-bookie.md | 69 ++++++++++++
.../docs/administration-isolation-broker.md | 57 ++++++++++
.../website-next/docs/administration-isolation.md | 125 ++++++---------------
site2/website-next/docs/reference-metrics.md | 5 +-
site2/website-next/sidebars.json | 14 ++-
site2/website-next/static/assets/isolation-1.png | Bin 0 -> 74607 bytes
site2/website-next/static/assets/isolation-2.png | Bin 0 -> 83257 bytes
site2/website-next/static/assets/isolation-3.png | Bin 0 -> 79529 bytes
site2/website-next/static/assets/isolation-4.png | Bin 0 -> 129134 bytes
9 files changed, 175 insertions(+), 95 deletions(-)
diff --git a/site2/website-next/docs/administration-isolation-bookie.md b/site2/website-next/docs/administration-isolation-bookie.md
new file mode 100644
index 00000000000..20ca5a155c0
--- /dev/null
+++ b/site2/website-next/docs/administration-isolation-bookie.md
@@ -0,0 +1,69 @@
+---
+id: administration-isolation-bookie
+title: Isolate bookies
+sidebar_label: "Isolate bookies"
+---
+
+````mdx-code-block
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+````
+
+
+A namespace can be isolated into user-defined groups of bookies, which guarantees all the data that belongs to the namespace is stored in desired bookies. The bookie affinity group uses the BookKeeper [rack-aware placement policy](https://bookkeeper.apache.org/docs/latest/api/javadoc/org/apache/bookkeeper/client/EnsemblePlacementPolicy.html) and it is a way to feed rack information which is stored as JSON format in znode.
+
+You can set a bookie affinity group using one of the following methods.
+
+````mdx-code-block
+<Tabs
+ defaultValue="Admin CLI"
+ values={[{"label":"Admin CLI","value":"Admin CLI"},{"label":"REST API","value":"REST API"},{"label":"Java admin API","value":"Java admin API"}]}>
+
+<TabItem value="Admin CLI">
+
+```
+
+pulsar-admin namespaces set-bookie-affinity-group options
+
+```
+
+For more information about the command `pulsar-admin namespaces set-bookie-affinity-group options`, see [Pulsar admin docs](https://pulsar.apache.org/tools/pulsar-admin/).
+
+**Example**
+
+```shell
+
+bin/pulsar-admin bookies set-bookie-rack \
+--bookie 127.0.0.1:3181 \
+--hostname 127.0.0.1:3181 \
+--group group-bookie1 \
+--rack rack1
+
+bin/pulsar-admin namespaces set-bookie-affinity-group public/default \
+--primary-group group-bookie1
+
+```
+
+:::note
+
+- Do not set a bookie rack name to slash (`/`) or an empty string (`""`) if you use Pulsar earlier than 2.7.5, 2.8.3, and 2.9.2. If you use Pulsar 2.7.5, 2.8.3, 2.9.2 or later versions, it falls back to `/default-rack` or `/default-region/default-rack`.
+- When `RackawareEnsemblePlacementPolicy` is enabled, the rack name is not allowed to contain slash (`/`) except for the beginning and end of the rack name string. For example, rack name like `/rack0` is okay, but `/rack/0` is not allowed.
+- When `RegionawareEnsemblePlacementPolicy` is enabled, the rack name can only contain one slash (`/`) except for the beginning and end of the rack name string. For example, rack name like `/region0/rack0` is okay, but `/region0rack0` and `/region0/rack/0` are not allowed.
+For the bookie rack name restrictions, see [pulsar-admin bookies set-bookie-rack](https://pulsar.apache.org/tools/pulsar-admin/).
+
+:::
+
+</TabItem>
+<TabItem value="REST API">
+
+[POST /admin/v2/namespaces/{tenant}/{namespace}/persistence/bookieAffinity](https://pulsar.apache.org/admin-rest-api/?version=master&apiversion=v2#operation/setBookieAffinityGroup)
+
+</TabItem>
+<TabItem value="Java admin API">
+
+For how to set bookie affinity group for a namespace using Java admin API, see [code](https://github.com/apache/pulsar/blob/master/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/internal/NamespacesImpl.java#L1164).
+
+</TabItem>
+
+</Tabs>
+````
diff --git a/site2/website-next/docs/administration-isolation-broker.md b/site2/website-next/docs/administration-isolation-broker.md
new file mode 100644
index 00000000000..11a857eacf6
--- /dev/null
+++ b/site2/website-next/docs/administration-isolation-broker.md
@@ -0,0 +1,57 @@
+---
+id: administration-isolation-broker
+title: Isolate brokers
+sidebar_label: "Isolate brokers"
+---
+
+````mdx-code-block
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+````
+
+
+In Pulsar, when namespaces (more specifically, namespace bundles) are assigned dynamically to brokers, the namespace isolation policy limits the set of brokers that can be used for assignment. Before topics are assigned to brokers, you can set the namespace isolation policy with a primary or a secondary regex to select desired brokers.
+
+You can set a namespace isolation policy for a cluster using one of the following methods.
+
+````mdx-code-block
+<Tabs
+ defaultValue="Admin CLI"
+ values={[{"label":"Admin CLI","value":"Admin CLI"},{"label":"REST API","value":"REST API"},{"label":"Java admin API","value":"Java admin API"}]}>
+
+<TabItem value="Admin CLI">
+
+```
+
+pulsar-admin ns-isolation-policy set options
+
+```
+
+For more information about the command `pulsar-admin ns-isolation-policy set options`, see [Pulsar admin docs](https://pulsar.apache.org/tools/pulsar-admin/).
+
+**Example**
+
+```shell
+
+bin/pulsar-admin ns-isolation-policy set \
+--auto-failover-policy-type min_available \
+--auto-failover-policy-params min_limit=1,usage_threshold=80 \
+--namespaces my-tenant/my-namespace \
+--primary 10.193.216.* my-cluster policy-name
+
+```
+
+</TabItem>
+<TabItem value="REST API">
+
+[PUT /admin/v2/namespaces/{tenant}/{namespace}](https://pulsar.apache.org/admin-rest-api/?version=master&apiversion=v2#operation/createNamespace)
+
+</TabItem>
+<TabItem value="Java admin API">
+
+For how to set namespace isolation policy using Java admin API, see [code](https://github.com/apache/pulsar/blob/master/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/internal/NamespacesImpl.java#L251).
+
+</TabItem>
+
+</Tabs>
+````
diff --git a/site2/website-next/docs/administration-isolation.md b/site2/website-next/docs/administration-isolation.md
index 22f9b94337d..02a11a95ca7 100644
--- a/site2/website-next/docs/administration-isolation.md
+++ b/site2/website-next/docs/administration-isolation.md
@@ -4,120 +4,61 @@ title: Pulsar isolation
sidebar_label: "Pulsar isolation"
---
-````mdx-code-block
-import Tabs from '@theme/Tabs';
-import TabItem from '@theme/TabItem';
-````
-
In an organization, a Pulsar instance provides services to multiple teams. When organizing the resources across multiple teams, you want to make a suitable isolation plan to avoid the resource competition between different teams and applications and provide high-quality messaging service. In this case, you need to take resource isolation into consideration and weigh your intended actions against expected and unexpected consequences.
-To enforce resource isolation, you can use the Pulsar isolation policy, which allows you to allocate resources (**broker** and **bookie**) for the namespace.
-
-## Broker isolation
-
-In Pulsar, when namespaces (more specifically, namespace bundles) are assigned dynamically to brokers, the namespace isolation policy limits the set of brokers that can be used for assignment. Before topics are assigned to brokers, you can set the namespace isolation policy with a primary or a secondary regex to select desired brokers.
-
-You can set a namespace isolation policy for a cluster using one of the following methods.
-
-````mdx-code-block
-<Tabs groupId="api-choice"
- defaultValue="Admin CLI"
- values={[{"label":"Admin CLI","value":"Admin CLI"},{"label":"REST API","value":"REST API"},{"label":"Java Admin API","value":"Java Admin API"}]}>
-
-<TabItem value="Admin CLI">
-
-```
-
-pulsar-admin ns-isolation-policy set options
-
-```
-
-For more information about the command `pulsar-admin ns-isolation-policy set options`, see [here](/tools/pulsar-admin/).
-
-**Example**
-
-```shell
-
-bin/pulsar-admin ns-isolation-policy set \
---auto-failover-policy-type min_available \
---auto-failover-policy-params min_limit=1,usage_threshold=80 \
---namespaces my-tenant/my-namespace \
---primary 10.193.216.* my-cluster policy-name
+The multi-layer and segment-centric architecture and hierarchical resource management of Pulsar provide a solid foundation for isolation, which allows you to isolate resources in your desired manner, prevent resource competition, and attain stability.
-```
-</TabItem>
-<TabItem value="REST API">
+## Isolation methods
+To enforce resource isolation within Pulsar, you can use the following methods of isolating broker and bookie resources:
+* [Isolate brokers](administration-isolation-broker.md)
+* [Isolate bookies](administration-isolation-bookie.md)
-[PUT /admin/v2/namespaces/{tenant}/{namespace}](/admin-rest-api/?version=master&apiversion=v2#operation/createNamespace)
-</TabItem>
-<TabItem value="Java Admin API">
+## Deployments to achieve isolation within Pulsar
-For how to set namespace isolation policy using Java admin API, see [here](https://github.com/apache/pulsar/blob/master/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/internal/NamespacesImpl.java#L251).
+### Separate Pulsar clusters
-</TabItem>
+The following illustration demonstrates the deployment of separate Pulsar clusters to achieve the highest-level isolation.
-</Tabs>
-````
+
-## Bookie isolation
+Here are some key points for understanding how it works:
+- Separate Pulsar clusters use a shared [configuration store](concepts-architecture-overview.md/#configuration-store).
+- Each cluster exposes its service through a DNS entry point and makes sure a client can access the cluster through the DNS entry point. Clients can use one or multiple Pulsar URLs that the Pulsar cluster exposes as the service URL.
+- Each Pulsar cluster has one or multiple brokers and bookies.
+- Each Pulsar cluster has one metadata store, which can be separated into [Pulsar metadata store](concepts-architecture-overview.md/#metadata-store) and [BookKeeper metadata store](https://bookkeeper.apache.org/docs/latest/getting-started/concepts/#metadata-storage).
-A namespace can be isolated into user-defined groups of bookies, which guarantees all the data that belongs to the namespace is stored in desired bookies. The bookie affinity group uses the BookKeeper [rack-aware placement policy](https://bookkeeper.apache.org/docs/latest/api/javadoc/org/apache/bookkeeper/client/EnsemblePlacementPolicy.html) and it is a way to feed rack information which is stored as JSON format in znode.
-
-You can set a bookie affinity group using one of the following methods.
-
-````mdx-code-block
-<Tabs groupId="api-choice"
- defaultValue="Admin CLI"
- values={[{"label":"Admin CLI","value":"Admin CLI"},{"label":"REST API","value":"REST API"},{"label":"Java Admin API","value":"Java Admin API"}]}>
-
-<TabItem value="Admin CLI">
-
-```
-
-pulsar-admin namespaces set-bookie-affinity-group options
-
-```
-
-For more information about the command `pulsar-admin namespaces set-bookie-affinity-group options`, see [here](/tools/pulsar-admin/).
-
-**Example**
+:::note
-```shell
+When using this approach, if you want to achieve namespace isolation, you need to specify a cluster for a namespace. The cluster must be in the allowed cluster list of the tenant. Topics under the namespace are assigned to this cluster.
-bin/pulsar-admin bookies set-bookie-rack \
---bookie 127.0.0.1:3181 \
---hostname 127.0.0.1:3181 \
---group group-bookie1 \
---rack rack1
+:::
-bin/pulsar-admin namespaces set-bookie-affinity-group public/default \
---primary-group group-bookie1
+### Shared BookKeeper cluster
-```
+The following illustration demonstrates the deployment of shared BookKeeper clusters to achieve isolation.
-:::note
+
-- Do not set a bookie rack name to slash (`/`) or an empty string (`""`) if you use Pulsar earlier than 2.7.5, 2.8.3, and 2.9.2. If you use Pulsar 2.7.5, 2.8.3, 2.9.2 or later versions, it falls back to `/default-rack` or `/default-region/default-rack`.
-- When `RackawareEnsemblePlacementPolicy` is enabled, the rack name is not allowed to contain slash (`/`) except for the beginning and end of the rack name string. For example, rack name like `/rack0` is okay, but `/rack/0` is not allowed.
-- When `RegionawareEnsemblePlacementPolicy` is enabled, the rack name can only contain one slash (`/`) except for the beginning and end of the rack name string. For example, rack name like `/region0/rack0` is okay, but `/region0rack0` and `/region0/rack/0` are not allowed.
-For the bookie rack name restrictions, see [pulsar-admin bookies set-bookie-rack](/tools/pulsar-admin/).
+Here are some key points for understanding how it works:
+- Separate Pulsar clusters share a BookKeeper cluster and a [configuration store](concepts-architecture-overview.md/#configuration-store).
+- Each cluster exposes its service through a DNS entry point and makes sure a client can access the cluster through the DNS entry point. Clients can use one or multiple Pulsar URLs that the Pulsar cluster exposes as the service URL.
+- Each Pulsar cluster has one or multiple brokers.
+- Each Pulsar cluster has one metadata store.
-:::
+As illustrated below, storage isolation is achieved by setting [bookie affinity groups](administration-isolation-bookie.md). All bookie groups use a shared BookKeeper cluster and a metadata store, and each bookie isolation group has one or several bookies. You can specify one or multiple primary/secondary groups for a namespace. Topics under the namespace are created on the bookies in the primary group firstly and then created on the bookies in the secondary group.
-</TabItem>
-<TabItem value="REST API">
+
-[POST /admin/v2/namespaces/{tenant}/{namespace}/persistence/bookieAffinity](/admin-rest-api/?version=master&apiversion=v2#operation/setBookieAffinityGroup)
+### Single Pulsar cluster
-</TabItem>
-<TabItem value="Java Admin API">
+The following illustration demonstrates how to achieve isolation inside a single Pulsar cluster.
-For how to set bookie affinity group for a namespace using Java admin API, see [here](https://github.com/apache/pulsar/blob/master/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/internal/NamespacesImpl.java#L1164).
+
-</TabItem>
+Here are some key points for understanding how it works:
+- Each cluster exposes its service through a DNS entry point and makes sure a client can access the cluster through the DNS entry point. Clients can use one or multiple Pulsar URLs that the Pulsar cluster exposes as the service URL.
+- Broker isolation is achieved by setting [namespace isolation policy](administration-isolation-broker.md).
-</Tabs>
-````
diff --git a/site2/website-next/docs/reference-metrics.md b/site2/website-next/docs/reference-metrics.md
index ec978f877c4..0b7c98f2dc3 100644
--- a/site2/website-next/docs/reference-metrics.md
+++ b/site2/website-next/docs/reference-metrics.md
@@ -97,14 +97,15 @@ in the `bookkeeper.conf` configuration file.
| auditor_NUM_LEDGERS_HAVING_NO_REPLICA_OF_AN_ENTRY | Gauge | Gauge for number of ledgers having an entry with all the replicas missing. |
| auditor_NUM_LEDGERS_HAVING_LESS_THAN_AQ_REPLICAS_OF_AN_ENTRY | Gauge | Gauge for number of ledgers having an entry with less than AQ number of replicas, this doesn't include ledgers counted towards numLedgersHavingNoReplicaOfAnEntry. |
| auditor_NUM_LEDGERS_HAVING_LESS_THAN_WQ_REPLICAS_OF_AN_ENTRY | Gauge | Gauge for number of ledgers having an entry with less than WQ number of replicas, this doesn't include ledgers counted towards numLedgersHavingLessThanAQReplicasOfAnEntry. |
+| election_attempts | Counter | The number of auditor election attempts. |
| replication_worker_NUM_BYTES_READ | Summary | The distribution of size of entries read by the replicator. |
| replication_worker_NUM_ENTRIES_READ | Counter | Number of entries read by the replicator. |
| replication_worker_NUM_ENTRIES_WRITTEN | Counter | Number of entries written by the replicator. |
| replication_worker_NUM_BYTES_WRITTEN | Summary | The distribution of size of entries written by the replicator. |
| replication_worker_READ_DATA_LATENCY | Summary | The distribution of latency of read entries by the replicator. |
| replication_worker_WRITE_DATA_LATENCY | Summary | The distribution of latency of write entries by the replicator. |
-| replication_worker_REPLICATE_EXCEPTION | Summary | Replication related exceptions. |
-| replication_worker_REREPLICATE_OP | Summary | Operation stats of re-replicating ledgers. |
+| replication_worker_exceptions | Summary | Replication related exceptions. |
+| replication_worker_rereplicate | Summary | Operation stats of re-replicating ledgers. |
| replication_worker_NUM_FULL_OR_PARTIAL_LEDGERS_REPLICATED | Counter | The number of ledgers re-replicated. |
| replication_worker_NUM_DEFER_LEDGER_LOCK_RELEASE_OF_FAILED_LEDGER | Counter | The number of defer-ledger-lock-releases of failed ledgers. |
| replication_worker_NUM_ENTRIES_UNABLE_TO_READ_FOR_REPLICATION | Counter | The number of entries ReplicationWorker unable to read. |
diff --git a/site2/website-next/sidebars.json b/site2/website-next/sidebars.json
index 960baf32f87..32bd994ecee 100644
--- a/site2/website-next/sidebars.json
+++ b/site2/website-next/sidebars.json
@@ -225,6 +225,7 @@
"label": "Administration",
"items": [
"administration-zk-bk",
+ "administration-metadata-store"
"administration-geo",
"administration-pulsar-manager",
"administration-stats",
@@ -232,7 +233,18 @@
"administration-proxy",
"administration-isolation",
"administration-upgrade",
- "administration-metadata-store"
+ {
+ "type": "category",
+ "label": "Pulsar isolation",
+ "link": {
+ "type": "doc",
+ "id": "administration-isolation"
+ },
+ "items": [
+ "administration-isolation-broker",
+ "administration-isolation-bookie"
+ ]
+ }
]
},
{
diff --git a/site2/website-next/static/assets/isolation-1.png b/site2/website-next/static/assets/isolation-1.png
new file mode 100644
index 00000000000..53c41192144
Binary files /dev/null and b/site2/website-next/static/assets/isolation-1.png differ
diff --git a/site2/website-next/static/assets/isolation-2.png b/site2/website-next/static/assets/isolation-2.png
new file mode 100644
index 00000000000..9ba8f609c04
Binary files /dev/null and b/site2/website-next/static/assets/isolation-2.png differ
diff --git a/site2/website-next/static/assets/isolation-3.png b/site2/website-next/static/assets/isolation-3.png
new file mode 100644
index 00000000000..ffe4545f87c
Binary files /dev/null and b/site2/website-next/static/assets/isolation-3.png differ
diff --git a/site2/website-next/static/assets/isolation-4.png b/site2/website-next/static/assets/isolation-4.png
new file mode 100644
index 00000000000..9b4670b8a21
Binary files /dev/null and b/site2/website-next/static/assets/isolation-4.png differ