You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by ur...@apache.org on 2022/02/17 08:31:23 UTC
[pulsar-site] 01/04: update 2.4.x
This is an automated email from the ASF dual-hosted git repository.
urfree pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git
commit 2a696df3eb288290c68c1a97c75d671a4eb80ec5
Author: LiLi <ur...@apache.org>
AuthorDate: Thu Feb 17 16:10:16 2022 +0800
update 2.4.x
Signed-off-by: LiLi <ur...@apache.org>
---
site2/website-next/migrate/migrate-chapter.js | 15 +-
site2/website-next/migrate/migrate-docs.js | 1 +
.../versioned_docs/version-2.4.0/adaptors-spark.md | 1 -
.../versioned_docs/version-2.4.0/adaptors-storm.md | 1 -
.../version-2.4.0/admin-api-brokers.md | 5 +-
.../version-2.4.0/admin-api-clusters.md | 7 +-
.../version-2.4.0/admin-api-namespaces.md | 124 +++---
.../admin-api-non-persistent-topics.md | 1 -
.../version-2.4.0/admin-api-partitioned-topics.md | 1 -
.../version-2.4.0/admin-api-permissions.md | 3 +-
.../version-2.4.0/admin-api-persistent-topics.md | 1 -
.../version-2.4.0/admin-api-schemas.md | 1 -
.../version-2.4.0/admin-api-tenants.md | 19 +-
.../version-2.4.0/administration-dashboard.md | 12 +-
.../version-2.4.0/administration-load-balance.md | 43 +-
.../version-2.4.0/administration-proxy.md | 3 +-
.../version-2.4.0/administration-stats.md | 1 -
.../version-2.4.0/client-libraries-cpp.md | 435 +++++++++++++++++----
.../version-2.4.0/client-libraries-websocket.md | 82 +++-
.../version-2.4.0/client-libraries.md | 4 +-
.../concepts-architecture-overview.md | 3 +-
.../version-2.4.0/concepts-authentication.md | 1 -
.../version-2.4.0/concepts-clients.md | 1 -
.../version-2.4.0/concepts-multi-tenancy.md | 1 -
.../version-2.4.0/concepts-overview.md | 1 -
.../version-2.4.0/concepts-replication.md | 1 -
.../version-2.4.0/concepts-topic-compaction.md | 1 -
.../version-2.4.0/cookbooks-bookkeepermetadata.md | 1 -
.../version-2.4.0/cookbooks-deduplication.md | 3 +-
.../version-2.4.0/cookbooks-encryption.md | 3 +-
.../version-2.4.0/cookbooks-message-queue.md | 1 -
.../version-2.4.0/cookbooks-non-persistent.md | 1 -
.../version-2.4.0/cookbooks-partitioned.md | 1 -
.../version-2.4.0/cookbooks-retention-expiry.md | 128 ++++--
.../versioned_docs/version-2.4.0/deploy-aws.md | 3 +-
.../deploy-bare-metal-multi-cluster.md | 101 ++---
.../versioned_docs/version-2.4.0/deploy-dcos.md | 56 +--
.../version-2.4.0/deploy-kubernetes.md | 1 -
.../version-2.4.0/deploy-monitoring.md | 15 +-
...nary-protocol.md => develop-binary-protocol.md} | 0
.../{developing-cpp.md => develop-cpp.md} | 0
.../version-2.4.0/develop-load-manager.md | 1 -
.../versioned_docs/version-2.4.0/develop-tools.md | 1 -
.../version-2.4.0/functions-deploying.md | 256 ------------
.../version-2.4.0/functions-guarantees.md | 42 --
.../version-2.4.0/functions-metrics.md | 1 -
.../versioned_docs/version-2.4.0/io-cdc.md | 1 -
.../versioned_docs/version-2.4.0/io-connectors.md | 18 +-
.../versioned_docs/version-2.4.0/io-develop.md | 1 -
.../versioned_docs/version-2.4.0/io-overview.md | 1 -
.../versioned_docs/version-2.4.0/pulsar-2.0.md | 1 -
.../{reference-pulsar-admin.md => pulsar-admin.md} | 0
.../version-2.4.0/reference-cli-tools.md | 12 +-
.../version-2.4.0/reference-terminology.md | 1 -
.../version-2.4.0/security-athenz.md | 1 -
.../version-2.4.0/security-authorization.md | 1 -
.../version-2.4.0/security-encryption.md | 1 -
.../version-2.4.0/security-extending.md | 1 -
.../version-2.4.0/sql-deployment-configurations.md | 94 ++++-
.../versioned_docs/version-2.4.0/sql-overview.md | 1 -
.../version-2.4.0/standalone-docker.md | 150 ++++---
...getting-started-standalone.md => standalone.md} | 0
.../versioned_docs/version-2.4.1/adaptors-kafka.md | 3 +-
.../versioned_docs/version-2.4.1/adaptors-spark.md | 1 -
.../versioned_docs/version-2.4.1/adaptors-storm.md | 1 -
.../version-2.4.1/admin-api-brokers.md | 5 +-
.../version-2.4.1/admin-api-clusters.md | 7 +-
.../version-2.4.1/admin-api-namespaces.md | 124 +++---
.../version-2.4.1/admin-api-permissions.md | 3 +-
.../version-2.4.1/admin-api-persistent-topics.md | 1 -
.../version-2.4.1/admin-api-schemas.md | 1 -
.../version-2.4.1/admin-api-tenants.md | 19 +-
.../version-2.4.1/administration-load-balance.md | 43 +-
.../version-2.4.1/administration-proxy.md | 3 +-
.../version-2.4.1/administration-stats.md | 1 -
.../version-2.4.1/administration-zk-bk.md | 1 -
.../version-2.4.1/client-libraries-cpp.md | 435 +++++++++++++++++----
.../version-2.4.1/client-libraries-go.md | 162 +++++++-
.../version-2.4.1/client-libraries-websocket.md | 82 +++-
.../version-2.4.1/client-libraries.md | 4 +-
.../concepts-architecture-overview.md | 3 +-
.../version-2.4.1/concepts-authentication.md | 1 -
.../version-2.4.1/concepts-clients.md | 1 -
.../version-2.4.1/concepts-multi-tenancy.md | 1 -
.../version-2.4.1/concepts-overview.md | 1 -
.../version-2.4.1/concepts-replication.md | 1 -
.../version-2.4.1/concepts-topic-compaction.md | 1 -
.../version-2.4.1/cookbooks-bookkeepermetadata.md | 1 -
.../version-2.4.1/cookbooks-deduplication.md | 3 +-
.../version-2.4.1/cookbooks-encryption.md | 3 +-
.../version-2.4.1/cookbooks-message-queue.md | 1 -
.../version-2.4.1/cookbooks-non-persistent.md | 1 -
.../version-2.4.1/cookbooks-partitioned.md | 1 -
.../version-2.4.1/cookbooks-retention-expiry.md | 128 ++++--
.../version-2.4.1/cookbooks-tiered-storage.md | 7 +-
.../versioned_docs/version-2.4.1/deploy-aws.md | 3 +-
.../deploy-bare-metal-multi-cluster.md | 101 ++---
.../deploy-bare-metal.md} | 0
.../versioned_docs/version-2.4.1/deploy-dcos.md | 56 +--
.../version-2.4.1/deploy-kubernetes.md | 1 -
.../version-2.4.1/deploy-monitoring.md | 15 +-
.../version-2.4.1/develop-binary-protocol.md | 105 +++--
.../version-2.4.1/develop-load-manager.md | 1 -
.../versioned_docs/version-2.4.1/develop-tools.md | 1 -
.../versioned_docs/version-2.4.1/io-cdc.md | 1 -
.../versioned_docs/version-2.4.1/io-overview.md | 1 -
.../versioned_docs/version-2.4.1/io-quickstart.md | 2 +-
.../versioned_docs/version-2.4.1/pulsar-2.0.md | 1 -
.../{reference-pulsar-admin.md => pulsar-admin.md} | 0
.../version-2.4.1/reference-cli-tools.md | 172 +++++---
.../version-2.4.1/reference-terminology.md | 1 -
.../version-2.4.1/security-encryption.md | 1 -
.../version-2.4.1/security-extending.md | 1 -
.../versioned_docs/version-2.4.1/security-jwt.md | 1 -
.../version-2.4.1/security-token-admin.md | 1 -
.../version-2.4.1/sql-deployment-configurations.md | 94 ++++-
.../version-2.4.1/sql-getting-started.md | 1 -
.../versioned_docs/version-2.4.1/sql-overview.md | 1 -
.../version-2.4.1/standalone-docker.md | 150 ++++---
...getting-started-standalone.md => standalone.md} | 0
.../versioned_docs/version-2.4.2/adaptors-kafka.md | 3 +-
.../versioned_docs/version-2.4.2/adaptors-spark.md | 1 -
.../versioned_docs/version-2.4.2/adaptors-storm.md | 1 -
.../version-2.4.2/admin-api-brokers.md | 5 +-
.../version-2.4.2/admin-api-clusters.md | 7 +-
.../version-2.4.2/admin-api-namespaces.md | 124 +++---
.../version-2.4.2/admin-api-permissions.md | 3 +-
.../version-2.4.2/admin-api-persistent-topics.md | 1 -
.../version-2.4.2/admin-api-schemas.md | 1 -
.../version-2.4.2/admin-api-tenants.md | 19 +-
.../version-2.4.2/administration-geo.md | 2 +-
.../version-2.4.2/administration-load-balance.md | 43 +-
.../version-2.4.2/administration-proxy.md | 3 +-
.../version-2.4.2/administration-stats.md | 1 -
.../version-2.4.2/administration-zk-bk.md | 1 -
.../version-2.4.2/client-libraries-cpp.md | 435 +++++++++++++++++----
.../version-2.4.2/client-libraries-go.md | 162 +++++++-
.../version-2.4.2/client-libraries-websocket.md | 82 +++-
.../version-2.4.2/client-libraries.md | 4 +-
.../concepts-architecture-overview.md | 3 +-
.../version-2.4.2/concepts-authentication.md | 1 -
.../version-2.4.2/concepts-clients.md | 1 -
.../version-2.4.2/concepts-multi-tenancy.md | 1 -
.../version-2.4.2/concepts-overview.md | 1 -
.../version-2.4.2/concepts-replication.md | 1 -
.../version-2.4.2/concepts-topic-compaction.md | 1 -
.../version-2.4.2/cookbooks-bookkeepermetadata.md | 1 -
.../version-2.4.2/cookbooks-deduplication.md | 3 +-
.../version-2.4.2/cookbooks-encryption.md | 3 +-
.../version-2.4.2/cookbooks-message-queue.md | 1 -
.../version-2.4.2/cookbooks-non-persistent.md | 1 -
.../version-2.4.2/cookbooks-partitioned.md | 1 -
.../version-2.4.2/cookbooks-retention-expiry.md | 128 ++++--
.../version-2.4.2/cookbooks-tiered-storage.md | 7 +-
.../versioned_docs/version-2.4.2/deploy-aws.md | 3 +-
.../deploy-bare-metal-multi-cluster.md | 101 ++---
.../deploy-bare-metal.md} | 0
.../versioned_docs/version-2.4.2/deploy-dcos.md | 56 +--
.../version-2.4.2/deploy-kubernetes.md | 1 -
.../version-2.4.2/deploy-monitoring.md | 15 +-
.../version-2.4.2/develop-binary-protocol.md | 105 +++--
.../version-2.4.2/develop-load-manager.md | 1 -
.../versioned_docs/version-2.4.2/develop-tools.md | 1 -
.../versioned_docs/version-2.4.2/io-cdc.md | 1 -
.../versioned_docs/version-2.4.2/io-overview.md | 1 -
.../versioned_docs/version-2.4.2/pulsar-2.0.md | 1 -
.../{reference-pulsar-admin.md => pulsar-admin.md} | 0
.../version-2.4.2/reference-cli-tools.md | 172 +++++---
.../version-2.4.2/reference-terminology.md | 1 -
.../version-2.4.2/security-encryption.md | 1 -
.../version-2.4.2/security-extending.md | 1 -
.../versioned_docs/version-2.4.2/security-jwt.md | 1 -
.../version-2.4.2/security-token-admin.md | 1 -
.../version-2.4.2/sql-deployment-configurations.md | 94 ++++-
.../version-2.4.2/sql-getting-started.md | 1 -
.../versioned_docs/version-2.4.2/sql-overview.md | 1 -
.../version-2.4.2/standalone-docker.md | 150 ++++---
...getting-started-standalone.md => standalone.md} | 0
site2/website-next/versions.json | 2 +-
179 files changed, 3363 insertions(+), 1829 deletions(-)
diff --git a/site2/website-next/migrate/migrate-chapter.js b/site2/website-next/migrate/migrate-chapter.js
index 102629c..5f042fb 100644
--- a/site2/website-next/migrate/migrate-chapter.js
+++ b/site2/website-next/migrate/migrate-chapter.js
@@ -43,6 +43,16 @@ const migrate = (version, category, cb) => {
return;
}
+ console.log(" [" + version + ":" + category + "]migrate...");
+ let existsSidebar = [];
+ for (let docsId of sidebar) {
+ let mdpath = migrateDocs(version, category, docsId, cb);
+ if (mdpath) {
+ existsSidebar.push(docsId);
+ }
+ }
+ sidebar = existsSidebar;
+
let new_sidebar_file = "";
let new_sidebar = {};
@@ -117,11 +127,6 @@ const migrate = (version, category, cb) => {
}
}
fs.writeFileSync(new_sidebar_file, JSON.stringify(new_sidebar, null, 2));
-
- console.log(" [" + version + ":" + category + "]migrate...");
- for (let docsId of sidebar) {
- migrateDocs(version, category, docsId, cb);
- }
};
module.exports = migrate;
diff --git a/site2/website-next/migrate/migrate-docs.js b/site2/website-next/migrate/migrate-docs.js
index e483c93..31f09a1 100644
--- a/site2/website-next/migrate/migrate-docs.js
+++ b/site2/website-next/migrate/migrate-docs.js
@@ -42,6 +42,7 @@ function migrate(version, chapter, docsId, cb) {
" was not fund, skip..."
);
}
+ return mdpath;
}
module.exports = migrate;
diff --git a/site2/website-next/versioned_docs/version-2.4.0/adaptors-spark.md b/site2/website-next/versioned_docs/version-2.4.0/adaptors-spark.md
index e14f13b..afa5a7e 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/adaptors-spark.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/adaptors-spark.md
@@ -2,7 +2,6 @@
id: adaptors-spark
title: Pulsar adaptor for Apache Spark
sidebar_label: "Apache Spark"
-original_id: adaptors-spark
---
## Spark Streaming receiver
diff --git a/site2/website-next/versioned_docs/version-2.4.0/adaptors-storm.md b/site2/website-next/versioned_docs/version-2.4.0/adaptors-storm.md
index 76d5071..9df9076 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/adaptors-storm.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/adaptors-storm.md
@@ -2,7 +2,6 @@
id: adaptors-storm
title: Pulsar adaptor for Apache Storm
sidebar_label: "Apache Storm"
-original_id: adaptors-storm
---
Pulsar Storm is an adaptor for integrating with [Apache Storm](http://storm.apache.org/) topologies. It provides core Storm implementations for sending and receiving data.
diff --git a/site2/website-next/versioned_docs/version-2.4.0/admin-api-brokers.md b/site2/website-next/versioned_docs/version-2.4.0/admin-api-brokers.md
index dbac453..10a90ca 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/admin-api-brokers.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/admin-api-brokers.md
@@ -2,7 +2,6 @@
id: admin-api-brokers
title: Managing Brokers
sidebar_label: "Brokers"
-original_id: admin-api-brokers
---
import Tabs from '@theme/Tabs';
@@ -26,9 +25,9 @@ Pulsar brokers consist of two components:
[Brokers](reference-terminology.md#broker) can be managed via:
-* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin) tool
+* The `brokers` command of the [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) tool
* The `/admin/v2/brokers` endpoint of the admin {@inject: rest:REST:/} API
-* The `brokers` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin.html} object in the [Java API](client-libraries-java)
+* The `brokers` method of the `PulsarAdmin` object in the [Java API](client-libraries-java)
In addition to being configurable when you start them up, brokers can also be [dynamically configured](#dynamic-broker-configuration).
diff --git a/site2/website-next/versioned_docs/version-2.4.0/admin-api-clusters.md b/site2/website-next/versioned_docs/version-2.4.0/admin-api-clusters.md
index 972c7e1..8687ae6 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/admin-api-clusters.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/admin-api-clusters.md
@@ -2,7 +2,6 @@
id: admin-api-clusters
title: Managing Clusters
sidebar_label: "Clusters"
-original_id: admin-api-clusters
---
import Tabs from '@theme/Tabs';
@@ -13,7 +12,7 @@ import TabItem from '@theme/TabItem';
>
> This page only shows **some frequently used operations**.
>
-> - For the latest and complete information about `Pulsar admin`, including commands, flags, descriptions, and more information, see [Pulsar admin doc](https://pulsar.apache.org/tools/pulsar-admin/).
+> - For the latest and complete information about `Pulsar admin`, including commands, flags, descriptions, and more, see [Pulsar admin doc](https://pulsar.apache.org/tools/pulsar-admin/)
>
> - For the latest and complete information about `REST API`, including parameters, responses, samples, and more, see {@inject: rest:REST:/} API doc.
>
@@ -24,9 +23,9 @@ servers (aka [bookies](reference-terminology.md#bookie)), and a [ZooKeeper](http
Clusters can be managed via:
-* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin) tool
+* The `clusters` command of the [`pulsar-admin`]([reference-pulsar-admin.md](https://pulsar.apache.org/tools/pulsar-admin/)) tool
* The `/admin/v2/clusters` endpoint of the admin {@inject: rest:REST:/} API
-* The `clusters` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java)
+* The `clusters` method of the `PulsarAdmin` object in the [Java API](client-libraries-java)
## Clusters resources
diff --git a/site2/website-next/versioned_docs/version-2.4.0/admin-api-namespaces.md b/site2/website-next/versioned_docs/version-2.4.0/admin-api-namespaces.md
index 216cb6f..c53fa3c 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/admin-api-namespaces.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/admin-api-namespaces.md
@@ -2,7 +2,6 @@
id: admin-api-namespaces
title: Managing Namespaces
sidebar_label: "Namespaces"
-original_id: admin-api-namespaces
---
import Tabs from '@theme/Tabs';
@@ -23,9 +22,9 @@ Pulsar [namespaces](reference-terminology.md#namespace) are logical groupings of
Namespaces can be managed via:
-* The [`namespaces`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin) tool
+* The `namespaces` command of the [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) tool
* The `/admin/v2/namespaces` endpoint of the admin {@inject: rest:REST:/} API
-* The `namespaces` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java)
+* The `namespaces` method of the `PulsarAdmin` object in the [Java API](client-libraries-java)
## Namespaces resources
@@ -49,8 +48,12 @@ $ pulsar-admin namespaces create test-tenant/test-namespace
</TabItem>
<TabItem value="REST API">
+```
+
{@inject: endpoint|PUT|/admin/v2/namespaces/:tenant/:namespace|operation/createNamespace?version=@pulsar:version_number@}
+```
+
</TabItem>
<TabItem value="Java">
@@ -105,8 +108,12 @@ $ pulsar-admin namespaces policies test-tenant/test-namespace
</TabItem>
<TabItem value="REST API">
+```
+
{@inject: endpoint|GET|/admin/v2/namespaces/:tenant/:namespace|operation/getPolicies?version=@pulsar:version_number@}
+```
+
</TabItem>
<TabItem value="Java">
@@ -142,8 +149,12 @@ test-tenant/ns2
</TabItem>
<TabItem value="REST API">
+```
+
{@inject: endpoint|GET|/admin/v2/namespaces/:tenant|operation/getTenantNamespaces?version=@pulsar:version_number@}
+```
+
</TabItem>
<TabItem value="Java">
@@ -177,8 +188,12 @@ $ pulsar-admin namespaces delete test-tenant/ns1
</TabItem>
<TabItem value="REST API">
+```
+
{@inject: endpoint|DELETE|/admin/v2/namespaces/:tenant/:namespace|operation/deleteNamespace?version=@pulsar:version_number@}
+```
+
</TabItem>
<TabItem value="Java">
@@ -196,7 +211,7 @@ admin.namespaces().deleteNamespace(namespace);
#### Set replication cluster
-It sets replication clusters for a namespace, so Pulsar can internally replicate publish message from one colo to another colo.
+You can set replication clusters for a namespace to enable Pulsar to internally replicate the published messages from one colocation facility to another.
<Tabs
defaultValue="pulsar-admin"
@@ -234,7 +249,7 @@ admin.namespaces().setNamespaceReplicationClusters(namespace, clusters);
#### Get replication cluster
-It gives a list of replication clusters for a given namespace.
+You can get the list of replication clusters for a given namespace.
<Tabs
defaultValue="pulsar-admin"
@@ -281,13 +296,13 @@ admin.namespaces().getNamespaceReplicationClusters(namespace)
Backlog quota helps the broker to restrict bandwidth/storage of a namespace once it reaches a certain threshold limit. Admin can set the limit and take corresponding action after the limit is reached.
- 1. producer_request_hold: broker will hold and not persist produce request payload
+ 1. producer_request_hold: broker holds but not persists produce request payload
- 2. producer_exception: broker disconnects with the client by giving an exception.
+ 2. producer_exception: broker disconnects with the client by giving an exception
- 3. consumer_backlog_eviction: broker will start discarding backlog messages
+ 3. consumer_backlog_eviction: broker starts discarding backlog messages
- Backlog quota restriction can be taken care by defining restriction of backlog-quota-type: destination_storage
+Backlog quota restriction can be taken care by defining restriction of backlog-quota-type: destination_storage.
<Tabs
defaultValue="pulsar-admin"
@@ -300,12 +315,6 @@ $ pulsar-admin namespaces set-backlog-quota --limit 10G --limitTime 36000 --poli
```
-```
-
-N/A
-
-```
-
</TabItem>
<TabItem value="REST API">
@@ -330,7 +339,7 @@ admin.namespaces().setBacklogQuota(namespace, new BacklogQuota(limit, limitTime,
#### Get backlog quota policies
-It shows a configured backlog quota for a given namespace.
+You can get a configured backlog quota for a given namespace.
<Tabs
defaultValue="pulsar-admin"
@@ -378,7 +387,7 @@ admin.namespaces().getBacklogQuotaMap(namespace);
#### Remove backlog quota policies
-It removes backlog quota policies for a given namespace
+You can remove backlog quota policies for a given namespace.
<Tabs
defaultValue="pulsar-admin"
@@ -391,12 +400,6 @@ $ pulsar-admin namespaces remove-backlog-quota test-tenant/ns1
```
-```
-
-N/A
-
-```
-
</TabItem>
<TabItem value="REST API">
@@ -423,7 +426,7 @@ admin.namespaces().removeBacklogQuota(namespace, backlogQuotaType)
#### Set persistence policies
-Persistence policies allow to configure persistency-level for all topic messages under a given namespace.
+Persistence policies allow users to configure persistency-level for all topic messages under a given namespace.
- Bookkeeper-ack-quorum: Number of acks (guaranteed copies) to wait for each entry, default: 0
@@ -444,12 +447,6 @@ $ pulsar-admin namespaces set-persistence --bookkeeper-ack-quorum 2 --bookkeeper
```
-```
-
-N/A
-
-```
-
</TabItem>
<TabItem value="REST API">
@@ -474,7 +471,7 @@ admin.namespaces().setPersistence(namespace,new PersistencePolicies(bookkeeperEn
#### Get persistence policies
-It shows the configured persistence policies of a given namespace.
+You can get the configured persistence policies of a given namespace.
<Tabs
defaultValue="pulsar-admin"
@@ -537,12 +534,6 @@ $ pulsar-admin namespaces unload --bundle 0x00000000_0xffffffff test-tenant/ns1
```
-```
-
-N/A
-
-```
-
</TabItem>
<TabItem value="REST API">
@@ -567,8 +558,7 @@ admin.namespaces().unloadNamespaceBundle(namespace, bundle)
#### Split namespace bundles
-Each namespace bundle can contain multiple topics and each bundle can be served by only one broker.
-If a single bundle is creating an excessive load on a broker, an admin splits the bundle using this command permitting one or more of the new bundles to be unloaded thus spreading the load across the brokers.
+One namespace bundle can contain multiple topics but can be served by only one broker. If a single bundle is creating an excessive load on a broker, an admin can split the bundle using the command below, permitting one or more of the new bundles to be unloaded, thus balancing the load across the brokers.
<Tabs
defaultValue="pulsar-admin"
@@ -581,12 +571,6 @@ $ pulsar-admin namespaces split-bundle --bundle 0x00000000_0xffffffff test-tenan
```
-```
-
-N/A
-
-```
-
</TabItem>
<TabItem value="REST API">
@@ -613,7 +597,7 @@ admin.namespaces().splitNamespaceBundle(namespace, bundle)
#### Set message-ttl
-It configures message’s time to live (in seconds) duration.
+You can configure the time to live (in seconds) duration for messages. In the example below, the message-ttl is set as 100s.
<Tabs
defaultValue="pulsar-admin"
@@ -626,12 +610,6 @@ $ pulsar-admin namespaces set-message-ttl --messageTTL 100 test-tenant/ns1
```
-```
-
-N/A
-
-```
-
</TabItem>
<TabItem value="REST API">
@@ -656,7 +634,7 @@ admin.namespaces().setNamespaceMessageTTL(namespace, messageTTL)
#### Get message-ttl
-It gives a message ttl of configured namespace.
+When the message-ttl for a namespace is set, you can use the command below to get the configured value. This example comtinues the example of the command `set message-ttl`, so the returned value is 100(s).
<Tabs
defaultValue="pulsar-admin"
@@ -684,6 +662,12 @@ $ pulsar-admin namespaces get-message-ttl test-tenant/ns1
```
+```
+
+100
+
+```
+
</TabItem>
<TabItem value="Java">
@@ -693,6 +677,12 @@ admin.namespaces().getNamespaceMessageTTL(namespace)
```
+```
+
+100
+
+```
+
</TabItem>
</Tabs>
@@ -712,12 +702,6 @@ $ pulsar-admin namespaces remove-message-ttl test-tenant/ns1
```
-```
-
-100
-
-```
-
</TabItem>
<TabItem value="REST API">
@@ -758,12 +742,6 @@ $ pulsar-admin namespaces clear-backlog --sub my-subscription test-tenant/ns1
```
-```
-
-N/A
-
-```
-
</TabItem>
<TabItem value="REST API">
@@ -801,12 +779,6 @@ $ pulsar-admin namespaces clear-backlog --bundle 0x00000000_0xffffffff --sub m
```
-```
-
-N/A
-
-```
-
</TabItem>
<TabItem value="REST API">
@@ -842,13 +814,7 @@ Each namespace contains multiple topics and the retention size (storage size) of
```
-$ pulsar-admin set-retention --size 100 --time 10 test-tenant/ns1
-
-```
-
-```
-
-N/A
+$ pulsar-admin namespaces set-retention --size 100 --time 10 test-tenant/ns1
```
@@ -932,9 +898,7 @@ disables the throttling.
:::note
- If neither `clusterDispatchRate` nor `topicDispatchRate` is configured, dispatch throttling is disabled.
->
- If `topicDispatchRate` is not configured, `clusterDispatchRate` takes effect.
->
- If `topicDispatchRate` is configured, `topicDispatchRate` takes effect.
:::
diff --git a/site2/website-next/versioned_docs/version-2.4.0/admin-api-non-persistent-topics.md b/site2/website-next/versioned_docs/version-2.4.0/admin-api-non-persistent-topics.md
index 12220de..78dac35 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/admin-api-non-persistent-topics.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/admin-api-non-persistent-topics.md
@@ -2,7 +2,6 @@
id: admin-api-non-persistent-topics
title: Managing non-persistent topics
sidebar_label: "Non-Persistent topics"
-original_id: admin-api-non-persistent-topics
---
For details of the content, refer to [manage topics](admin-api-topics).
\ No newline at end of file
diff --git a/site2/website-next/versioned_docs/version-2.4.0/admin-api-partitioned-topics.md b/site2/website-next/versioned_docs/version-2.4.0/admin-api-partitioned-topics.md
index 6734586..7221b3d 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/admin-api-partitioned-topics.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/admin-api-partitioned-topics.md
@@ -2,7 +2,6 @@
id: admin-api-partitioned-topics
title: Managing partitioned topics
sidebar_label: "Partitioned topics"
-original_id: admin-api-partitioned-topics
---
For details of the content, refer to [manage topics](admin-api-topics).
\ No newline at end of file
diff --git a/site2/website-next/versioned_docs/version-2.4.0/admin-api-permissions.md b/site2/website-next/versioned_docs/version-2.4.0/admin-api-permissions.md
index e2ca469..faedbf1 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/admin-api-permissions.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/admin-api-permissions.md
@@ -2,7 +2,6 @@
id: admin-api-permissions
title: Managing permissions
sidebar_label: "Permissions"
-original_id: admin-api-permissions
---
import Tabs from '@theme/Tabs';
@@ -13,7 +12,7 @@ import TabItem from '@theme/TabItem';
>
> This page only shows **some frequently used operations**.
>
-> - For the latest and complete information about `Pulsar admin`, including commands, flags, descriptions, and more information, see [Pulsar admin doc](https://pulsar.apache.org/tools/pulsar-admin/).
+> - For the latest and complete information about `Pulsar admin`, including commands, flags, descriptions, and more, see [Pulsar admin doc](https://pulsar.apache.org/tools/pulsar-admin/)
>
> - For the latest and complete information about `REST API`, including parameters, responses, samples, and more, see {@inject: rest:REST:/} API doc.
>
diff --git a/site2/website-next/versioned_docs/version-2.4.0/admin-api-persistent-topics.md b/site2/website-next/versioned_docs/version-2.4.0/admin-api-persistent-topics.md
index b6d293b..8a7abae 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/admin-api-persistent-topics.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/admin-api-persistent-topics.md
@@ -2,7 +2,6 @@
id: admin-api-persistent-topics
title: Managing persistent topics
sidebar_label: "Persistent topics"
-original_id: admin-api-persistent-topics
---
For details of the content, refer to [manage topics](admin-api-topics).
\ No newline at end of file
diff --git a/site2/website-next/versioned_docs/version-2.4.0/admin-api-schemas.md b/site2/website-next/versioned_docs/version-2.4.0/admin-api-schemas.md
index 9ffe21f..8399a03 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/admin-api-schemas.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/admin-api-schemas.md
@@ -2,6 +2,5 @@
id: admin-api-schemas
title: Managing Schemas
sidebar_label: "Schemas"
-original_id: admin-api-schemas
---
diff --git a/site2/website-next/versioned_docs/version-2.4.0/admin-api-tenants.md b/site2/website-next/versioned_docs/version-2.4.0/admin-api-tenants.md
index fe68336..570ac31 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/admin-api-tenants.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/admin-api-tenants.md
@@ -2,7 +2,6 @@
id: admin-api-tenants
title: Managing Tenants
sidebar_label: "Tenants"
-original_id: admin-api-tenants
---
import Tabs from '@theme/Tabs';
@@ -13,7 +12,7 @@ import TabItem from '@theme/TabItem';
>
> This page only shows **some frequently used operations**.
>
-> - For the latest and complete information about `Pulsar admin`, including commands, flags, descriptions, and more information, see [Pulsar admin doc](https://pulsar.apache.org/tools/pulsar-admin/).
+> - For the latest and complete information about `Pulsar admin`, including commands, flags, descriptions, and more, see [Pulsar admin doc](https://pulsar.apache.org/tools/pulsar-admin/)
>
> - For the latest and complete information about `REST API`, including parameters, responses, samples, and more, see {@inject: rest:REST:/} API doc.
>
@@ -80,22 +79,26 @@ $ pulsar-admin tenants create my-tenant
```
-When creating a tenant, you can assign admin roles using the `-r`/`--admin-roles` flag. You can specify multiple roles as a comma-separated list. Here are some examples:
+When creating a tenant, you can optionally assign admin roles using the `-r`/`--admin-roles`
+flag, and clusters using the `-c`/`--allowed-clusters` flag. You can specify multiple values
+as a comma-separated list. Here are some examples:
```shell
$ pulsar-admin tenants create my-tenant \
- --admin-roles role1,role2,role3
+ --admin-roles role1,role2,role3 \
+ --allowed-clusters cluster1
$ pulsar-admin tenants create my-tenant \
-r role1
+ -c cluster1
```
</TabItem>
<TabItem value="REST API">
-{@inject: endpoint|POST|/admin/v2/tenants/:tenant|operation/createTenant?version=@pulsar:version_number@}
+{@inject: endpoint|PUT|/admin/v2/tenants/:tenant|operation/createTenant?version=@pulsar:version_number@}
</TabItem>
<TabItem value="JAVA">
@@ -140,7 +143,7 @@ $ pulsar-admin tenants get my-tenant
</TabItem>
<TabItem value="REST API">
-{@inject: endpoint|GET|/admin/v2/tenants/:cluster|operation/getTenant?version=@pulsar:version_number@}
+{@inject: endpoint|GET|/admin/v2/tenants/:tenant|operation/getTenant?version=@pulsar:version_number@}
</TabItem>
<TabItem value="JAVA">
@@ -175,7 +178,7 @@ $ pulsar-admin tenants delete my-tenant
</TabItem>
<TabItem value="REST API">
-{@inject: endpoint|DELETE|/admin/v2/tenants/:cluster|operation/deleteTenant?version=@pulsar:version_number@}
+{@inject: endpoint|DELETE|/admin/v2/tenants/:tenant|operation/deleteTenant?version=@pulsar:version_number@}
</TabItem>
<TabItem value="JAVA">
@@ -210,7 +213,7 @@ $ pulsar-admin tenants update my-tenant
</TabItem>
<TabItem value="REST API">
-{@inject: endpoint|DELETE|/admin/v2/tenants/:cluster|operation/updateTenant?version=@pulsar:version_number@}
+{@inject: endpoint|POST|/admin/v2/tenants/:tenant|operation/updateTenant?version=@pulsar:version_number@}
</TabItem>
<TabItem value="JAVA">
diff --git a/site2/website-next/versioned_docs/version-2.4.0/administration-dashboard.md b/site2/website-next/versioned_docs/version-2.4.0/administration-dashboard.md
index 514b076..1eb0404 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/administration-dashboard.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/administration-dashboard.md
@@ -7,7 +7,7 @@ original_id: administration-dashboard
:::note
-Pulsar dashboard is deprecated. If you want to manage and monitor the stats of your topics, use [Pulsar Manager](administration-pulsar-manager).
+Pulsar dashboard is deprecated. We recommend you use [Pulsar Manager](administration-pulsar-manager) to manage and monitor the stats of your topics.
:::
@@ -53,17 +53,17 @@ $ docker run -p 80:80 \
```
-You need to specify only one service URL for a Pulsar cluster. Internally, the collector figures out all the existing clusters and the brokers from where it needs to pull the metrics. If you connect the dashboard to Pulsar running in standalone mode, the URL is `http://<broker-ip>:8080` by default. `<broker-ip>` is the ip address or hostname of the machine running Pulsar standalone. The ip address or hostname should be accessible from the docker instance running dashboard.
+You need to specify only one service URL for a Pulsar cluster. Internally, the collector figures out all the existing clusters and the brokers from where it needs to pull the metrics. If you connect the dashboard to Pulsar running in standalone mode, the URL is `http://<broker-ip>:8080` by default. `<broker-ip>` is the IP address or hostname of the machine that runs Pulsar standalone. The IP address or hostname should be accessible from the running dashboard in the docker instance.
-Once the Docker container runs, the web dashboard is accessible via `localhost` or whichever host that Docker uses.
+Once the Docker container starts, the web dashboard is accessible via `localhost` or whichever host that Docker uses.
-> The `SERVICE_URL` that the dashboard uses needs to be reachable from inside the Docker container
+> The `SERVICE_URL` that the dashboard uses needs to be reachable from inside the Docker container.
If the Pulsar service runs in standalone mode in `localhost`, the `SERVICE_URL` has to
-be the IP of the machine.
+be the IP address of the machine.
Similarly, given the Pulsar standalone advertises itself with localhost by default, you need to
-explicitly set the advertise address to the host IP. For example:
+explicitly set the advertise address to the host IP address. For example:
```shell
diff --git a/site2/website-next/versioned_docs/version-2.4.0/administration-load-balance.md b/site2/website-next/versioned_docs/version-2.4.0/administration-load-balance.md
index 3efba60..834b156 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/administration-load-balance.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/administration-load-balance.md
@@ -2,13 +2,11 @@
id: administration-load-balance
title: Pulsar load balance
sidebar_label: "Load balance"
-original_id: administration-load-balance
---
## Load balance across Pulsar brokers
-Pulsar is an horizontally scalable messaging system, so the traffic
-in a logical cluster must be spread across all the available Pulsar brokers as evenly as possible, which is a core requirement.
+Pulsar is an horizontally scalable messaging system, so the traffic in a logical cluster must be balanced across all the available Pulsar brokers as evenly as possible, which is a core requirement.
You can use multiple settings and tools to control the traffic distribution which require a bit of context to understand how the traffic is managed in Pulsar. Though, in most cases, the core requirement mentioned above is true out of the box and you should not worry about it.
@@ -36,11 +34,9 @@ Instead of individual topic or partition assignment, each broker takes ownership
The namespace is the "administrative" unit: many config knobs or operations are done at the namespace level.
-For assignment, a namespaces is sharded into a list of "bundles", with each bundle comprising
-a portion of overall hash range of the namespace.
+For assignment, a namespaces is sharded into a list of "bundles", with each bundle comprising a portion of overall hash range of the namespace.
-Topics are assigned to a particular bundle by taking the hash of the topic name and checking in which
-bundle the hash falls into.
+Topics are assigned to a particular bundle by taking the hash of the topic name and checking in which bundle the hash falls into.
Each bundle is independent of the others and thus is independently assigned to different brokers.
@@ -72,8 +68,7 @@ On the same note, it is beneficial to start with more bundles than the number of
### Unload topics and bundles
-You can "unload" a topic in Pulsar with admin operation. Unloading means to close the topics,
-release ownership and reassign the topics to a new broker, based on current load.
+You can "unload" a topic in Pulsar with admin operation. Unloading means to close the topics, release ownership and reassign the topics to a new broker, based on current load.
When unloading happens, the client experiences a small latency blip, typically in the order of tens of milliseconds, while the topic is reassigned.
@@ -97,9 +92,11 @@ pulsar-admin namespaces unload tenant/namespace
### Split namespace bundles
-Since the load for the topics in a bundle might change over time, or predicting upfront might just be hard, brokers can split bundles into two. The new smaller bundles can be reassigned to different brokers.
+Since the load for the topics in a bundle might change over time and predicting the load might be hard, bundle split is designed to deal with these issues. The broker splits a bundle into two and the new smaller bundles can be reassigned to different brokers.
-The splitting happens based on some tunable thresholds. Any existing bundle that exceeds any of the threshold is a candidate to be split. By default the newly split bundles are also immediately offloaded to other brokers, to facilitate the traffic distribution.
+The splitting is based on some tunable thresholds. Any existing bundle that exceeds any of the threshold is a candidate to be split. By default the newly split bundles are also immediately offloaded to other brokers, to facilitate the traffic distribution.
+
+You can split namespace bundles in two ways, by setting `supportedNamespaceBundleSplitAlgorithms` to `range_equally_divide` or `topic_count_equally_divide` in `broker.conf` file. The former splits the bundle into two parts with the same hash range size; the latter splits the bundle into two parts with the same number of topics. You can also configure other parameters for namespace bundles.
```properties
@@ -130,13 +127,11 @@ loadBalancerNamespaceMaximumBundles=128
The support for automatic load shedding is available in the load manager of Pulsar. This means that whenever the system recognizes a particular broker is overloaded, the system forces some traffic to be reassigned to less loaded brokers.
-When a broker is identified as overloaded, the broker forces to "unload" a subset of the bundles, the
-ones with higher traffic, that make up for the overload percentage.
+When a broker is identified as overloaded, the broker forces to "unload" a subset of the bundles, the ones with higher traffic, that make up for the overload percentage.
For example, the default threshold is 85% and if a broker is over quota at 95% CPU usage, then the broker unloads the percent difference plus a 5% margin: `(95% - 85%) + 5% = 15%`.
-Given the selection of bundles to offload is based on traffic (as a proxy measure for cpu, network
-and memory), broker unloads bundles for at least 15% of traffic.
+Given the selection of bundles to offload is based on traffic (as a proxy measure for cpu, network and memory), broker unloads bundles for at least 15% of traffic.
The automatic load shedding is enabled by default and you can disable the automatic load shedding with this setting:
@@ -160,6 +155,20 @@ loadBalancerSheddingGracePeriodMinutes=30
```
+Pulsar supports three types of shedding strategies:
+
+##### ThresholdShedder
+This strategy tends to shed the bundles if any broker's usage is above the configured threshold. It does this by first computing the average resource usage per broker for the whole cluster. The resource usage for each broker is calculated using the following method: LocalBrokerData#getMaxResourceUsageWithWeight). The weights for each resource are configurable. Historical observations are included in the running average based on the broker's setting for loadBalancerHistoryResourcePercenta [...]
+`loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder`
+
+##### OverloadShedder
+This strategy will attempt to shed exactly one bundle on brokers which are overloaded, that is, whose maximum system resource usage exceeds loadBalancerBrokerOverloadedThresholdPercentage. To see which resources are considered when determining the maximum system resource. A bundle is recommended for unloading off that broker if and only if the following conditions hold: The broker has at least two bundles assigned and the broker has at least one bundle that has not been unloaded recently [...]
+`loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.impl.OverloadShedder`
+
+##### UniformLoadShedder
+This strategy tends to distribute load uniformly across all brokers. This strategy checks laod difference between broker with highest load and broker with lowest load. If the difference is higher than configured thresholds `loadBalancerMsgRateDifferenceShedderThreshold` and `loadBalancerMsgThroughputMultiplierDifferenceShedderThreshold` then it finds out bundles which can be unloaded to distribute traffic evenly across all brokers. Configure broker with below value to use this strategy.
+`loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.impl.UniformLoadShedder`
+
#### Broker overload thresholds
The determinations of when a broker is overloaded is based on threshold of CPU, network and memory usage. Whenever either of those metrics reaches the threshold, the system triggers the shedding (if enabled).
@@ -175,9 +184,7 @@ loadBalancerBrokerOverloadedThresholdPercentage=85
Pulsar gathers the usage stats from the system metrics.
-In case of network utilization, in some cases the network interface speed that Linux reports is
-not correct and needs to be manually overridden. This is the case in AWS EC2 instances with 1Gbps
-NIC speed for which the OS reports 10Gbps speed.
+In case of network utilization, in some cases the network interface speed that Linux reports is not correct and needs to be manually overridden. This is the case in AWS EC2 instances with 1Gbps NIC speed for which the OS reports 10Gbps speed.
Because of the incorrect max speed, the Pulsar load manager might think the broker has not reached the NIC capacity, while in fact the broker already uses all the bandwidth and the traffic is slowed down.
diff --git a/site2/website-next/versioned_docs/version-2.4.0/administration-proxy.md b/site2/website-next/versioned_docs/version-2.4.0/administration-proxy.md
index c046ed3..3cef937 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/administration-proxy.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/administration-proxy.md
@@ -2,10 +2,9 @@
id: administration-proxy
title: Pulsar proxy
sidebar_label: "Pulsar proxy"
-original_id: administration-proxy
---
-Pulsar proxy is an optional gateway. Pulsar proxy is used when direction connections between clients and Pulsar brokers are either infeasible or undesirable. For example, when you run Pulsar in a cloud environment or on [Kubernetes](https://kubernetes.io) or an analogous platform, you can run Pulsar proxy.
+Pulsar proxy is an optional gateway. Pulsar proxy is used when direct connections between clients and Pulsar brokers are either infeasible or undesirable. For example, when you run Pulsar in a cloud environment or on [Kubernetes](https://kubernetes.io) or an analogous platform, you can run Pulsar proxy.
## Configure the proxy
diff --git a/site2/website-next/versioned_docs/version-2.4.0/administration-stats.md b/site2/website-next/versioned_docs/version-2.4.0/administration-stats.md
index ac0c036..2ccd73c 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/administration-stats.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/administration-stats.md
@@ -2,7 +2,6 @@
id: administration-stats
title: Pulsar stats
sidebar_label: "Pulsar statistics"
-original_id: administration-stats
---
## Partitioned topics
diff --git a/site2/website-next/versioned_docs/version-2.4.0/client-libraries-cpp.md b/site2/website-next/versioned_docs/version-2.4.0/client-libraries-cpp.md
index 333ec67..958861a 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/client-libraries-cpp.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/client-libraries-cpp.md
@@ -2,7 +2,6 @@
id: client-libraries-cpp
title: Pulsar C++ client
sidebar_label: "C++"
-original_id: client-libraries-cpp
---
You can use Pulsar C++ client to create Pulsar producers and consumers in C++.
@@ -11,7 +10,7 @@ All the methods in producer, consumer, and reader of a C++ client are thread-saf
## Supported platforms
-Pulsar C++ client is supported on **Linux** and **MacOS** platforms.
+Pulsar C++ client is supported on **Linux** ,**MacOS** and **Windows** platforms.
[Doxygen](http://www.doxygen.nl/)-generated API docs for the C++ client are available [here](/api/cpp).
@@ -21,8 +20,8 @@ You need to install the following components before using the C++ client:
* [CMake](https://cmake.org/)
* [Boost](http://www.boost.org/)
-* [Protocol Buffers](https://developers.google.com/protocol-buffers/) 2.6
-* [libcurl](https://curl.haxx.se/libcurl/)
+* [Protocol Buffers](https://developers.google.com/protocol-buffers/) >= 3
+* [libcurl](https://curl.se/libcurl/)
* [Google Test](https://github.com/google/googletest)
## Linux
@@ -147,6 +146,12 @@ $ rpm -ivh apache-pulsar-client*.rpm
After you install RPM successfully, Pulsar libraries are in the `/usr/lib` directory.
+:::note
+
+If you get the error that `libpulsar.so: cannot open shared object file: No such file or directory` when starting Pulsar client, you may need to run `ldconfig` first.
+
+:::
+
### Install Debian
1. Download a Debian package from the links in the table.
@@ -236,10 +241,8 @@ $ export OPENSSL_INCLUDE_DIR=/usr/local/opt/openssl/include/
$ export OPENSSL_ROOT_DIR=/usr/local/opt/openssl/
# Protocol Buffers installation
-$ brew tap homebrew/versions
-$ brew install protobuf260
-$ brew install boost
-$ brew install log4cxx
+$ brew install protobuf boost boost-python log4cxx
+# If you are using python3, you need to install boost-python3
# Google Test installation
$ git clone https://github.com/google/googletest.git
@@ -269,6 +272,50 @@ brew install libpulsar
```
+## Windows (64-bit)
+
+### Compilation
+
+1. Clone the Pulsar repository.
+
+```shell
+
+$ git clone https://github.com/apache/pulsar
+
+```
+
+2. Install all necessary dependencies.
+
+```shell
+
+cd ${PULSAR_HOME}/pulsar-client-cpp
+vcpkg install --feature-flags=manifests --triplet x64-windows
+
+```
+
+3. Build C++ libraries.
+
+```shell
+
+cmake -B ./build -A x64 -DBUILD_PYTHON_WRAPPER=OFF -DBUILD_TESTS=OFF -DVCPKG_TRIPLET=x64-windows -DCMAKE_BUILD_TYPE=Release -S .
+cmake --build ./build --config Release
+
+```
+
+> **NOTE**
+>
+> 1. For Windows 32-bit, you need to use `-A Win32` and `-DVCPKG_TRIPLET=x86-windows`.
+> 2. For MSVC Debug mode, you need to replace `Release` with `Debug` for both `CMAKE_BUILD_TYPE` variable and `--config` option.
+
+4. Client libraries are available in the following places.
+
+```
+
+${PULSAR_HOME}/pulsar-client-cpp/build/lib/Release/pulsar.lib
+${PULSAR_HOME}/pulsar-client-cpp/build/lib/Release/pulsar.dll
+
+```
+
## Connection URLs
To connect Pulsar using client libraries, you need to specify a Pulsar protocol URL.
@@ -299,109 +346,361 @@ pulsar+ssl://pulsar.us-west.example.com:6651
## Create a consumer
-To use Pulsar as a consumer, you need to create a consumer on the C++ client. The following is an example.
+To use Pulsar as a consumer, you need to create a consumer on the C++ client. There are two main ways of using the consumer:
+- [Blocking style](#blocking-example): synchronously calling `receive(msg)`.
+- [Non-blocking](#consumer-with-a-message-listener) (event based) style: using a message listener.
+
+### Blocking example
+
+The benefit of this approach is that it is the simplest code. Simply keeps calling `receive(msg)` which blocks until a message is received.
+
+This example starts a subscription at the earliest offset and consumes 100 messages.
```c++
-Client client("pulsar://localhost:6650");
+#include <pulsar/Client.h>
+
+using namespace pulsar;
+
+int main() {
+ Client client("pulsar://localhost:6650");
+
+ Consumer consumer;
+ ConsumerConfiguration config;
+ config.setSubscriptionInitialPosition(InitialPositionEarliest);
+ Result result = client.subscribe("persistent://public/default/my-topic", "consumer-1", config, consumer);
+ if (result != ResultOk) {
+ std::cout << "Failed to subscribe: " << result << std::endl;
+ return -1;
+ }
-Consumer consumer;
-Result result = client.subscribe("my-topic", "my-subscription-name", consumer);
-if (result != ResultOk) {
- LOG_ERROR("Failed to subscribe: " << result);
- return -1;
+ Message msg;
+ int ctr = 0;
+ // consume 100 messages
+ while (ctr < 100) {
+ consumer.receive(msg);
+ std::cout << "Received: " << msg
+ << " with payload '" << msg.getDataAsString() << "'" << std::endl;
+
+ consumer.acknowledge(msg);
+ ctr++;
+ }
+
+ std::cout << "Finished consuming synchronously!" << std::endl;
+
+ client.close();
+ return 0;
}
-Message msg;
+```
+
+### Consumer with a message listener
+
+You can avoid running a loop with blocking calls with an event based style by using a message listener which is invoked for each message that is received.
+
+This example starts a subscription at the earliest offset and consumes 100 messages.
-while (true) {
- consumer.receive(msg);
- LOG_INFO("Received: " << msg
- << " with payload '" << msg.getDataAsString() << "'");
+```c++
+
+#include <pulsar/Client.h>
+#include <atomic>
+#include <thread>
+
+using namespace pulsar;
+
+std::atomic<uint32_t> messagesReceived;
+
+void handleAckComplete(Result res) {
+ std::cout << "Ack res: " << res << std::endl;
+}
- consumer.acknowledge(msg);
+void listener(Consumer consumer, const Message& msg) {
+ std::cout << "Got message " << msg << " with content '" << msg.getDataAsString() << "'" << std::endl;
+ messagesReceived++;
+ consumer.acknowledgeAsync(msg.getMessageId(), handleAckComplete);
}
-client.close();
+int main() {
+ Client client("pulsar://localhost:6650");
+
+ Consumer consumer;
+ ConsumerConfiguration config;
+ config.setMessageListener(listener);
+ config.setSubscriptionInitialPosition(InitialPositionEarliest);
+ Result result = client.subscribe("persistent://public/default/my-topic", "consumer-1", config, consumer);
+ if (result != ResultOk) {
+ std::cout << "Failed to subscribe: " << result << std::endl;
+ return -1;
+ }
+
+ // wait for 100 messages to be consumed
+ while (messagesReceived < 100) {
+ std::this_thread::sleep_for(std::chrono::milliseconds(100));
+ }
+
+ std::cout << "Finished consuming asynchronously!" << std::endl;
+
+ client.close();
+ return 0;
+}
```
## Create a producer
-To use Pulsar as a producer, you need to create a producer on the C++ client. The following is an example.
+To use Pulsar as a producer, you need to create a producer on the C++ client. There are two main ways of using a producer:
+- [Blocking style](#simple-blocking-example) : each call to `send` waits for an ack from the broker.
+- [Non-blocking asynchronous style](#non-blocking-example) : `sendAsync` is called instead of `send` and a callback is supplied for when the ack is received from the broker.
+
+### Simple blocking example
+
+This example sends 100 messages using the blocking style. While simple, it does not produce high throughput as it waits for each ack to come back before sending the next message.
```c++
-Client client("pulsar://localhost:6650");
+#include <pulsar/Client.h>
+#include <thread>
-Producer producer;
-Result result = client.createProducer("my-topic", producer);
-if (result != ResultOk) {
- LOG_ERROR("Error creating producer: " << result);
- return -1;
-}
+using namespace pulsar;
+
+int main() {
+ Client client("pulsar://localhost:6650");
+
+ Result result = client.createProducer("persistent://public/default/my-topic", producer);
+ if (result != ResultOk) {
+ std::cout << "Error creating producer: " << result << std::endl;
+ return -1;
+ }
+
+ // Send 100 messages synchronously
+ int ctr = 0;
+ while (ctr < 100) {
+ std::string content = "msg" + std::to_string(ctr);
+ Message msg = MessageBuilder().setContent(content).setProperty("x", "1").build();
+ Result result = producer.send(msg);
+ if (result != ResultOk) {
+ std::cout << "The message " << content << " could not be sent, received code: " << result << std::endl;
+ } else {
+ std::cout << "The message " << content << " sent successfully" << std::endl;
+ }
+
+ std::this_thread::sleep_for(std::chrono::milliseconds(100));
+ ctr++;
+ }
+
+ std::cout << "Finished producing synchronously!" << std::endl;
-// Publish 10 messages to the topic
-for (int i = 0; i < 10; i++){
- Message msg = MessageBuilder().setContent("my-message").build();
- Result res = producer.send(msg);
- LOG_INFO("Message sent: " << res);
+ client.close();
+ return 0;
}
-client.close();
```
-## Enable authentication in connection URLs
-If you use TLS authentication when connecting to Pulsar, you need to add `ssl` in the connection URLs, and the default port is `6651`. The following is an example.
+### Non-blocking example
-```cpp
+This example sends 100 messages using the non-blocking style calling `sendAsync` instead of `send`. This allows the producer to have multiple messages inflight at a time which increases throughput.
-ClientConfiguration config = ClientConfiguration();
-config.setUseTls(true);
-config.setTlsTrustCertsFilePath("/path/to/cacert.pem");
-config.setTlsAllowInsecureConnection(false);
-config.setAuth(pulsar::AuthTls::create(
- "/path/to/client-cert.pem", "/path/to/client-key.pem"););
+The producer configuration `blockIfQueueFull` is useful here to avoid `ResultProducerQueueIsFull` errors when the internal queue for outgoing send requests becomes full. Once the internal queue is full, `sendAsync` becomes blocking which can make your code simpler.
-Client client("pulsar+ssl://my-broker.com:6651", config);
+Without this configuration, the result code `ResultProducerQueueIsFull` is passed to the callback. You must decide how to deal with that (retry, discard etc).
+
+```c++
+
+#include <pulsar/Client.h>
+#include <thread>
+
+using namespace pulsar;
+
+std::atomic<uint32_t> acksReceived;
+
+void callback(Result code, const MessageId& msgId, std::string msgContent) {
+ // message processing logic here
+ std::cout << "Received ack for msg: " << msgContent << " with code: "
+ << code << " -- MsgID: " << msgId << std::endl;
+ acksReceived++;
+}
+
+int main() {
+ Client client("pulsar://localhost:6650");
+
+ ProducerConfiguration producerConf;
+ producerConf.setBlockIfQueueFull(true);
+ Producer producer;
+ Result result = client.createProducer("persistent://public/default/my-topic",
+ producerConf, producer);
+ if (result != ResultOk) {
+ std::cout << "Error creating producer: " << result << std::endl;
+ return -1;
+ }
+
+ // Send 100 messages asynchronously
+ int ctr = 0;
+ while (ctr < 100) {
+ std::string content = "msg" + std::to_string(ctr);
+ Message msg = MessageBuilder().setContent(content).setProperty("x", "1").build();
+ producer.sendAsync(msg, std::bind(callback,
+ std::placeholders::_1, std::placeholders::_2, content));
+
+ std::this_thread::sleep_for(std::chrono::milliseconds(100));
+ ctr++;
+ }
+
+ // wait for 100 messages to be acked
+ while (acksReceived < 100) {
+ std::this_thread::sleep_for(std::chrono::milliseconds(100));
+ }
+
+ std::cout << "Finished producing asynchronously!" << std::endl;
+
+ client.close();
+ return 0;
+}
```
-For complete examples, refer to [C++ client examples](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/examples).
+### Partitioned topics and lazy producers
-## Schema
+When scaling out a Pulsar topic, you may configure a topic to have hundreds of partitions. Likewise, you may have also scaled out your producers so there are hundreds or even thousands of producers. This can put some strain on the Pulsar brokers as when you create a producer on a partitioned topic, internally it creates one internal producer per partition which involves communications to the brokers for each one. So for a topic with 1000 partitions and 1000 producers, it ends up creating [...]
-This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started).
+You can reduce the load caused by this combination of a large number of partitions and many producers by doing the following:
+- use SinglePartition partition routing mode (this ensures that all messages are only sent to a single, randomly selected partition)
+- use non-keyed messages (when messages are keyed, routing is based on the hash of the key and so messages will end up being sent to multiple partitions)
+- use lazy producers (this ensures that an internal producer is only created on demand when a message needs to be routed to a partition)
-### Create producer with Avro schema
+With our example above, that reduces the number of internal producers spread out over the 1000 producer apps from 1,000,000 to just 1000.
-The following example shows how to create a producer with an Avro schema.
+Note that there can be extra latency for the first message sent. If you set a low send timeout, this timeout could be reached if the initial connection handshake is slow to complete.
-```cpp
+```c++
-static const std::string exampleSchema =
- "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
- "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
-Producer producer;
ProducerConfiguration producerConf;
-producerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
-client.createProducer("topic-avro", producerConf, producer);
+producerConf.setPartitionsRoutingMode(ProducerConfiguration::UseSinglePartition);
+producerConf.setLazyStartPartitionedProducers(true);
```
-### Create consumer with Avro schema
-
-The following example shows how to create a consumer with an Avro schema.
+## Enable authentication in connection URLs
+If you use TLS authentication when connecting to Pulsar, you need to add `ssl` in the connection URLs, and the default port is `6651`. The following is an example.
```cpp
-static const std::string exampleSchema =
- "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
- "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
-ConsumerConfiguration consumerConf;
-Consumer consumer;
-consumerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
-client.subscribe("topic-avro", "sub-2", consumerConf, consumer)
+ClientConfiguration config = ClientConfiguration();
+config.setUseTls(true);
+config.setTlsTrustCertsFilePath("/path/to/cacert.pem");
+config.setTlsAllowInsecureConnection(false);
+config.setAuth(pulsar::AuthTls::create(
+ "/path/to/client-cert.pem", "/path/to/client-key.pem"););
+
+Client client("pulsar+ssl://my-broker.com:6651", config);
```
+For complete examples, refer to [C++ client examples](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/examples).
+
+## Schema
+
+This section describes some examples about schema. For more information about
+schema, see [Pulsar schema](schema-get-started).
+
+### Avro schema
+
+- The following example shows how to create a producer with an Avro schema.
+
+ ```cpp
+
+ static const std::string exampleSchema =
+ "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
+ "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
+ Producer producer;
+ ProducerConfiguration producerConf;
+ producerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
+ client.createProducer("topic-avro", producerConf, producer);
+
+ ```
+
+- The following example shows how to create a consumer with an Avro schema.
+
+ ```cpp
+
+ static const std::string exampleSchema =
+ "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
+ "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
+ ConsumerConfiguration consumerConf;
+ Consumer consumer;
+ consumerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
+ client.subscribe("topic-avro", "sub-2", consumerConf, consumer)
+
+ ```
+
+### ProtobufNative schema
+
+The following example shows how to create a producer and a consumer with a ProtobufNative schema.
+
+1. Generate the `User` class using Protobuf3.
+
+ :::note
+
+ You need to use Protobuf3 or later versions.
+
+ :::
+
+
+
+ ```protobuf
+
+ syntax = "proto3";
+
+ message User {
+ string name = 1;
+ int32 age = 2;
+ }
+
+ ```
+
+
+2. Include the `ProtobufNativeSchema.h` in your source code. Ensure the Protobuf dependency has been added to your project.
+
+
+ ```c++
+
+ #include <pulsar/ProtobufNativeSchema.h>
+
+ ```
+
+
+3. Create a producer to send a `User` instance.
+
+
+ ```c++
+
+ ProducerConfiguration producerConf;
+ producerConf.setSchema(createProtobufNativeSchema(User::GetDescriptor()));
+ Producer producer;
+ client.createProducer("topic-protobuf", producerConf, producer);
+ User user;
+ user.set_name("my-name");
+ user.set_age(10);
+ std::string content;
+ user.SerializeToString(&content);
+ producer.send(MessageBuilder().setContent(content).build());
+
+ ```
+
+
+4. Create a consumer to receive a `User` instance.
+
+
+ ```c++
+
+ ConsumerConfiguration consumerConf;
+ consumerConf.setSchema(createProtobufNativeSchema(User::GetDescriptor()));
+ consumerConf.setSubscriptionInitialPosition(InitialPositionEarliest);
+ Consumer consumer;
+ client.subscribe("topic-protobuf", "my-sub", consumerConf, consumer);
+ Message msg;
+ consumer.receive(msg);
+ User user2;
+ user2.ParseFromArray(msg.getData(), msg.getLength());
+
+ ```
+
diff --git a/site2/website-next/versioned_docs/version-2.4.0/client-libraries-websocket.md b/site2/website-next/versioned_docs/version-2.4.0/client-libraries-websocket.md
index bc13b43..c663f97 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/client-libraries-websocket.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/client-libraries-websocket.md
@@ -2,7 +2,6 @@
id: client-libraries-websocket
title: Pulsar WebSocket API
sidebar_label: "WebSocket"
-original_id: client-libraries-websocket
---
Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page.
@@ -190,7 +189,7 @@ Key | Type | Required? | Explanation
`maxRedeliverCount` | int | no | Define a [maxRedeliverCount](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html#deadLetterPolicy-org.apache.pulsar.client.api.DeadLetterPolicy-) for the consumer (default: 0). Activates [Dead Letter Topic](https://github.com/apache/pulsar/wiki/PIP-22%3A-Pulsar-Dead-Letter-Topic) feature.
`deadLetterTopic` | string | no | Define a [deadLetterTopic](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html#deadLetterPolicy-org.apache.pulsar.client.api.DeadLetterPolicy-) for the consumer (default: {topic}-{subscription}-DLQ). Activates [Dead Letter Topic](https://github.com/apache/pulsar/wiki/PIP-22%3A-Pulsar-Dead-Letter-Topic) feature.
`pullMode` | boolean | no | Enable pull mode (default: false). See "Flow Control" below.
-`negativeAckRedeliveryDelay` | int | no | When a message is negatively acknowledged, it will be redelivered to the DLQ.
+`negativeAckRedeliveryDelay` | int | no | When a message is negatively acknowledged, the delay time before the message is redelivered (in milliseconds). The default value is 60000.
`token` | string | no | Authentication token, this is used for the browser javascript client
NB: these parameter (except `pullMode`) apply to the internal consumer of the WebSocket service.
@@ -204,23 +203,60 @@ Server will push messages on the WebSocket session:
```json
{
- "messageId": "CAAQAw==",
- "payload": "SGVsbG8gV29ybGQ=",
- "properties": {"key1": "value1", "key2": "value2"},
- "publishTime": "2016-08-30 16:45:57.785",
- "redeliveryCount": 4
+ "messageId": "CAMQADAA",
+ "payload": "hvXcJvHW7kOSrUn17P2q71RA5SdiXwZBqw==",
+ "properties": {},
+ "publishTime": "2021-10-29T16:01:38.967-07:00",
+ "redeliveryCount": 0,
+ "encryptionContext": {
+ "keys": {
+ "client-rsa.pem": {
+ "keyValue": "jEuwS+PeUzmCo7IfLNxqoj4h7txbLjCQjkwpaw5AWJfZ2xoIdMkOuWDkOsqgFmWwxiecakS6GOZHs94x3sxzKHQx9Oe1jpwBg2e7L4fd26pp+WmAiLm/ArZJo6JotTeFSvKO3u/yQtGTZojDDQxiqFOQ1ZbMdtMZA8DpSMuq+Zx7PqLo43UdW1+krjQfE5WD+y+qE3LJQfwyVDnXxoRtqWLpVsAROlN2LxaMbaftv5HckoejJoB4xpf/dPOUqhnRstwQHf6klKT5iNhjsY4usACt78uILT0pEPd14h8wEBidBz/vAlC/zVMEqiDVzgNS7dqEYS4iHbf7cnWVCn3Hxw==",
+ "metadata": {}
+ }
+ },
+ "param": "Tfu1PxVm6S9D3+Hk",
+ "compressionType": "NONE",
+ "uncompressedMessageSize": 0,
+ "batchSize": {
+ "empty": false,
+ "present": true
+ }
+ }
}
```
-Key | Type | Required? | Explanation
-:---|:-----|:----------|:-----------
-`messageId` | string | yes | Message ID
-`payload` | string | yes | Base-64 encoded payload
-`publishTime` | string | yes | Publish timestamp
-`redeliveryCount` | number | yes | Number of times this message was already delivered
-`properties` | key-value pairs | no | Application-defined properties
-`key` | string | no | Original routing key set by producer
+Below are the parameters in the WebSocket consumer response.
+
+- General parameters
+
+ Key | Type | Required? | Explanation
+ :---|:-----|:----------|:-----------
+ `messageId` | string | yes | Message ID
+ `payload` | string | yes | Base-64 encoded payload
+ `publishTime` | string | yes | Publish timestamp
+ `redeliveryCount` | number | yes | Number of times this message was already delivered
+ `properties` | key-value pairs | no | Application-defined properties
+ `key` | string | no | Original routing key set by producer
+ `encryptionContext` | EncryptionContext | no | Encryption context that consumers can use to decrypt received messages
+ `param` | string | no | Initialization vector for cipher (Base64 encoding)
+ `batchSize` | string | no | Number of entries in a message (if it is a batch message)
+ `uncompressedMessageSize` | string | no | Message size before compression
+ `compressionType` | string | no | Algorithm used to compress the message payload
+
+- `encryptionContext` related parameter
+
+ Key | Type | Required? | Explanation
+ :---|:-----|:----------|:-----------
+ `keys` |key-EncryptionKey pairs | yes | Key in `key-EncryptionKey` pairs is an encryption key name. Value in `key-EncryptionKey` pairs is an encryption key object.
+
+- `encryptionKey` related parameters
+
+ Key | Type | Required? | Explanation
+ :---|:-----|:----------|:-----------
+ `keyValue` | string | yes | Encryption key (Base64 encoding)
+ `metadata` | key-value pairs | no | Application-defined metadata
#### Acknowledging the message
@@ -454,9 +490,15 @@ TOPIC = scheme + '://localhost:8080/ws/v2/producer/persistent/public/default/my-
ws = websocket.create_connection(TOPIC)
+# encode message
+s = "Hello World"
+firstEncoded = s.encode("UTF-8")
+binaryEncoded = base64.b64encode(firstEncoded)
+payloadString = binaryEncoded.decode('UTF-8')
+
# Send one message as JSON
ws.send(json.dumps({
- 'payload' : base64.b64encode('Hello World'),
+ 'payload' : payloadString,
'properties': {
'key1' : 'value1',
'key2' : 'value2'
@@ -466,9 +508,9 @@ ws.send(json.dumps({
response = json.loads(ws.recv())
if response['result'] == 'ok':
- print 'Message published successfully'
+ print( 'Message published successfully')
else:
- print 'Failed to publish message:', response
+ print('Failed to publish message:', response)
ws.close()
```
@@ -495,7 +537,7 @@ while True:
msg = json.loads(ws.recv())
if not msg: break
- print "Received: {} - payload: {}".format(msg, base64.b64decode(msg['payload']))
+ print( "Received: {} - payload: {}".format(msg, base64.b64decode(msg['payload'])))
# Acknowledge successful processing
ws.send(json.dumps({'messageId' : msg['messageId']}))
@@ -525,7 +567,7 @@ while True:
msg = json.loads(ws.recv())
if not msg: break
- print "Received: {} - payload: {}".format(msg, base64.b64decode(msg['payload']))
+ print ( "Received: {} - payload: {}".format(msg, base64.b64decode(msg['payload'])))
# Acknowledge successful processing
ws.send(json.dumps({'messageId' : msg['messageId']}))
diff --git a/site2/website-next/versioned_docs/version-2.4.0/client-libraries.md b/site2/website-next/versioned_docs/version-2.4.0/client-libraries.md
index 23e5a06..ab5b7c4 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/client-libraries.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/client-libraries.md
@@ -2,7 +2,6 @@
id: client-libraries
title: Pulsar client libraries
sidebar_label: "Overview"
-original_id: client-libraries
---
Pulsar supports the following client libraries:
@@ -16,7 +15,7 @@ Pulsar supports the following client libraries:
- [C# client](client-libraries-dotnet)
## Feature matrix
-Pulsar client feature matrix for different languages is listed on [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page.
+Pulsar client feature matrix for different languages is listed on [Pulsar Feature Matrix (Client and Function)](https://github.com/apache/pulsar/wiki/PIP-108%3A-Pulsar-Feature-Matrix-%28Client-and-Function%29) page.
## Third-party clients
@@ -33,3 +32,4 @@ Besides the official released clients, multiple projects on developing Pulsar cl
| Scala | [pulsar4s](https://github.com/sksamuel/pulsar4s) | [sksamuel](https://github.com/sksamuel) | [](https://opensource.org/licenses/Apache-2.0) | Idomatic, typesafe, and reactive Scala client for Apache Pulsar |
| Rust | [pulsar-rs](https://github.com/wyyerd/pulsar-rs) | [Wyyerd Group](https://github.com/wyyerd) | [](https://opensource.org/licenses/Apache-2.0) | Future-based Rust bindings for Apache Pulsar |
| .NET | [pulsar-client-dotnet](https://github.com/fsharplang-ru/pulsar-client-dotnet) | [Lanayx](https://github.com/Lanayx) | [](https://opensource.org/licenses/MIT) | Native .NET client for C#/F#/VB |
+| Node.js | [pulsar-flex](https://github.com/ayeo-flex-org/pulsar-flex) | [Daniel Sinai](https://github.com/danielsinai), [Ron Farkash](https://github.com/ronfarkash), [Gal Rosenberg](https://github.com/galrose)| [](https://opensource.org/licenses/MIT) | Native Nodejs client |
diff --git a/site2/website-next/versioned_docs/version-2.4.0/concepts-architecture-overview.md b/site2/website-next/versioned_docs/version-2.4.0/concepts-architecture-overview.md
index 6a501d2..8fe0717 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/concepts-architecture-overview.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/concepts-architecture-overview.md
@@ -2,7 +2,6 @@
id: concepts-architecture-overview
title: Architecture Overview
sidebar_label: "Architecture"
-original_id: concepts-architecture-overview
---
At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves.
@@ -146,7 +145,7 @@ Some important things to know about the Pulsar proxy:
## Service discovery
-[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide.
+[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL.
You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means.
diff --git a/site2/website-next/versioned_docs/version-2.4.0/concepts-authentication.md b/site2/website-next/versioned_docs/version-2.4.0/concepts-authentication.md
index b375ecb..335da8d 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/concepts-authentication.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/concepts-authentication.md
@@ -2,7 +2,6 @@
id: concepts-authentication
title: Authentication and Authorization
sidebar_label: "Authentication and Authorization"
-original_id: concepts-authentication
---
Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants.
diff --git a/site2/website-next/versioned_docs/version-2.4.0/concepts-clients.md b/site2/website-next/versioned_docs/version-2.4.0/concepts-clients.md
index b68f76a..65201b5 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/concepts-clients.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/concepts-clients.md
@@ -2,7 +2,6 @@
id: concepts-clients
title: Pulsar Clients
sidebar_label: "Clients"
-original_id: concepts-clients
---
Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications.
diff --git a/site2/website-next/versioned_docs/version-2.4.0/concepts-multi-tenancy.md b/site2/website-next/versioned_docs/version-2.4.0/concepts-multi-tenancy.md
index be752cc..8a17e72 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/concepts-multi-tenancy.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/concepts-multi-tenancy.md
@@ -2,7 +2,6 @@
id: concepts-multi-tenancy
title: Multi Tenancy
sidebar_label: "Multi Tenancy"
-original_id: concepts-multi-tenancy
---
Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed.
diff --git a/site2/website-next/versioned_docs/version-2.4.0/concepts-overview.md b/site2/website-next/versioned_docs/version-2.4.0/concepts-overview.md
index b903fa4..c76032c 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/concepts-overview.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/concepts-overview.md
@@ -2,7 +2,6 @@
id: concepts-overview
title: Pulsar Overview
sidebar_label: "Overview"
-original_id: concepts-overview
---
Pulsar is a multi-tenant, high-performance solution for server-to-server messaging. Pulsar was originally developed by Yahoo, it is under the stewardship of the [Apache Software Foundation](https://www.apache.org/).
diff --git a/site2/website-next/versioned_docs/version-2.4.0/concepts-replication.md b/site2/website-next/versioned_docs/version-2.4.0/concepts-replication.md
index 6e23962..11677cc 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/concepts-replication.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/concepts-replication.md
@@ -2,7 +2,6 @@
id: concepts-replication
title: Geo Replication
sidebar_label: "Geo Replication"
-original_id: concepts-replication
---
Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo) in Pulsar enables you to do that.
diff --git a/site2/website-next/versioned_docs/version-2.4.0/concepts-topic-compaction.md b/site2/website-next/versioned_docs/version-2.4.0/concepts-topic-compaction.md
index c85e703..3356298 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/concepts-topic-compaction.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/concepts-topic-compaction.md
@@ -2,7 +2,6 @@
id: concepts-topic-compaction
title: Topic Compaction
sidebar_label: "Topic Compaction"
-original_id: concepts-topic-compaction
---
Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time int [...]
diff --git a/site2/website-next/versioned_docs/version-2.4.0/cookbooks-bookkeepermetadata.md b/site2/website-next/versioned_docs/version-2.4.0/cookbooks-bookkeepermetadata.md
index b0fa98d..187cb65 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/cookbooks-bookkeepermetadata.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/cookbooks-bookkeepermetadata.md
@@ -1,7 +1,6 @@
---
id: cookbooks-bookkeepermetadata
title: BookKeeper Ledger Metadata
-original_id: cookbooks-bookkeepermetadata
---
Pulsar stores data on BookKeeper ledgers, you can understand the contents of a ledger by inspecting the metadata attached to the ledger.
diff --git a/site2/website-next/versioned_docs/version-2.4.0/cookbooks-deduplication.md b/site2/website-next/versioned_docs/version-2.4.0/cookbooks-deduplication.md
index 1669afa..307fe03 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/cookbooks-deduplication.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/cookbooks-deduplication.md
@@ -1,8 +1,7 @@
---
id: cookbooks-deduplication
title: Message deduplication
-sidebar_label: "Message deduplication"
-original_id: cookbooks-deduplication
+sidebar_label: "Message deduplication "
---
import Tabs from '@theme/Tabs';
diff --git a/site2/website-next/versioned_docs/version-2.4.0/cookbooks-encryption.md b/site2/website-next/versioned_docs/version-2.4.0/cookbooks-encryption.md
index f0d8fb8..fbd1c97 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/cookbooks-encryption.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/cookbooks-encryption.md
@@ -1,8 +1,7 @@
---
id: cookbooks-encryption
title: Pulsar Encryption
-sidebar_label: "Encryption"
-original_id: cookbooks-encryption
+sidebar_label: "Encryption "
---
Pulsar encryption allows applications to encrypt messages at the producer and decrypt at the consumer. Encryption is performed using the public/private key pair configured by the application. Encrypted messages can only be decrypted by consumers with a valid key.
diff --git a/site2/website-next/versioned_docs/version-2.4.0/cookbooks-message-queue.md b/site2/website-next/versioned_docs/version-2.4.0/cookbooks-message-queue.md
index eb43cbd..9b93a94 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/cookbooks-message-queue.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/cookbooks-message-queue.md
@@ -2,7 +2,6 @@
id: cookbooks-message-queue
title: Using Pulsar as a message queue
sidebar_label: "Message queue"
-original_id: cookbooks-message-queue
---
Message queues are essential components of many large-scale data architectures. If every single work object that passes through your system absolutely *must* be processed in spite of the slowness or downright failure of this or that system component, there's a good chance that you'll need a message queue to step in and ensure that unprocessed data is retained---with correct ordering---until the required actions are taken.
diff --git a/site2/website-next/versioned_docs/version-2.4.0/cookbooks-non-persistent.md b/site2/website-next/versioned_docs/version-2.4.0/cookbooks-non-persistent.md
index 391569a..d40c4fb 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/cookbooks-non-persistent.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/cookbooks-non-persistent.md
@@ -2,7 +2,6 @@
id: cookbooks-non-persistent
title: Non-persistent messaging
sidebar_label: "Non-persistent messaging"
-original_id: cookbooks-non-persistent
---
**Non-persistent topics** are Pulsar topics in which message data is *never* [persistently stored](concepts-architecture-overview.md#persistent-storage) and kept only in memory. This cookbook provides:
diff --git a/site2/website-next/versioned_docs/version-2.4.0/cookbooks-partitioned.md b/site2/website-next/versioned_docs/version-2.4.0/cookbooks-partitioned.md
index 7882fb9..2589693 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/cookbooks-partitioned.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/cookbooks-partitioned.md
@@ -2,6 +2,5 @@
id: cookbooks-partitioned
title: Partitioned topics
sidebar_label: "Partitioned Topics"
-original_id: cookbooks-partitioned
---
For details of the content, refer to [manage topics](admin-api-topics).
\ No newline at end of file
diff --git a/site2/website-next/versioned_docs/version-2.4.0/cookbooks-retention-expiry.md b/site2/website-next/versioned_docs/version-2.4.0/cookbooks-retention-expiry.md
index b9353b5..738cf42 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/cookbooks-retention-expiry.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/cookbooks-retention-expiry.md
@@ -2,7 +2,6 @@
id: cookbooks-retention-expiry
title: Message retention and expiry
sidebar_label: "Message retention and expiry"
-original_id: cookbooks-retention-expiry
---
import Tabs from '@theme/Tabs';
@@ -36,7 +35,7 @@ By default, when a Pulsar message arrives at a broker, the message is stored unt
Retention policies are useful when you use the Reader interface. The Reader interface does not use acknowledgements, and messages do not exist within backlogs. It is required to configure retention for Reader-only use cases.
-When you set a retention policy on topics in a namespace, you must set **both** a *size limit* and a *time limit*. You can refer to the following table to set retention policies in `pulsar-admin` and Java.
+When you set a retention policy on topics in a namespace, you must set **both** a *size limit* (via `defaultRetentionSizeInMB`) and a *time limit* (via `defaultRetentionTimeInMinutes`) . You can refer to the following table to set retention policies in `pulsar-admin` and Java.
|Time limit|Size limit| Message retention |
|----------|----------|------------------------|
@@ -152,7 +151,10 @@ admin.namespaces().setRetention(namespace, policies);
You can fetch the retention policy for a namespace by specifying the namespace. The output will be a JSON object with two keys: `retentionTimeInMinutes` and `retentionSizeInMB`.
-#### pulsar-admin
+<Tabs
+ defaultValue="pulsar-admin"
+ values={[{"label":"pulsar-admin","value":"pulsar-admin"},{"label":"REST API","value":"REST API"},{"label":"Java","value":"Java"}]}>
+<TabItem value="pulsar-admin">
Use the [`get-retention`](reference-pulsar-admin.md#namespaces) subcommand and specify the namespace.
@@ -168,11 +170,13 @@ $ pulsar-admin namespaces get-retention my-tenant/my-ns
```
-#### REST API
+</TabItem>
+<TabItem value="REST API">
{@inject: endpoint|GET|/admin/v2/namespaces/:tenant/:namespace/retention|operation/getRetention?version=@pulsar:version_number@}
-#### Java
+</TabItem>
+<TabItem value="Java">
```java
@@ -180,15 +184,17 @@ admin.namespaces().getRetention(namespace);
```
+</TabItem>
+
+</Tabs>
+
## Backlog quotas
*Backlogs* are sets of unacknowledged messages for a topic that have been stored by bookies. Pulsar stores all unacknowledged messages in backlogs until they are processed and acknowledged.
-You can control the allowable size of backlogs, at the namespace level, using *backlog quotas*. Setting a backlog quota involves setting:
+You can control the allowable size and/or time of backlogs, at the namespace level, using *backlog quotas*. Setting a backlog quota involves setting:
-TODO: Expand on is this per backlog or per topic?
-
-* an allowable *size threshold* for each topic in the namespace
+* an allowable *size and/or time threshold* for each topic in the namespace
* a *retention policy* that determines which action the [broker](reference-terminology.md#broker) takes if the threshold is exceeded.
The following retention policies are available:
@@ -210,9 +216,12 @@ Backlog quotas are handled at the namespace level. They can be managed via:
You can set a size and/or time threshold and backlog retention policy for all of the topics in a [namespace](reference-terminology.md#namespace) by specifying the namespace, a size limit and/or a time limit in second, and a policy by name.
-#### pulsar-admin
+<Tabs
+ defaultValue="pulsar-admin"
+ values={[{"label":"pulsar-admin","value":"pulsar-admin"},{"label":"REST API","value":"REST API"},{"label":"Java","value":"Java"}]}>
+<TabItem value="pulsar-admin">
-Use the [`set-backlog-quota`](reference-pulsar-admin.md#namespaces) subcommand and specify a namespace, a size limit using the `-l`/`--limit` flag, and a retention policy using the `-p`/`--policy` flag.
+Use the [`set-backlog-quota`](reference-pulsar-admin.md#namespaces) subcommand and specify a namespace, a size limit using the `-l`/`--limit` , `-lt`/`--limitTime` flag to limit backlog, a retention policy using the `-p`/`--policy` flag and a policy type using `-t`/`--type` (default is destination_storage).
##### Example
@@ -220,16 +229,26 @@ Use the [`set-backlog-quota`](reference-pulsar-admin.md#namespaces) subcommand a
$ pulsar-admin namespaces set-backlog-quota my-tenant/my-ns \
--limit 2G \
- --limitTime 36000 \
--policy producer_request_hold
```
-#### REST API
+```shell
+
+$ pulsar-admin namespaces set-backlog-quota my-tenant/my-ns/my-topic \
+--limitTime 3600 \
+--policy producer_request_hold \
+--type message_age
+
+```
+
+</TabItem>
+<TabItem value="REST API">
{@inject: endpoint|POST|/admin/v2/namespaces/:tenant/:namespace/backlogQuota|operation/getBacklogQuotaMap?version=@pulsar:version_number@}
-#### Java
+</TabItem>
+<TabItem value="Java">
```java
@@ -240,11 +259,18 @@ admin.namespaces().setBacklogQuota(namespace, quota);
```
+</TabItem>
+
+</Tabs>
+
### Get backlog threshold and backlog retention policy
You can see which size threshold and backlog retention policy has been applied to a namespace.
-#### pulsar-admin
+<Tabs
+ defaultValue="pulsar-admin"
+ values={[{"label":"pulsar-admin","value":"pulsar-admin"},{"label":"REST API","value":"REST API"},{"label":"Java","value":"Java"}]}>
+<TabItem value="pulsar-admin">
Use the [`get-backlog-quotas`](reference-pulsar-admin.md#pulsar-admin-namespaces-get-backlog-quotas) subcommand and specify a namespace. Here's an example:
@@ -260,11 +286,13 @@ $ pulsar-admin namespaces get-backlog-quotas my-tenant/my-ns
```
-#### REST API
+</TabItem>
+<TabItem value="REST API">
{@inject: endpoint|GET|/admin/v2/namespaces/:tenant/:namespace/backlogQuotaMap|operation/getBacklogQuotaMap?version=@pulsar:version_number@}
-#### Java
+</TabItem>
+<TabItem value="Java">
```java
@@ -273,11 +301,18 @@ Map<BacklogQuota.BacklogQuotaType,BacklogQuota> quotas =
```
+</TabItem>
+
+</Tabs>
+
### Remove backlog quotas
-#### pulsar-admin
+<Tabs
+ defaultValue="pulsar-admin"
+ values={[{"label":"pulsar-admin","value":"pulsar-admin"},{"label":"REST API","value":"REST API"},{"label":"Java","value":"Java"}]}>
+<TabItem value="pulsar-admin">
-Use the [`remove-backlog-quota`](reference-pulsar-admin.md#pulsar-admin-namespaces-remove-backlog-quota) subcommand and specify a namespace. Here's an example:
+Use the [`remove-backlog-quota`](reference-pulsar-admin.md#pulsar-admin-namespaces-remove-backlog-quota) subcommand and specify a namespace, use `t`/`--type` to specify backlog type to remove(default is destination_storage). Here's an example:
```shell
@@ -285,11 +320,13 @@ $ pulsar-admin namespaces remove-backlog-quota my-tenant/my-ns
```
-#### REST API
+</TabItem>
+<TabItem value="REST API">
{@inject: endpoint|DELETE|/admin/v2/namespaces/:tenant/:namespace/backlogQuota|operation/removeBacklogQuota?version=@pulsar:version_number@}
-#### Java
+</TabItem>
+<TabItem value="Java">
```java
@@ -297,6 +334,10 @@ admin.namespaces().removeBacklogQuota(namespace);
```
+</TabItem>
+
+</Tabs>
+
### Clear backlog
#### pulsar-admin
@@ -319,7 +360,10 @@ By default, Pulsar stores all unacknowledged messages forever. This can lead to
### Set the TTL for a namespace
-#### pulsar-admin
+<Tabs
+ defaultValue="pulsar-admin"
+ values={[{"label":"pulsar-admin","value":"pulsar-admin"},{"label":"REST API","value":"REST API"},{"label":"Java","value":"Java"}]}>
+<TabItem value="pulsar-admin">
Use the [`set-message-ttl`](reference-pulsar-admin.md#pulsar-admin-namespaces-set-message-ttl) subcommand and specify a namespace and a TTL (in seconds) using the `-ttl`/`--messageTTL` flag.
@@ -332,11 +376,13 @@ $ pulsar-admin namespaces set-message-ttl my-tenant/my-ns \
```
-#### REST API
+</TabItem>
+<TabItem value="REST API">
{@inject: endpoint|POST|/admin/v2/namespaces/:tenant/:namespace/messageTTL|operation/setNamespaceMessageTTL?version=@pulsar:version_number@}
-#### Java
+</TabItem>
+<TabItem value="Java">
```java
@@ -344,9 +390,16 @@ admin.namespaces().setNamespaceMessageTTL(namespace, ttlInSeconds);
```
+</TabItem>
+
+</Tabs>
+
### Get the TTL configuration for a namespace
-#### pulsar-admin
+<Tabs
+ defaultValue="pulsar-admin"
+ values={[{"label":"pulsar-admin","value":"pulsar-admin"},{"label":"REST API","value":"REST API"},{"label":"Java","value":"Java"}]}>
+<TabItem value="pulsar-admin">
Use the [`get-message-ttl`](reference-pulsar-admin.md#pulsar-admin-namespaces-get-message-ttl) subcommand and specify a namespace.
@@ -359,11 +412,13 @@ $ pulsar-admin namespaces get-message-ttl my-tenant/my-ns
```
-#### REST API
+</TabItem>
+<TabItem value="REST API">
{@inject: endpoint|GET|/admin/v2/namespaces/:tenant/:namespace/messageTTL|operation/getNamespaceMessageTTL?version=@pulsar:version_number@}
-#### Java
+</TabItem>
+<TabItem value="Java">
```java
@@ -371,9 +426,16 @@ admin.namespaces().getNamespaceMessageTTL(namespace)
```
+</TabItem>
+
+</Tabs>
+
### Remove the TTL configuration for a namespace
-#### pulsar-admin
+<Tabs
+ defaultValue="pulsar-admin"
+ values={[{"label":"pulsar-admin","value":"pulsar-admin"},{"label":"REST API","value":"REST API"},{"label":"Java","value":"Java"}]}>
+<TabItem value="pulsar-admin">
Use the [`remove-message-ttl`](reference-pulsar-admin.md#pulsar-admin-namespaces-remove-message-ttl) subcommand and specify a namespace.
@@ -385,11 +447,13 @@ $ pulsar-admin namespaces remove-message-ttl my-tenant/my-ns
```
-#### REST API
+</TabItem>
+<TabItem value="REST API">
{@inject: endpoint|DELETE|/admin/v2/namespaces/:tenant/:namespace/messageTTL|operation/removeNamespaceMessageTTL?version=@pulsar:version_number@}
-#### Java
+</TabItem>
+<TabItem value="Java">
```java
@@ -397,6 +461,10 @@ admin.namespaces().removeNamespaceMessageTTL(namespace)
```
+</TabItem>
+
+</Tabs>
+
## Delete messages from namespaces
If you do not have any retention period and that you never have much of a backlog, the upper limit for retaining messages, which are acknowledged, equals to the Pulsar segment rollover period + entry log rollover period + (garbage collection interval * garbage collection ratios).
diff --git a/site2/website-next/versioned_docs/version-2.4.0/deploy-aws.md b/site2/website-next/versioned_docs/version-2.4.0/deploy-aws.md
index 6323051..2034749 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/deploy-aws.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/deploy-aws.md
@@ -2,7 +2,6 @@
id: deploy-aws
title: Deploying a Pulsar cluster on AWS using Terraform and Ansible
sidebar_label: "Amazon Web Services"
-original_id: deploy-aws
---
> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster).
@@ -148,7 +147,7 @@ Variable name | Description | Default
When you run the Ansible playbook, the following AWS resources are used:
* 9 total [Elastic Compute Cloud](https://aws.amazon.com/ec2) (EC2) instances running the [ami-9fa343e7](https://access.redhat.com/articles/3135091) Amazon Machine Image (AMI), which runs [Red Hat Enterprise Linux (RHEL) 7.4](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/7.4_release_notes/index). By default, that includes:
- * 3 small VMs for ZooKeeper ([t2.small](https://www.ec2instances.info/?selected=t2.small) instances)
+ * 3 small VMs for ZooKeeper ([t3.small](https://www.ec2instances.info/?selected=t3.small) instances)
* 3 larger VMs for BookKeeper [bookies](reference-terminology.md#bookie) ([i3.xlarge](https://www.ec2instances.info/?selected=i3.xlarge) instances)
* 2 larger VMs for Pulsar [brokers](reference-terminology.md#broker) ([c5.2xlarge](https://www.ec2instances.info/?selected=c5.2xlarge) instances)
* 1 larger VMs for Pulsar [proxy](reference-terminology.md#proxy) ([c5.2xlarge](https://www.ec2instances.info/?selected=c5.2xlarge) instances)
diff --git a/site2/website-next/versioned_docs/version-2.4.0/deploy-bare-metal-multi-cluster.md b/site2/website-next/versioned_docs/version-2.4.0/deploy-bare-metal-multi-cluster.md
index 643c122..9dd2526 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/deploy-bare-metal-multi-cluster.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/deploy-bare-metal-multi-cluster.md
@@ -2,38 +2,30 @@
id: deploy-bare-metal-multi-cluster
title: Deploying a multi-cluster on bare metal
sidebar_label: "Bare metal multi-cluster"
-original_id: deploy-bare-metal-multi-cluster
---
:::tip
-1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with
-Pulsar or using it in a startup or on a single team, you had better opt for a single cluster. For instructions on deploying a single cluster,
-see the guide [here](deploy-bare-metal).
-2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors`
-package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you
-run a separate cluster of function workers for [Pulsar Functions](functions-overview).
-3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`
-package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure
-this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage).
+1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster).
+2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview).
+3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage).
:::
-A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo). Deploying a multi-cluster Pulsar instance involves the following basic steps:
+A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo).Deploying a multi-cluster Pulsar instance consists of the following steps:
-* Deploying two separate [ZooKeeper](#deploy-zookeeper) quorums: a [local](#deploy-local-zookeeper) quorum for each cluster in the instance and a [configuration store](#configuration-store) quorum for instance-wide tasks
-* Initializing [cluster metadata](#cluster-metadata-initialization) for each cluster
-* Deploying a [BookKeeper cluster](#deploy-bookkeeper) of bookies in each Pulsar cluster
-* Deploying [brokers](#deploy-brokers) in each Pulsar cluster
+1. Deploying two separate ZooKeeper quorums: a local quorum for each cluster in the instance and a configuration store quorum for instance-wide tasks
+2. Initializing cluster metadata for each cluster
+3. Deploying a BookKeeper cluster of bookies in each Pulsar cluster
+4. Deploying brokers in each Pulsar cluster
-If you want to deploy a single Pulsar cluster, see [Clusters and Brokers](getting-started-standalone.md#start-the-cluster).
> #### Run Pulsar locally or on Kubernetes?
-> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar [...]
+> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services.
## System requirement
-Currently, Pulsar is available for 64-bit **macOS**, **Linux**, and **Windows**. To use Pulsar, you need to install 64-bit JRE/JDK 8 or later versions.
+Currently, Pulsar is available for 64-bit **macOS**, **Linux**, and **Windows**. You need to install 64-bit JRE/JDK 8 or later versions.
:::note
@@ -68,8 +60,6 @@ $ cd apache-pulsar-@pulsar:version@
```
-## What your package contains
-
The Pulsar binary package initially contains the following directories:
Directory | Contains
@@ -93,17 +83,17 @@ Directory | Contains
Each Pulsar instance relies on two separate ZooKeeper quorums.
-* [Local ZooKeeper](#deploy-local-zookeeper) operates at the cluster level and provides cluster-specific configuration management and coordination. Each Pulsar cluster needs to have a dedicated ZooKeeper cluster.
-* [Configuration Store](#deploy-the-configuration-store) operates at the instance level and provides configuration management for the entire system (and thus across clusters). An independent cluster of machines or the same machines that local ZooKeeper uses can provide the configuration store quorum.
+* Local ZooKeeper operates at the cluster level and provides cluster-specific configuration management and coordination. Each Pulsar cluster needs a dedicated ZooKeeper cluster.
+* Configuration Store operates at the instance level and provides configuration management for the entire system (and thus across clusters). An independent cluster of machines or the same machines that local ZooKeeper uses can provide the configuration store quorum.
-The configuration store quorum can be provided by an independent cluster of machines or by the same machines used by local ZooKeeper.
+You can use an independent cluster of machines or the same machines used by local ZooKeeper to provide the configuration store quorum.
### Deploy local ZooKeeper
ZooKeeper manages a variety of essential coordination-related and configuration-related tasks for Pulsar.
-You need to stand up one local ZooKeeper cluster *per Pulsar cluster* for deploying a Pulsar instance.
+You need to stand up one local ZooKeeper cluster per Pulsar cluster for deploying a Pulsar instance.
To begin, add all ZooKeeper servers to the quorum configuration specified in the [`conf/zookeeper.conf`](reference-configuration.md#zookeeper) file. Add a `server.N` line for each node in the cluster to the configuration, where `N` is the number of the ZooKeeper node. The following is an example for a three-node cluster:
@@ -117,7 +107,11 @@ server.3=zk3.us-west.example.com:2888:3888
On each host, you need to specify the ID of the node in the `myid` file of each node, which is in `data/zookeeper` folder of each server by default (you can change the file location via the [`dataDir`](reference-configuration.md#zookeeper-dataDir) parameter).
-> See the [Multi-server setup guide](https://zookeeper.apache.org/doc/r3.4.10/zookeeperAdmin.html#sc_zkMulitServerSetup) in the ZooKeeper documentation for detailed information on `myid` and more.
+:::tip
+
+See the [Multi-server setup guide](https://zookeeper.apache.org/doc/r3.4.10/zookeeperAdmin.html#sc_zkMulitServerSetup) in the ZooKeeper documentation for detailed information on `myid` and more.
+
+:::
On a ZooKeeper server at `zk1.us-west.example.com`, for example, you could set the `myid` value like this:
@@ -140,15 +134,15 @@ $ bin/pulsar-daemon start zookeeper
### Deploy the configuration store
-The ZooKeeper cluster that is configured and started up in the section above is a *local* ZooKeeper cluster that you can use to manage a single Pulsar cluster. In addition to a local cluster, however, a full Pulsar instance also requires a configuration store for handling some instance-level configuration and coordination tasks.
+The ZooKeeper cluster configured and started up in the section above is a local ZooKeeper cluster that you can use to manage a single Pulsar cluster. In addition to a local cluster, however, a full Pulsar instance also requires a configuration store for handling some instance-level configuration and coordination tasks.
-If you deploy a [single-cluster](#single-cluster-pulsar-instance) instance, you do not need a separate cluster for the configuration store. If, however, you deploy a [multi-cluster](#multi-cluster-pulsar-instance) instance, you should stand up a separate ZooKeeper cluster for configuration tasks.
+If you deploy a single-cluster instance, you do not need a separate cluster for the configuration store. If, however, you deploy a multi-cluster instance, you should stand up a separate ZooKeeper cluster for configuration tasks.
#### Single-cluster Pulsar instance
If your Pulsar instance consists of just one cluster, then you can deploy a configuration store on the same machines as the local ZooKeeper quorum but run on different TCP ports.
-To deploy a ZooKeeper configuration store in a single-cluster instance, add the same ZooKeeper servers that the local quorum uses to the configuration file in [`conf/global_zookeeper.conf`](reference-configuration.md#configuration-store) using the same method for [local ZooKeeper](#local-zookeeper), but make sure to use a different port (2181 is the default for ZooKeeper). The following is an example that uses port 2184 for a three-node ZooKeeper cluster:
+To deploy a ZooKeeper configuration store in a single-cluster instance, add the same ZooKeeper servers that the local quorum. You need to use the configuration file in [`conf/global_zookeeper.conf`](reference-configuration.md#configuration-store) using the same method for [local ZooKeeper](#local-zookeeper), but make sure to use a different port (2181 is the default for ZooKeeper). The following is an example that uses port 2184 for a three-node ZooKeeper cluster:
```properties
@@ -165,13 +159,11 @@ As before, create the `myid` files for each server on `data/global-zookeeper/myi
When you deploy a global Pulsar instance, with clusters distributed across different geographical regions, the configuration store serves as a highly available and strongly consistent metadata store that can tolerate failures and partitions spanning whole regions.
-The key here is to make sure the ZK quorum members are spread across at least 3 regions and that other regions run as observers.
+The key here is to make sure the ZK quorum members are spread across at least 3 regions, and other regions run as observers.
-Again, given the very low expected load on the configuration store servers, you can
-share the same hosts used for the local ZooKeeper quorum.
+Again, given the very low expected load on the configuration store servers, you can share the same hosts used for the local ZooKeeper quorum.
-For example, assume a Pulsar instance with the following clusters `us-west`,
-`us-east`, `us-central`, `eu-central`, `ap-south`. Also assume, each cluster has its own local ZK servers named such as the following:
+For example, assume a Pulsar instance with the following clusters `us-west`, `us-east`, `us-central`, `eu-central`, `ap-south`. Also assume, each cluster has its own local ZK servers named such as the following:
```
@@ -179,8 +171,7 @@ zk[1-3].${CLUSTER}.example.com
```
-In this scenario if you want to pick the quorum participants from few clusters and
-let all the others be ZK observers. For example, to form a 7 servers quorum, you can pick 3 servers from `us-west`, 2 from `us-central` and 2 from `us-east`.
+In this scenario if you want to pick the quorum participants from few clusters and let all the others be ZK observers. For example, to form a 7 servers quorum, you can pick 3 servers from `us-west`, 2 from `us-central` and 2 from `us-east`.
This method guarantees that writes to configuration store is possible even if one of these regions is unreachable.
@@ -227,7 +218,7 @@ $ bin/pulsar-daemon start configuration-store
## Cluster metadata initialization
-Once you set up the cluster-specific ZooKeeper and configuration store quorums for your instance, you need to write some metadata to ZooKeeper for each cluster in your instance. **you only needs to write these metadata once**.
+Once you set up the cluster-specific ZooKeeper and configuration store quorums for your instance, you need to write some metadata to ZooKeeper for each cluster in your instance. **you only need to write these metadata once**.
You can initialize this metadata using the [`initialize-cluster-metadata`](reference-cli-tools.md#pulsar-initialize-cluster-metadata) command of the [`pulsar`](reference-cli-tools.md#pulsar) CLI tool. The following is an example:
@@ -260,7 +251,7 @@ Make sure to run `initialize-cluster-metadata` for each cluster in your instance
BookKeeper provides [persistent message storage](concepts-architecture-overview.md#persistent-storage) for Pulsar.
-Each Pulsar broker needs to have its own cluster of bookies. The BookKeeper cluster shares a local ZooKeeper quorum with the Pulsar cluster.
+Each Pulsar broker needs its own cluster of bookies. The BookKeeper cluster shares a local ZooKeeper quorum with the Pulsar cluster.
### Configure bookies
@@ -280,7 +271,7 @@ $ bin/pulsar-daemon start bookie
You can verify that the bookie works properly using the `bookiesanity` command for the [BookKeeper shell](reference-cli-tools.md#bookkeeper-shell):
-```shell
+```bash
$ bin/bookkeeper shell bookiesanity
@@ -304,7 +295,7 @@ Bookie hosts are responsible for storing message data on disk. In order for book
Message entries written to bookies are always synced to disk before returning an acknowledgement to the Pulsar broker. To ensure low write latency, BookKeeper is
designed to use multiple devices:
-* A **journal** to ensure durability. For sequential writes, having fast [fsync](https://linux.die.net/man/2/fsync) operations on bookie hosts is critical. Typically, small and fast [solid-state drives](https://en.wikipedia.org/wiki/Solid-state_drive) (SSDs) should suffice, or [hard disk drives](https://en.wikipedia.org/wiki/Hard_disk_drive) (HDDs) with a [RAID](https://en.wikipedia.org/wiki/RAID)s controller and a battery-backed write cache. Both solutions can reach fsync latency of ~0.4 ms.
+* A **journal** to ensure durability. For sequential writes, having fast [fsync](https://linux.die.net/man/2/fsync) operations on bookie hosts is critical. Typically, small and fast [solid-state drives](https://en.wikipedia.org/wiki/Solid-state_drive) (SSDs) should suffice, or [hard disk drives](https://en.wikipedia.org/wiki/Hard_disk_drive) (HDDs) with a [RAID](https://en.wikipedia.org/wiki/RAID) controller and a battery-backed write cache. Both solutions can reach fsync latency of ~0.4 ms.
* A **ledger storage device** is where data is stored until all consumers acknowledge the message. Writes happen in the background, so write I/O is not a big concern. Reads happen sequentially most of the time and the backlog is drained only in case of consumer drain. To store large amounts of data, a typical configuration involves multiple HDDs with a RAID controller.
@@ -371,39 +362,13 @@ $ bin/pulsar broker
## Service discovery
-[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup).
+[Clients](getting-started-clients) connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL.
-You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means.
+You can use your own service discovery system. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means.
-> #### Service discovery already provided by many scheduling systems
+> **Service discovery already provided by many scheduling systems**
> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism.
-
-### Service discovery setup
-
-The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol) of Pulsar.
-
-To get started setting up the built-in service of discovery of Pulsar, you need to change a few parameters in the [`conf/discovery.conf`](reference-configuration.md#service-discovery) configuration file. Set the [`zookeeperServers`](reference-configuration.md#service-discovery-zookeeperServers) parameter to the ZooKeeper quorum connection string of the cluster and the [`configurationStoreServers`](reference-configuration.md#service-discovery-configurationStoreServers) setting to the [con [...]
-store](reference-terminology.md#configuration-store) quorum connection string.
-
-```properties
-
-# Zookeeper quorum connection string
-zookeeperServers=zk1.us-west.example.com:2181,zk2.us-west.example.com:2181,zk3.us-west.example.com:2181
-
-# Global configuration store connection string
-configurationStoreServers=zk1.us-west.example.com:2184,zk2.us-west.example.com:2184,zk3.us-west.example.com:2184
-
-```
-
-To start the discovery service:
-
-```shell
-
-$ bin/pulsar-daemon start discovery
-
-```
-
## Admin client and verification
At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients.
diff --git a/site2/website-next/versioned_docs/version-2.4.0/deploy-dcos.md b/site2/website-next/versioned_docs/version-2.4.0/deploy-dcos.md
index f5f8d1f..07f446e 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/deploy-dcos.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/deploy-dcos.md
@@ -7,18 +7,17 @@ original_id: deploy-dcos
:::tip
-If you want to enable all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of
-`apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors).
+To enable all built-in [Pulsar IO](io-overview) connectors in your Pulsar deploymente, we recommend you use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image; the former has already bundled [all built-in connectors](io-overview.md#working-with-connectors).
:::
-[DC/OS](https://dcos.io/) (the <strong>D</strong>ata<strong>C</strong>enter <strong>O</strong>perating <strong>S</strong>ystem) is a distributed operating system used for deploying and managing applications and systems on [Apache Mesos](http://mesos.apache.org/). DC/OS is an open-source tool that [Mesosphere](https://mesosphere.com/) creates and maintains .
+[DC/OS](https://dcos.io/) (the <strong>D</strong>ata<strong>C</strong>enter <strong>O</strong>perating <strong>S</strong>ystem) is a distributed operating system for deploying and managing applications and systems on [Apache Mesos](http://mesos.apache.org/). DC/OS is an open-source tool created and maintained by [Mesosphere](https://mesosphere.com/).
Apache Pulsar is available as a [Marathon Application Group](https://mesosphere.github.io/marathon/docs/application-groups.html), which runs multiple applications as manageable sets.
## Prerequisites
-In order to run Pulsar on DC/OS, you need the following:
+You need to prepare your environment before running Pulsar on DC/OS.
* DC/OS version [1.9](https://docs.mesosphere.com/1.9/) or higher
* A [DC/OS cluster](https://docs.mesosphere.com/1.9/installing/) with at least three agent nodes
@@ -37,7 +36,7 @@ Each node in the DC/OS-managed Mesos cluster must have at least:
* 4 GB of memory
* 60 GB of total persistent disk
-Alternatively, you can change the configuration in `PulsarGroups.json` according to match your resources of DC/OS cluster.
+Alternatively, you can change the configuration in `PulsarGroups.json` accordingly to match your resources of the DC/OS cluster.
## Deploy Pulsar using the DC/OS command interface
@@ -56,9 +55,9 @@ This command deploys Docker container instances in three groups, which together
* 1 [Prometheus](http://prometheus.io/) instance and 1 [Grafana](https://grafana.com/) instance
-> When you run DC/OS, a ZooKeeper cluster already runs at `master.mesos:2181`, thus you do not have to install or start up ZooKeeper separately.
+> When you run DC/OS, a ZooKeeper cluster will be running at `master.mesos:2181`, thus you do not have to install or start up ZooKeeper separately.
-After executing the `dcos` command above, click on the **Services** tab in the DC/OS [GUI interface](https://docs.mesosphere.com/latest/gui/), which you can access at [http://m1.dcos](http://m1.dcos) in this example. You should see several applications in the process of deploying.
+After executing the `dcos` command above, click the **Services** tab in the DC/OS [GUI interface](https://docs.mesosphere.com/latest/gui/), which you can access at [http://m1.dcos](http://m1.dcos) in this example. You should see several applications during the deployment.
@@ -66,15 +65,15 @@ After executing the `dcos` command above, click on the **Services** tab in the D
## The BookKeeper group
-To monitor the status of the BookKeeper cluster deployment, click on the **bookkeeper** group in the parent **pulsar** group.
+To monitor the status of the BookKeeper cluster deployment, click the **bookkeeper** group in the parent **pulsar** group.
-At this point, 3 [bookies](reference-terminology.md#bookie) should be shown as green, which means that the bookies have been deployed successfully and are now running.
+At this point, the status of the 3 [bookies](reference-terminology.md#bookie) are green, which means that the bookies have been deployed successfully and are running.
-You can also click into each bookie instance to get more detailed information, such as the bookie running log.
+You can also click each bookie instance to get more detailed information, such as the bookie running log.
@@ -82,23 +81,23 @@ To display information about the BookKeeper in ZooKeeper, you can visit [http://
-## The Pulsar broker Group
+## The Pulsar broker group
-Similar to the BookKeeper group above, click into the **brokers** to check the status of the Pulsar brokers.
+Similar to the BookKeeper group above, click **brokers** to check the status of the Pulsar brokers.
-You can also click into each broker instance to get more detailed information, such as the broker running log.
+You can also click each broker instance to get more detailed information, such as the broker running log.
-Broker cluster information in Zookeeper is also available through the web UI. In this example, you can see that the `loadbalance` and `managed-ledgers` directories have been created.
+Broker cluster information in ZooKeeper is also available through the web UI. In this example, you can see that the `loadbalance` and `managed-ledgers` directories have been created.
-## Monitor Group
+## Monitor group
The **monitory** group consists of Prometheus and Grafana.
@@ -106,17 +105,17 @@ The **monitory** group consists of Prometheus and Grafana.
### Prometheus
-Click into the instance of `prom` to get the endpoint of Prometheus, which is `192.168.65.121:9090` in this example.
+Click the instance of `prom` to get the endpoint of Prometheus, which is `192.168.65.121:9090` in this example.
-If you click that endpoint, you can see the Prometheus dashboard. The [http://192.168.65.121:9090/targets](http://192.168.65.121:9090/targets) URL display all the bookies and brokers.
+If you click that endpoint, you can see the Prometheus dashboard. All the bookies and brokers are listed on [http://192.168.65.121:9090/targets](http://192.168.65.121:9090/targets).
### Grafana
-Click into `grafana` to get the endpoint for Grafana, which is `192.168.65.121:3000` in this example.
+Click `grafana` to get the endpoint for Grafana, which is `192.168.65.121:3000` in this example.
@@ -130,7 +129,7 @@ Now that you have a fully deployed Pulsar cluster, you can run a simple consumer
### Download and prepare the Pulsar Java tutorial
-You can clone a [Pulsar Java tutorial](https://github.com/streamlio/pulsar-java-tutorial) repo. This repo contains a simple Pulsar consumer and producer (you can find more information in the `README` file of the repo).
+You can clone a [Pulsar Java tutorial](https://github.com/streamlio/pulsar-java-tutorial) repo. This repo contains a simple Pulsar consumer and producer (you can find more information in the `README` file in this repo).
```bash
@@ -138,12 +137,13 @@ $ git clone https://github.com/streamlio/pulsar-java-tutorial
```
-Change the `SERVICE_URL` from `pulsar://localhost:6650` to `pulsar://a1.dcos:6650` in both [`ConsumerTutorial.java`](https://github.com/streamlio/pulsar-java-tutorial/blob/master/src/main/java/tutorial/ConsumerTutorial.java) and [`ProducerTutorial.java`](https://github.com/streamlio/pulsar-java-tutorial/blob/master/src/main/java/tutorial/ProducerTutorial.java).
-The `pulsar://a1.dcos:6650` endpoint is for the broker service. You can fetch the endpoint details for each broker instance from the DC/OS GUI. `a1.dcos` is a DC/OS client agent, which runs a broker. The client agent IP address can also replace this.
+Change the `SERVICE_URL` from `pulsar://localhost:6650` to `pulsar://a1.dcos:6650` in both [`ConsumerTutorial.java`](https://github.com/streamlio/pulsar-java-tutorial/blob/master/src/main/java/tutorial/ConsumerTutorial.java) file and [`ProducerTutorial.java`](https://github.com/streamlio/pulsar-java-tutorial/blob/master/src/main/java/tutorial/ProducerTutorial.java) file.
-Now, change the message number from 10 to 10000000 in main method of [`ProducerTutorial.java`](https://github.com/streamlio/pulsar-java-tutorial/blob/master/src/main/java/tutorial/ProducerTutorial.java) so that it can produce more messages.
+The `pulsar://a1.dcos:6650` endpoint is for the broker service. You can fetch the endpoint details for each broker instance from the DC/OS GUI. `a1.dcos` is a DC/OS client agent that runs a broker, and you can replace it with the client agent IP address.
-Now compile the project code using the command below:
+Now, you can change the message number from 10 to 10000000 in the main method in [`ProducerTutorial.java`](https://github.com/streamlio/pulsar-java-tutorial/blob/master/src/main/java/tutorial/ProducerTutorial.java) file to produce more messages.
+
+Then, you can compile the project code using the command below:
```bash
@@ -169,7 +169,7 @@ $ mvn exec:java -Dexec.mainClass="tutorial.ProducerTutorial"
```
-You can see the producer producing messages and the consumer consuming messages through the DC/OS GUI.
+You see that the producer is producing messages and the consumer is consuming messages through the DC/OS GUI.
@@ -177,20 +177,20 @@ You can see the producer producing messages and the consumer consuming messages
### View Grafana metric output
-While the producer and consumer run, you can access running metrics information from Grafana.
+While the producer and consumer are running, you can access the running metrics from Grafana.
## Uninstall Pulsar
-You can shut down and uninstall the `pulsar` application from DC/OS at any time in the following two ways:
+You can shut down and uninstall the `pulsar` application from DC/OS at any time in one of the following two ways:
-1. Using the DC/OS GUI, you can choose **Delete** at the right end of Pulsar group.
+1. Click the three dots at the right end of Pulsar group and choose **Delete** on the DC/OS GUI.
-2. You can use the following command:
+2. Use the command below.
```bash
diff --git a/site2/website-next/versioned_docs/version-2.4.0/deploy-kubernetes.md b/site2/website-next/versioned_docs/version-2.4.0/deploy-kubernetes.md
index dc7123d..4e170dc 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/deploy-kubernetes.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/deploy-kubernetes.md
@@ -2,7 +2,6 @@
id: deploy-kubernetes
title: Deploy Pulsar on Kubernetes
sidebar_label: "Kubernetes"
-original_id: deploy-kubernetes
---
To get up and running with these charts as fast as possible, in a **non-production** use case, we provide
diff --git a/site2/website-next/versioned_docs/version-2.4.0/deploy-monitoring.md b/site2/website-next/versioned_docs/version-2.4.0/deploy-monitoring.md
index 074ce3f..95ccdd6 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/deploy-monitoring.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/deploy-monitoring.md
@@ -2,7 +2,6 @@
id: deploy-monitoring
title: Monitor
sidebar_label: "Monitor"
-original_id: deploy-monitoring
---
You can use different ways to monitor a Pulsar cluster, exposing both metrics related to the usage of topics and the overall health of the individual components of the cluster.
@@ -127,17 +126,7 @@ The per-topic dashboard instructions are available at [Pulsar manager](administr
You can use grafana to create dashboard driven by the data that is stored in Prometheus.
-When you deploy Pulsar on Kubernetes, a `pulsar-grafana` Docker image is enabled by default. You can use the docker image with the principal dashboards.
-
-Enter the command below to use the dashboard manually:
-
-```shell
-
-docker run -p3000:3000 \
- -e PROMETHEUS_URL=http://$PROMETHEUS_HOST:9090/ \
- apachepulsar/pulsar-grafana:latest
-
-```
+When you deploy Pulsar on Kubernetes with the Pulsar Helm Chart, a `pulsar-grafana` Docker image is enabled by default. You can use the docker image with the principal dashboards.
The following are some Grafana dashboards examples:
@@ -145,4 +134,4 @@ The following are some Grafana dashboards examples:
- [apache-pulsar-grafana-dashboard](https://github.com/streamnative/apache-pulsar-grafana-dashboard): a collection of Grafana dashboard templates for different Pulsar components running on both Kubernetes and on-premise machines.
## Alerting rules
-You can set alerting rules according to your Pulsar environment. To configure alerting rules for Apache Pulsar, refer to [alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/).
+You can set alerting rules according to your Pulsar environment. To configure alerting rules for Apache Pulsar, refer to [alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/).
\ No newline at end of file
diff --git a/site2/website-next/versioned_docs/version-2.4.0/developing-binary-protocol.md b/site2/website-next/versioned_docs/version-2.4.0/develop-binary-protocol.md
similarity index 100%
rename from site2/website-next/versioned_docs/version-2.4.0/developing-binary-protocol.md
rename to site2/website-next/versioned_docs/version-2.4.0/develop-binary-protocol.md
diff --git a/site2/website-next/versioned_docs/version-2.4.0/developing-cpp.md b/site2/website-next/versioned_docs/version-2.4.0/develop-cpp.md
similarity index 100%
rename from site2/website-next/versioned_docs/version-2.4.0/developing-cpp.md
rename to site2/website-next/versioned_docs/version-2.4.0/develop-cpp.md
diff --git a/site2/website-next/versioned_docs/version-2.4.0/develop-load-manager.md b/site2/website-next/versioned_docs/version-2.4.0/develop-load-manager.md
index 509209b..9687f30 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/develop-load-manager.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/develop-load-manager.md
@@ -2,7 +2,6 @@
id: develop-load-manager
title: Modular load manager
sidebar_label: "Modular load manager"
-original_id: develop-load-manager
---
The *modular load manager*, implemented in [`ModularLoadManagerImpl`](https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImpl.java), is a flexible alternative to the previously implemented load manager, [`SimpleLoadManagerImpl`](https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/SimpleLoadManagerImpl.java), which attempts to simplify how load [...]
diff --git a/site2/website-next/versioned_docs/version-2.4.0/develop-tools.md b/site2/website-next/versioned_docs/version-2.4.0/develop-tools.md
index b545779..d034926 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/develop-tools.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/develop-tools.md
@@ -2,7 +2,6 @@
id: develop-tools
title: Simulation tools
sidebar_label: "Simulation tools"
-original_id: develop-tools
---
It is sometimes necessary create an test environment and incur artificial load to observe how well load managers
diff --git a/site2/website-next/versioned_docs/version-2.4.0/functions-deploying.md b/site2/website-next/versioned_docs/version-2.4.0/functions-deploying.md
index 8ad8dbe..fabdbb7 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/functions-deploying.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/functions-deploying.md
@@ -2,260 +2,4 @@
id: functions-deploying
title: Deploying and managing Pulsar Functions
sidebar_label: "Deploying functions"
-original_id: functions-deploying
---
-
-At the moment, there are two deployment modes available for Pulsar Functions:
-
-Mode | Description
-:----|:-----------
-Local run mode | The function runs in your local environment, for example on your laptop
-Cluster mode | The function runs *inside of* your Pulsar cluster, on the same machines as your Pulsar brokers
-
-> #### Contributing new deployment modes
-> The Pulsar Functions feature was designed, however, with extensibility in mind. Other deployment options will be available in the future. If you'd like to add a new deployment option, we recommend getting in touch with the Pulsar developer community at [dev@pulsar.apache.org](mailto:dev@pulsar.apache.org).
-
-## Requirements
-
-In order to deploy and manage Pulsar Functions, you need to have a Pulsar cluster running. There are several options for this:
-
-* You can run a [standalone cluster](getting-started-standalone) locally on your own machine
-* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal.md), [DC/OS](deploy-dcos), and more
-
-If you're running a non-[standalone](reference-terminology.md#standalone) cluster, you'll need to obtain the service URL for the cluster. How you obtain the service URL will depend on how you deployed your Pulsar cluster.
-
-If you're going to deploy and trigger python user-defined functions, you should install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) first.
-
-## Command-line interface
-
-Pulsar Functions are deployed and managed using the [`pulsar-admin functions`](reference-pulsar-admin.md#functions) interface, which contains commands such as [`create`](reference-pulsar-admin.md#functions-create) for deploying functions in [cluster mode](#cluster-mode), [`trigger`](reference-pulsar-admin.md#trigger) for [triggering](#triggering-pulsar-functions) functions, [`list`](reference-pulsar-admin.md#list-2) for listing deployed functions, and several others.
-
-### Fully Qualified Function Name (FQFN)
-
-Each Pulsar Function has a **Fully Qualified Function Name** (FQFN) that consists of three elements: the function's tenant, namespace, and function name. FQFN's look like this:
-
-```http
-
-tenant/namespace/name
-
-```
-
-FQFNs enable you to, for example, create multiple functions with the same name provided that they're in different namespaces.
-
-### Default arguments
-
-When managing Pulsar Functions, you'll need to specify a variety of information about those functions, including tenant, namespace, input and output topics, etc. There are some parameters, however, that have default values that will be supplied if omitted. The table below lists the defaults:
-
-Parameter | Default
-:---------|:-------
-Function name | Whichever value is specified for the class name (minus org, library, etc.). The flag `--classname org.example.MyFunction`, for example, would give the function a name of `MyFunction`.
-Tenant | Derived from the input topics' names. If the input topics are under the `marketing` tenant---i.e. the topic names have the form `persistent://marketing/{namespace}/{topicName}`---then the tenant will be `marketing`.
-Namespace | Derived from the input topics' names. If the input topics are under the `asia` namespace under the `marketing` tenant---i.e. the topic names have the form `persistent://marketing/asia/{topicName}`, then the namespace will be `asia`.
-Output topic | `{input topic}-{function name}-output`. A function with an input topic name of `incoming` and a function name of `exclamation`, for example, would have an output topic of `incoming-exclamation-output`.
-Subscription type | For at-least-once and at-most-once [processing guarantees](functions-guarantees), the [`SHARED`](concepts-messaging.md#shared) is applied by default; for effectively-once guarantees, [`FAILOVER`](concepts-messaging.md#failover) is applied
-Processing guarantees | [`ATLEAST_ONCE`](functions-guarantees)
-Pulsar service URL | `pulsar://localhost:6650`
-
-#### Example use of defaults
-
-Take this `create` command:
-
-```bash
-
-$ bin/pulsar-admin functions create \
- --jar my-pulsar-functions.jar \
- --classname org.example.MyFunction \
- --inputs my-function-input-topic1,my-function-input-topic2
-
-```
-
-The created function would have default values supplied for the function name (`MyFunction`), tenant (`public`), namespace (`default`), subscription type (`SHARED`), processing guarantees (`ATLEAST_ONCE`), and Pulsar service URL (`pulsar://localhost:6650`).
-
-## Local run mode
-
-If you run a Pulsar Function in **local run** mode, it will run on the machine from which the command is run (this could be your laptop, an [AWS EC2](https://aws.amazon.com/ec2/) instance, etc.). Here's an example [`localrun`](reference-pulsar-admin.md#localrun) command:
-
-```bash
-
-$ bin/pulsar-admin functions localrun \
- --py myfunc.py \
- --classname myfunc.SomeFunction \
- --inputs persistent://public/default/input-1 \
- --output persistent://public/default/output-1
-
-```
-
-By default, the function will connect to a Pulsar cluster running on the same machine, via a local [broker](reference-terminology.md#broker) service URL of `pulsar://localhost:6650`. If you'd like to use local run mode to run a function but connect it to a non-local Pulsar cluster, you can specify a different broker URL using the `--brokerServiceUrl` flag. Here's an example:
-
-```bash
-
-$ bin/pulsar-admin functions localrun \
- --broker-service-url pulsar://my-cluster-host:6650 \
- # Other function parameters
-
-```
-
-## Cluster mode
-
-When you run a Pulsar Function in **cluster mode**, the function code will be uploaded to a Pulsar broker and run *alongside the broker* rather than in your [local environment](#local-run-mode). You can run a function in cluster mode using the [`create`](reference-pulsar-admin.md#create-1) command. Here's an example:
-
-```bash
-
-$ bin/pulsar-admin functions create \
- --py myfunc.py \
- --classname myfunc.SomeFunction \
- --inputs persistent://public/default/input-1 \
- --output persistent://public/default/output-1
-
-```
-
-### Updating cluster mode functions
-
-You can use the [`update`](reference-pulsar-admin.md#update-1) command to update a Pulsar Function running in cluster mode. This command, for example, would update the function created in the section [above](#cluster-mode):
-
-```bash
-
-$ bin/pulsar-admin functions update \
- --py myfunc.py \
- --classname myfunc.SomeFunction \
- --inputs persistent://public/default/new-input-topic \
- --output persistent://public/default/new-output-topic
-
-```
-
-### Parallelism
-
-Pulsar Functions run as processes called **instances**. When you run a Pulsar Function, it runs as a single instance by default (and in [local run mode](#local-run-mode) you can *only* run a single instance of a function).
-
-You can also specify the *parallelism* of a function, i.e. the number of instances to run, when you create the function. You can set the parallelism factor using the `--parallelism` flag of the [`create`](reference-pulsar-admin.md#functions-create) command. Here's an example:
-
-```bash
-
-$ bin/pulsar-admin functions create \
- --parallelism 3 \
- # Other function info
-
-```
-
-You can adjust the parallelism of an already created function using the [`update`](reference-pulsar-admin.md#update-1) interface.
-
-```bash
-
-$ bin/pulsar-admin functions update \
- --parallelism 5 \
- # Other function
-
-```
-
-If you're specifying a function's configuration via YAML, use the `parallelism` parameter. Here's an example config file:
-
-```yaml
-
-# function-config.yaml
-parallelism: 3
-inputs:
-- persistent://public/default/input-1
-output: persistent://public/default/output-1
-# other parameters
-
-```
-
-And here's the corresponding update command:
-
-```bash
-
-$ bin/pulsar-admin functions update \
- --function-config-file function-config.yaml
-
-```
-
-### Function instance resources
-
-When you run Pulsar Functions in [cluster run](#cluster-mode) mode, you can specify the resources that are assigned to each function [instance](#parallelism):
-
-Resource | Specified as... | Runtimes
-:--------|:----------------|:--------
-CPU | The number of cores | Docker (coming soon)
-RAM | The number of bytes | Process, Docker
-Disk space | The number of bytes | Docker
-
-Here's an example function creation command that allocates 8 cores, 8 GB of RAM, and 10 GB of disk space to a function:
-
-```bash
-
-$ bin/pulsar-admin functions create \
- --jar target/my-functions.jar \
- --classname org.example.functions.MyFunction \
- --cpu 8 \
- --ram 8589934592 \
- --disk 10737418240
-
-```
-
-> #### Resources are *per instance*
-> The resources that you apply to a given Pulsar Function are applied to each [instance](#parallelism) of the function. If you apply 8 GB of RAM to a function with a parallelism of 5, for example, then you are applying 40 GB of RAM total for the function. You should always make sure to factor parallelism---i.e. the number of instances---into your resource calculations
-
-## Triggering Pulsar Functions
-
-If a Pulsar Function is running in [cluster mode](#cluster-mode), you can **trigger** it at any time using the command line. Triggering a function means that you send a message with a specific value to the function and get the function's output (if any) via the command line.
-
-> Triggering a function is ultimately no different from invoking a function by producing a message on one of the function's input topics. The [`pulsar-admin functions trigger`](reference-pulsar-admin.md#trigger) command is essentially a convenient mechanism for sending messages to functions without needing to use the [`pulsar-client`](reference-cli-tools.md#pulsar-client) tool or a language-specific client library.
-
-To show an example of function triggering, let's start with a simple [Python function](functions-api.md#functions-for-python) that returns a simple string based on the input:
-
-```python
-
-# myfunc.py
-def process(input):
- return "This function has been triggered with a value of {0}".format(input)
-
-```
-
-Let's run that function in [local run mode](functions-deploying.md#local-run-mode):
-
-```bash
-
-$ bin/pulsar-admin functions create \
- --tenant public \
- --namespace default \
- --name myfunc \
- --py myfunc.py \
- --classname myfunc \
- --inputs persistent://public/default/in \
- --output persistent://public/default/out
-
-```
-
-Now let's make a consumer listen on the output topic for messages coming from the `myfunc` function using the [`pulsar-client consume`](reference-cli-tools.md#consume) command:
-
-```bash
-
-$ bin/pulsar-client consume persistent://public/default/out \
- --subscription-name my-subscription
- --num-messages 0 # Listen indefinitely
-
-```
-
-Now let's trigger that function:
-
-```bash
-
-$ bin/pulsar-admin functions trigger \
- --tenant public \
- --namespace default \
- --name myfunc \
- --trigger-value "hello world"
-
-```
-
-The consumer listening on the output topic should then produce this in its logs:
-
-```
-
------ got message -----
-This function has been triggered with a value of hello world
-
-```
-
-> #### Topic info not required
-> In the `trigger` command above, you may have noticed that you only need to specify basic information about the function (tenant, namespace, and name). To trigger the function, you didn't need to know the function's input topic(s).
diff --git a/site2/website-next/versioned_docs/version-2.4.0/functions-guarantees.md b/site2/website-next/versioned_docs/version-2.4.0/functions-guarantees.md
index d9b1438..aefa8d3 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/functions-guarantees.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/functions-guarantees.md
@@ -2,46 +2,4 @@
id: functions-guarantees
title: Processing guarantees
sidebar_label: "Processing guarantees"
-original_id: functions-guarantees
---
-
-Pulsar Functions provides three different messaging semantics that you can apply to any function:
-
-Delivery semantics | Description
-:------------------|:-------
-**At-most-once** delivery | Each message that is sent to the function will most likely be processed but also may not be (hence the "at most")
-**At-least-once** delivery | Each message that is sent to the function could be processed more than once (hence the "at least")
-**Effectively-once** delivery | Each message that is sent to the function will have one output associated with it
-
-## Applying processing guarantees to a function
-
-You can set the processing guarantees for a Pulsar Function when you create the Function. This [`pulsar-function create`](reference-pulsar-admin.md#create-1) command, for example, would apply effectively-once guarantees to the Function:
-
-```bash
-
-$ bin/pulsar-admin functions create \
- --processing-guarantees EFFECTIVELY_ONCE \
- # Other function configs
-
-```
-
-The available options are:
-
-* `ATMOST_ONCE`
-* `ATLEAST_ONCE`
-* `EFFECTIVELY_ONCE`
-
-> By default, Pulsar Functions provide at-least-once delivery guarantees. So if you create a function without supplying a value for the `--processingGuarantees` flag, then the function will provide at-least-once guarantees.
-
-## Updating the processing guarantees of a function
-
-You can change the processing guarantees applied to a function once it's already been created using the [`update`](reference-pulsar-admin.md#update-1) command. Here's an example:
-
-```bash
-
-$ bin/pulsar-admin functions update \
- --processing-guarantees ATMOST_ONCE \
- # Other function configs
-
-```
-
diff --git a/site2/website-next/versioned_docs/version-2.4.0/functions-metrics.md b/site2/website-next/versioned_docs/version-2.4.0/functions-metrics.md
index 8add669..e76c556 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/functions-metrics.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/functions-metrics.md
@@ -2,6 +2,5 @@
id: functions-metrics
title: Metrics for Pulsar Functions
sidebar_label: "Metrics"
-original_id: functions-metrics
---
diff --git a/site2/website-next/versioned_docs/version-2.4.0/io-cdc.md b/site2/website-next/versioned_docs/version-2.4.0/io-cdc.md
index 20f16ae..df37397 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/io-cdc.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/io-cdc.md
@@ -2,7 +2,6 @@
id: io-cdc
title: CDC connector
sidebar_label: "CDC connector"
-original_id: io-cdc
---
CDC source connectors capture log changes of databases (such as MySQL, MongoDB, and PostgreSQL) into Pulsar.
diff --git a/site2/website-next/versioned_docs/version-2.4.0/io-connectors.md b/site2/website-next/versioned_docs/version-2.4.0/io-connectors.md
index 8db368e..9e8ede9 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/io-connectors.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/io-connectors.md
@@ -2,7 +2,6 @@
id: io-connectors
title: Built-in connector
sidebar_label: "Built-in connector"
-original_id: io-connectors
---
Pulsar distribution includes a set of common connectors that have been packaged and tested with the rest of Apache Pulsar. These connectors import and export data from some of the most commonly used data systems.
@@ -46,6 +45,23 @@ Pulsar has various source connectors, which are sorted alphabetically as below.
* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mongodb/src/main/java/org/apache/pulsar/io/debezium/mongodb/DebeziumMongoDbSource.java)
+### Debezium Oracle
+
+* [Configuration](io-debezium-source.md#configuration)
+
+* [Example](io-debezium-source.md#example-of-oracle)
+
+* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/oracle/src/main/java/org/apache/pulsar/io/debezium/oracle/DebeziumOracleSource.java)
+
+### Debezium Microsoft SQL Server
+
+* [Configuration](io-debezium-source.md#configuration)
+
+* [Example](io-debezium-source.md#example-of-microsoft-sql)
+
+* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mssql/src/main/java/org/apache/pulsar/io/debezium/mssql/DebeziumMsSqlSource.java)
+
+
### DynamoDB
* [Configuration](io-dynamodb-source.md#configuration)
diff --git a/site2/website-next/versioned_docs/version-2.4.0/io-develop.md b/site2/website-next/versioned_docs/version-2.4.0/io-develop.md
index 2328f37..bbd6501 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/io-develop.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/io-develop.md
@@ -2,7 +2,6 @@
id: io-develop
title: How to develop Pulsar connectors
sidebar_label: "Develop"
-original_id: io-develop
---
import Tabs from '@theme/Tabs';
diff --git a/site2/website-next/versioned_docs/version-2.4.0/io-overview.md b/site2/website-next/versioned_docs/version-2.4.0/io-overview.md
index 68960a8..810de78 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/io-overview.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/io-overview.md
@@ -2,7 +2,6 @@
id: io-overview
title: Pulsar connector overview
sidebar_label: "Overview"
-original_id: io-overview
---
import Tabs from '@theme/Tabs';
diff --git a/site2/website-next/versioned_docs/version-2.4.0/pulsar-2.0.md b/site2/website-next/versioned_docs/version-2.4.0/pulsar-2.0.md
index 11c5e66..560c8c1 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/pulsar-2.0.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/pulsar-2.0.md
@@ -2,7 +2,6 @@
id: pulsar-2.0
title: Pulsar 2.0
sidebar_label: "Pulsar 2.0"
-original_id: pulsar-2.0
---
Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more.
diff --git a/site2/website-next/versioned_docs/version-2.4.0/reference-pulsar-admin.md b/site2/website-next/versioned_docs/version-2.4.0/pulsar-admin.md
similarity index 100%
rename from site2/website-next/versioned_docs/version-2.4.0/reference-pulsar-admin.md
rename to site2/website-next/versioned_docs/version-2.4.0/pulsar-admin.md
diff --git a/site2/website-next/versioned_docs/version-2.4.0/reference-cli-tools.md b/site2/website-next/versioned_docs/version-2.4.0/reference-cli-tools.md
index 38f52d7..aa12b21 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/reference-cli-tools.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/reference-cli-tools.md
@@ -750,7 +750,7 @@ $ bookkeeper command
```
Commands
-* `auto-recovery`
+* `autorecovery`
* `bookie`
* `localbookie`
* `upgrade`
@@ -773,13 +773,13 @@ The table below lists the environment variables that you can use to configure th
### `auto-recovery`
-Runs an auto-recovery service daemon
+Runs an auto-recovery service
Usage
```bash
-$ bookkeeper auto-recovery options
+$ bookkeeper autorecovery options
```
@@ -787,7 +787,7 @@ Options
|Flag|Description|Default|
|---|---|---|
-|`-c`, `--conf`|Configuration for the auto-recovery daemon||
+|`-c`, `--conf`|Configuration for the auto-recovery||
### `bookie`
@@ -805,7 +805,7 @@ Options
|Flag|Description|Default|
|---|---|---|
-|`-c`, `--conf`|Configuration for the auto-recovery daemon||
+|`-c`, `--conf`|Configuration for the auto-recovery||
|-readOnly|Force start a read-only bookie server|false|
|-withAutoRecovery|Start auto-recovery service bookie server|false|
@@ -836,7 +836,7 @@ Options
|Flag|Description|Default|
|---|---|---|
-|`-c`, `--conf`|Configuration for the auto-recovery daemon||
+|`-c`, `--conf`|Configuration for the auto-recovery||
|`-u`, `--upgrade`|Upgrade the bookie’s directories||
diff --git a/site2/website-next/versioned_docs/version-2.4.0/reference-terminology.md b/site2/website-next/versioned_docs/version-2.4.0/reference-terminology.md
index d0e7368..ebc114d 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/reference-terminology.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/reference-terminology.md
@@ -2,7 +2,6 @@
id: reference-terminology
title: Pulsar Terminology
sidebar_label: "Terminology"
-original_id: reference-terminology
---
Here is a glossary of terms related to Apache Pulsar:
diff --git a/site2/website-next/versioned_docs/version-2.4.0/security-athenz.md b/site2/website-next/versioned_docs/version-2.4.0/security-athenz.md
index 947c3f4..ba27ba4 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/security-athenz.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/security-athenz.md
@@ -2,7 +2,6 @@
id: security-athenz
title: Authentication using Athenz
sidebar_label: "Authentication using Athenz"
-original_id: security-athenz
---
[Athenz](https://github.com/AthenZ/athenz) is a role-based authentication/authorization system. In Pulsar, you can use Athenz role tokens (also known as *z-tokens*) to establish the identify of the client.
diff --git a/site2/website-next/versioned_docs/version-2.4.0/security-authorization.md b/site2/website-next/versioned_docs/version-2.4.0/security-authorization.md
index e678587..b1003d2 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/security-authorization.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/security-authorization.md
@@ -2,7 +2,6 @@
id: security-authorization
title: Authentication and authorization in Pulsar
sidebar_label: "Authorization and ACLs"
-original_id: security-authorization
---
diff --git a/site2/website-next/versioned_docs/version-2.4.0/security-encryption.md b/site2/website-next/versioned_docs/version-2.4.0/security-encryption.md
index cc43082..90d0dbe 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/security-encryption.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/security-encryption.md
@@ -2,7 +2,6 @@
id: security-encryption
title: Pulsar Encryption
sidebar_label: "End-to-End Encryption"
-original_id: security-encryption
---
import Tabs from '@theme/Tabs';
diff --git a/site2/website-next/versioned_docs/version-2.4.0/security-extending.md b/site2/website-next/versioned_docs/version-2.4.0/security-extending.md
index c088e3a..dd0030e 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/security-extending.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/security-extending.md
@@ -2,7 +2,6 @@
id: security-extending
title: Extending Authentication and Authorization in Pulsar
sidebar_label: "Extending"
-original_id: security-extending
---
Pulsar provides a way to use custom authentication and authorization mechanisms.
diff --git a/site2/website-next/versioned_docs/version-2.4.0/sql-deployment-configurations.md b/site2/website-next/versioned_docs/version-2.4.0/sql-deployment-configurations.md
index 6c6fd87..9e7ff5a 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/sql-deployment-configurations.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/sql-deployment-configurations.md
@@ -2,7 +2,6 @@
id: sql-deployment-configurations
title: Pulsar SQL configuration and deployment
sidebar_label: "Configuration and deployment"
-original_id: sql-deployment-configurations
---
You can configure Presto Pulsar connector and deploy a cluster with the following instruction.
@@ -27,6 +26,84 @@ pulsar.entry-read-batch-size=100
# default number of splits to use per query
pulsar.target-num-splits=4
+# max size of one batch message (default value is 5MB)
+pulsar.max-message-size=5242880
+
+# number of split used when querying data from pulsar
+pulsar.target-num-splits=2
+
+# size of queue to buffer entry read from pulsar
+pulsar.max-split-entry-queue-size=1000
+
+# size of queue to buffer message extract from entries
+pulsar.max-split-message-queue-size=10000
+
+# status provider to record connector metrics
+pulsar.stats-provider=org.apache.bookkeeper.stats.NullStatsProvider
+
+# config in map format for stats provider e.g. {"key1":"val1","key2":"val2"}
+pulsar.stats-provider-configs={}
+
+# whether to rewrite Pulsar's default topic delimiter '/'
+pulsar.namespace-delimiter-rewrite-enable=false
+
+# delimiter used to rewrite Pulsar's default delimiter '/', use if default is causing incompatibility with other system like Superset
+pulsar.rewrite-namespace-delimiter=“/”
+
+# maximum number of thread pool size for ledger offloader.
+pulsar.managed-ledger-offload-max-threads=2
+
+# driver used to offload or read cold data to or from long-term storage
+pulsar.managed-ledger-offload-driver=null
+
+# directory to load offloaders nar file.
+pulsar.offloaders-directory="./offloaders"
+
+# properties and configurations related to specific offloader implementation as map e.g. {"key1":"val1","key2":"val2"}
+pulsar.offloader-properties={}
+
+# authentication plugin used to authenticate to Pulsar cluster
+pulsar.auth-plugin=null
+
+# authentication parameter used to authenticate to the Pulsar cluster as a string e.g. "key1:val1,key2:val2".
+pulsar.auth-params=null
+
+# whether the Pulsar client accept an untrusted TLS certificate from broker
+pulsar.tls-allow-insecure-connection=null
+
+# whether to allow hostname verification when a client connects to broker over TLS.
+pulsar.tls-hostname-verification-enable=null
+
+# path for the trusted TLS certificate file of Pulsar broker
+pulsar.tls-trust-cert-file-path=null
+
+# set the threshold for BookKeeper request throttle, default is disabled
+pulsar.bookkeeper-throttle-value=0
+
+# set the number of IO thread
+pulsar.bookkeeper-num-io-threads=2 * Runtime.getRuntime().availableProcessors()
+
+# set the number of worker thread
+pulsar.bookkeeper-num-worker-threads=Runtime.getRuntime().availableProcessors()
+
+# whether to use BookKeeper V2 wire protocol
+pulsar.bookkeeper-use-v2-protocol=true
+
+# interval to check the need for sending an explicit LAC, default is disabled
+pulsar.bookkeeper-explicit-interval=0
+
+# size for managed ledger entry cache (in MB).
+pulsar.managed-ledger-cache-size-MB=0
+
+# number of threads to be used for managed ledger tasks dispatching
+pulsar.managed-ledger-num-worker-threads=Runtime.getRuntime().availableProcessors()
+
+# number of threads to be used for managed ledger scheduled tasks
+pulsar.managed-ledger-num-scheduler-threads=Runtime.getRuntime().availableProcessors()
+
+# directory used to store extraction NAR file
+pulsar.nar-extraction-directory=System.getProperty("java.io.tmpdir")
+
```
You can connect Presto to a Pulsar cluster with multiple hosts. To configure multiple hosts for brokers, add multiple URLs to `pulsar.web-service-url`. To configure multiple hosts for ZooKeeper, add multiple URIs to `pulsar.zookeeper-uri`. The following is an example.
@@ -38,6 +115,21 @@ pulsar.zookeeper-uri=localhost1,localhost2:2181
```
+A frequently asked question is why my latest message not showing up when querying with Pulsar SQL.
+It's not a bug but controlled by a setting, by default BookKeeper LAC only advanced when subsequent entries are added.
+If there is no subsequent entries added, the last entry written will not be visible to readers until the ledger is closed.
+This is not a problem for Pulsar which uses managed ledger, but Pulsar SQL directly read from BookKeeper ledger.
+We can add following setting to change the behavior:
+In Broker config, set
+bookkeeperExplicitLacIntervalInMills > 0
+bookkeeperUseV2WireProtocol=false
+
+And in Presto config, set
+pulsar.bookkeeper-explicit-interval > 0
+pulsar.bookkeeper-use-v2-protocol=false
+
+However,keep in mind that using bk V3 protocol will introduce additional GC overhead to BK as it uses Protobuf.
+
## Query data from existing Presto clusters
If you already have a Presto cluster, you can copy the Presto Pulsar connector plugin to your existing cluster. Download the archived plugin package with the following command.
diff --git a/site2/website-next/versioned_docs/version-2.4.0/sql-overview.md b/site2/website-next/versioned_docs/version-2.4.0/sql-overview.md
index 4a4d5f0..2f827f4 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/sql-overview.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/sql-overview.md
@@ -2,7 +2,6 @@
id: sql-overview
title: Pulsar SQL Overview
sidebar_label: "Overview"
-original_id: sql-overview
---
Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL)](https://trino.io/).
diff --git a/site2/website-next/versioned_docs/version-2.4.0/standalone-docker.md b/site2/website-next/versioned_docs/version-2.4.0/standalone-docker.md
index 05ac2a1..7ee20c2 100644
--- a/site2/website-next/versioned_docs/version-2.4.0/standalone-docker.md
+++ b/site2/website-next/versioned_docs/version-2.4.0/standalone-docker.md
@@ -2,14 +2,11 @@
id: standalone-docker
title: Set up a standalone Pulsar in Docker
sidebar_label: "Run Pulsar in Docker"
-original_id: standalone-docker
---
-For local development and testing, you can run Pulsar in standalone
-mode on your own machine within a Docker container.
+For local development and testing, you can run Pulsar in standalone mode on your own machine within a Docker container.
-If you have not installed Docker, download the [Community edition](https://www.docker.com/community-edition)
-and follow the instructions for your OS.
+If you have not installed Docker, download the [Community edition](https://www.docker.com/community-edition) and follow the instructions for your OS.
## Start Pulsar in Docker
@@ -17,13 +14,7 @@ and follow the instructions for your OS.
```shell
- $ docker run -it \
- -p 6650:6650 \
- -p 8080:8080 \
- --mount source=pulsardata,target=/pulsar/data \
- --mount source=pulsarconf,target=/pulsar/conf \
- apachepulsar/pulsar:@pulsar:version@ \
- bin/pulsar standalone
+ $ docker run -it -p 6650:6650 -p 8080:8080 --mount source=pulsardata,target=/pulsar/data --mount source=pulsarconf,target=/pulsar/conf apachepulsar/pulsar:@pulsar:version@ bin/pulsar standalone
```
@@ -36,8 +27,9 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
```
-2017-08-09 22:34:04,030 - INFO - [main:WebService@213] - Web Service started at http://127.0.0.1:8080
-2017-08-09 22:34:04,038 - INFO - [main:PulsarService@335] - messaging service is ready, bootstrap service on port=8080, broker url=pulsar://127.0.0.1:6650, cluster=standalone, configs=org.apache.pulsar.broker.ServiceConfiguration@4db60246
+08:18:30.970 [main] INFO org.apache.pulsar.broker.web.WebService - HTTP Service started at http://0.0.0.0:8080
+...
+07:53:37.322 [main] INFO org.apache.pulsar.broker.PulsarService - messaging service is ready, bootstrap service port = 8080, broker url= pulsar://localhost:6650, cluster=standalone, configs=org.apache.pulsar.broker.ServiceConfiguration@98b63c1
...
```
@@ -60,7 +52,7 @@ use one of these root URLs to interact with your cluster:
* `pulsar://localhost:6650`
* `http://localhost:8080`
-The following example will guide you get started with Pulsar quickly by using the [Python](client-libraries-python)
+The following example will guide you get started with Pulsar quickly by using the [Python client API](client-libraries-python)
client API.
Install the Pulsar Python client library directly from [PyPI](https://pypi.org/project/pulsar-client/):
@@ -128,51 +120,93 @@ The output is something like this:
```json
{
- "averageMsgSize": 0.0,
- "msgRateIn": 0.0,
- "msgRateOut": 0.0,
- "msgThroughputIn": 0.0,
- "msgThroughputOut": 0.0,
- "publishers": [
- {
- "address": "/172.17.0.1:35048",
- "averageMsgSize": 0.0,
- "clientVersion": "1.19.0-incubating",
- "connectedSince": "2017-08-09 20:59:34.621+0000",
- "msgRateIn": 0.0,
- "msgThroughputIn": 0.0,
- "producerId": 0,
- "producerName": "standalone-0-1"
- }
- ],
- "replication": {},
- "storageSize": 16,
- "subscriptions": {
- "my-sub": {
- "blockedSubscriptionOnUnackedMsgs": false,
- "consumers": [
+ "msgRateIn": 0.0,
+ "msgThroughputIn": 0.0,
+ "msgRateOut": 1.8332950480217471,
+ "msgThroughputOut": 91.33142602871978,
+ "bytesInCounter": 7097,
+ "msgInCounter": 143,
+ "bytesOutCounter": 6607,
+ "msgOutCounter": 133,
+ "averageMsgSize": 0.0,
+ "msgChunkPublished": false,
+ "storageSize": 7097,
+ "backlogSize": 0,
+ "offloadedStorageSize": 0,
+ "publishers": [
{
- "address": "/172.17.0.1:35064",
- "availablePermits": 996,
- "blockedConsumerOnUnackedMsgs": false,
- "clientVersion": "1.19.0-incubating",
- "connectedSince": "2017-08-09 21:05:39.222+0000",
- "consumerName": "166111",
- "msgRateOut": 0.0,
- "msgRateRedeliver": 0.0,
- "msgThroughputOut": 0.0,
- "unackedMessages": 0
+ "accessMode": "Shared",
+ "msgRateIn": 0.0,
+ "msgThroughputIn": 0.0,
+ "averageMsgSize": 0.0,
+ "chunkedMessageRate": 0.0,
+ "producerId": 0,
+ "metadata": {},
+ "address": "/127.0.0.1:35604",
+ "connectedSince": "2021-07-04T09:05:43.04788Z",
+ "clientVersion": "2.8.0",
+ "producerName": "standalone-2-5"
+ }
+ ],
+ "waitingPublishers": 0,
+ "subscriptions": {
+ "my-sub": {
+ "msgRateOut": 1.8332950480217471,
+ "msgThroughputOut": 91.33142602871978,
+ "bytesOutCounter": 6607,
+ "msgOutCounter": 133,
+ "msgRateRedeliver": 0.0,
+ "chunkedMessageRate": 0,
+ "msgBacklog": 0,
+ "backlogSize": 0,
+ "msgBacklogNoDelayed": 0,
+ "blockedSubscriptionOnUnackedMsgs": false,
+ "msgDelayed": 0,
+ "unackedMessages": 0,
+ "type": "Exclusive",
+ "activeConsumerName": "3c544f1daa",
+ "msgRateExpired": 0.0,
+ "totalMsgExpired": 0,
+ "lastExpireTimestamp": 0,
+ "lastConsumedFlowTimestamp": 1625389101290,
+ "lastConsumedTimestamp": 1625389546070,
+ "lastAckedTimestamp": 1625389546162,
+ "lastMarkDeleteAdvancedTimestamp": 1625389546163,
+ "consumers": [
+ {
+ "msgRateOut": 1.8332950480217471,
+ "msgThroughputOut": 91.33142602871978,
+ "bytesOutCounter": 6607,
+ "msgOutCounter": 133,
+ "msgRateRedeliver": 0.0,
+ "chunkedMessageRate": 0.0,
+ "consumerName": "3c544f1daa",
+ "availablePermits": 867,
+ "unackedMessages": 0,
+ "avgMessagesPerEntry": 6,
+ "blockedConsumerOnUnackedMsgs": false,
+ "lastAckedTimestamp": 1625389546162,
+ "lastConsumedTimestamp": 1625389546070,
+ "metadata": {},
+ "address": "/127.0.0.1:35472",
+ "connectedSince": "2021-07-04T08:58:21.287682Z",
+ "clientVersion": "2.8.0"
+ }
+ ],
+ "isDurable": true,
+ "isReplicated": false,
+ "allowOutOfOrderDelivery": false,
+ "consumersAfterMarkDeletePosition": {},
+ "nonContiguousDeletedMessagesRanges": 0,
+ "nonContiguousDeletedMessagesRangesSerializedSize": 0,
+ "durable": true,
+ "replicated": false
}
- ],
- "msgBacklog": 0,
- "msgRateExpired": 0.0,
- "msgRateOut": 0.0,
- "msgRateRedeliver": 0.0,
- "msgThroughputOut": 0.0,
- "type": "Exclusive",
- "unackedMessages": 0
- }
- }
+ },
+ "replication": {},
+ "deduplicationStatus": "Disabled",
+ "nonContiguousDeletedMessagesRanges": 0,
+ "nonContiguousDeletedMessagesRangesSerializedSize": 0
}
```
diff --git a/site2/website-next/versioned_docs/version-2.4.0/getting-started-standalone.md b/site2/website-next/versioned_docs/version-2.4.0/standalone.md
similarity index 100%
rename from site2/website-next/versioned_docs/version-2.4.0/getting-started-standalone.md
rename to site2/website-next/versioned_docs/version-2.4.0/standalone.md
diff --git a/site2/website-next/versioned_docs/version-2.4.1/adaptors-kafka.md b/site2/website-next/versioned_docs/version-2.4.1/adaptors-kafka.md
index ad0d886..27382e9 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/adaptors-kafka.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/adaptors-kafka.md
@@ -2,7 +2,6 @@
id: adaptors-kafka
title: Pulsar adaptor for Apache Kafka
sidebar_label: "Kafka client wrapper"
-original_id: adaptors-kafka
---
@@ -261,6 +260,7 @@ You can configure Pulsar authentication provider directly from the Kafka propert
| [`pulsar.producer.batching.enabled`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ProducerConfiguration.html#setBatchingEnabled-boolean-) | `true` | Control whether automatic batching of messages is enabled for the producer. |
| [`pulsar.producer.batching.max.messages`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ProducerConfiguration.html#setBatchingMaxMessages-int-) | `1000` | The maximum number of messages in a batch. |
| [`pulsar.block.if.producer.queue.full`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ProducerConfiguration.html#setBlockIfQueueFull-boolean-) | | Specify the block producer if queue is full. |
+| [`pulsar.crypto.reader.factory.class.name`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ProducerConfiguration.html#setCryptoKeyReader-org.apache.pulsar.client.api.CryptoKeyReader-) | | Specify the CryptoReader-Factory(`CryptoKeyReaderFactory`) classname which allows producer to create CryptoKeyReader. |
### Pulsar consumer Properties
@@ -272,3 +272,4 @@ You can configure Pulsar authentication provider directly from the Kafka propert
| [`pulsar.consumer.acknowledgments.group.time.millis`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html#acknowledgmentGroupTime-long-java.util.concurrent.TimeUnit-) | 100 | Set the maximum amount of group time for consumers to send the acknowledgments to the broker. |
| [`pulsar.consumer.total.receiver.queue.size.across.partitions`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerConfiguration.html#setMaxTotalReceiverQueueSizeAcrossPartitions-int-) | 50000 | Set the maximum size of the total receiver queue across partitions. |
| [`pulsar.consumer.subscription.topics.mode`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html#subscriptionTopicsMode-Mode-) | PersistentOnly | Set the subscription topic mode for consumers. |
+| [`pulsar.crypto.reader.factory.class.name`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ProducerConfiguration.html#setCryptoKeyReader-org.apache.pulsar.client.api.CryptoKeyReader-) | | Specify the CryptoReader-Factory(`CryptoKeyReaderFactory`) classname which allows consumer to create CryptoKeyReader. |
diff --git a/site2/website-next/versioned_docs/version-2.4.1/adaptors-spark.md b/site2/website-next/versioned_docs/version-2.4.1/adaptors-spark.md
index e14f13b..afa5a7e 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/adaptors-spark.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/adaptors-spark.md
@@ -2,7 +2,6 @@
id: adaptors-spark
title: Pulsar adaptor for Apache Spark
sidebar_label: "Apache Spark"
-original_id: adaptors-spark
---
## Spark Streaming receiver
diff --git a/site2/website-next/versioned_docs/version-2.4.1/adaptors-storm.md b/site2/website-next/versioned_docs/version-2.4.1/adaptors-storm.md
index 76d5071..9df9076 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/adaptors-storm.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/adaptors-storm.md
@@ -2,7 +2,6 @@
id: adaptors-storm
title: Pulsar adaptor for Apache Storm
sidebar_label: "Apache Storm"
-original_id: adaptors-storm
---
Pulsar Storm is an adaptor for integrating with [Apache Storm](http://storm.apache.org/) topologies. It provides core Storm implementations for sending and receiving data.
diff --git a/site2/website-next/versioned_docs/version-2.4.1/admin-api-brokers.md b/site2/website-next/versioned_docs/version-2.4.1/admin-api-brokers.md
index dbac453..10a90ca 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/admin-api-brokers.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/admin-api-brokers.md
@@ -2,7 +2,6 @@
id: admin-api-brokers
title: Managing Brokers
sidebar_label: "Brokers"
-original_id: admin-api-brokers
---
import Tabs from '@theme/Tabs';
@@ -26,9 +25,9 @@ Pulsar brokers consist of two components:
[Brokers](reference-terminology.md#broker) can be managed via:
-* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin) tool
+* The `brokers` command of the [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) tool
* The `/admin/v2/brokers` endpoint of the admin {@inject: rest:REST:/} API
-* The `brokers` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin.html} object in the [Java API](client-libraries-java)
+* The `brokers` method of the `PulsarAdmin` object in the [Java API](client-libraries-java)
In addition to being configurable when you start them up, brokers can also be [dynamically configured](#dynamic-broker-configuration).
diff --git a/site2/website-next/versioned_docs/version-2.4.1/admin-api-clusters.md b/site2/website-next/versioned_docs/version-2.4.1/admin-api-clusters.md
index 972c7e1..8687ae6 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/admin-api-clusters.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/admin-api-clusters.md
@@ -2,7 +2,6 @@
id: admin-api-clusters
title: Managing Clusters
sidebar_label: "Clusters"
-original_id: admin-api-clusters
---
import Tabs from '@theme/Tabs';
@@ -13,7 +12,7 @@ import TabItem from '@theme/TabItem';
>
> This page only shows **some frequently used operations**.
>
-> - For the latest and complete information about `Pulsar admin`, including commands, flags, descriptions, and more information, see [Pulsar admin doc](https://pulsar.apache.org/tools/pulsar-admin/).
+> - For the latest and complete information about `Pulsar admin`, including commands, flags, descriptions, and more, see [Pulsar admin doc](https://pulsar.apache.org/tools/pulsar-admin/)
>
> - For the latest and complete information about `REST API`, including parameters, responses, samples, and more, see {@inject: rest:REST:/} API doc.
>
@@ -24,9 +23,9 @@ servers (aka [bookies](reference-terminology.md#bookie)), and a [ZooKeeper](http
Clusters can be managed via:
-* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin) tool
+* The `clusters` command of the [`pulsar-admin`]([reference-pulsar-admin.md](https://pulsar.apache.org/tools/pulsar-admin/)) tool
* The `/admin/v2/clusters` endpoint of the admin {@inject: rest:REST:/} API
-* The `clusters` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java)
+* The `clusters` method of the `PulsarAdmin` object in the [Java API](client-libraries-java)
## Clusters resources
diff --git a/site2/website-next/versioned_docs/version-2.4.1/admin-api-namespaces.md b/site2/website-next/versioned_docs/version-2.4.1/admin-api-namespaces.md
index 216cb6f..c53fa3c 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/admin-api-namespaces.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/admin-api-namespaces.md
@@ -2,7 +2,6 @@
id: admin-api-namespaces
title: Managing Namespaces
sidebar_label: "Namespaces"
-original_id: admin-api-namespaces
---
import Tabs from '@theme/Tabs';
@@ -23,9 +22,9 @@ Pulsar [namespaces](reference-terminology.md#namespace) are logical groupings of
Namespaces can be managed via:
-* The [`namespaces`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin) tool
+* The `namespaces` command of the [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) tool
* The `/admin/v2/namespaces` endpoint of the admin {@inject: rest:REST:/} API
-* The `namespaces` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java)
+* The `namespaces` method of the `PulsarAdmin` object in the [Java API](client-libraries-java)
## Namespaces resources
@@ -49,8 +48,12 @@ $ pulsar-admin namespaces create test-tenant/test-namespace
</TabItem>
<TabItem value="REST API">
+```
+
{@inject: endpoint|PUT|/admin/v2/namespaces/:tenant/:namespace|operation/createNamespace?version=@pulsar:version_number@}
+```
+
</TabItem>
<TabItem value="Java">
@@ -105,8 +108,12 @@ $ pulsar-admin namespaces policies test-tenant/test-namespace
</TabItem>
<TabItem value="REST API">
+```
+
{@inject: endpoint|GET|/admin/v2/namespaces/:tenant/:namespace|operation/getPolicies?version=@pulsar:version_number@}
+```
+
</TabItem>
<TabItem value="Java">
@@ -142,8 +149,12 @@ test-tenant/ns2
</TabItem>
<TabItem value="REST API">
+```
+
{@inject: endpoint|GET|/admin/v2/namespaces/:tenant|operation/getTenantNamespaces?version=@pulsar:version_number@}
+```
+
</TabItem>
<TabItem value="Java">
@@ -177,8 +188,12 @@ $ pulsar-admin namespaces delete test-tenant/ns1
</TabItem>
<TabItem value="REST API">
+```
+
{@inject: endpoint|DELETE|/admin/v2/namespaces/:tenant/:namespace|operation/deleteNamespace?version=@pulsar:version_number@}
+```
+
</TabItem>
<TabItem value="Java">
@@ -196,7 +211,7 @@ admin.namespaces().deleteNamespace(namespace);
#### Set replication cluster
-It sets replication clusters for a namespace, so Pulsar can internally replicate publish message from one colo to another colo.
+You can set replication clusters for a namespace to enable Pulsar to internally replicate the published messages from one colocation facility to another.
<Tabs
defaultValue="pulsar-admin"
@@ -234,7 +249,7 @@ admin.namespaces().setNamespaceReplicationClusters(namespace, clusters);
#### Get replication cluster
-It gives a list of replication clusters for a given namespace.
+You can get the list of replication clusters for a given namespace.
<Tabs
defaultValue="pulsar-admin"
@@ -281,13 +296,13 @@ admin.namespaces().getNamespaceReplicationClusters(namespace)
Backlog quota helps the broker to restrict bandwidth/storage of a namespace once it reaches a certain threshold limit. Admin can set the limit and take corresponding action after the limit is reached.
- 1. producer_request_hold: broker will hold and not persist produce request payload
+ 1. producer_request_hold: broker holds but not persists produce request payload
- 2. producer_exception: broker disconnects with the client by giving an exception.
+ 2. producer_exception: broker disconnects with the client by giving an exception
- 3. consumer_backlog_eviction: broker will start discarding backlog messages
+ 3. consumer_backlog_eviction: broker starts discarding backlog messages
- Backlog quota restriction can be taken care by defining restriction of backlog-quota-type: destination_storage
+Backlog quota restriction can be taken care by defining restriction of backlog-quota-type: destination_storage.
<Tabs
defaultValue="pulsar-admin"
@@ -300,12 +315,6 @@ $ pulsar-admin namespaces set-backlog-quota --limit 10G --limitTime 36000 --poli
```
-```
-
-N/A
-
-```
-
</TabItem>
<TabItem value="REST API">
@@ -330,7 +339,7 @@ admin.namespaces().setBacklogQuota(namespace, new BacklogQuota(limit, limitTime,
#### Get backlog quota policies
-It shows a configured backlog quota for a given namespace.
+You can get a configured backlog quota for a given namespace.
<Tabs
defaultValue="pulsar-admin"
@@ -378,7 +387,7 @@ admin.namespaces().getBacklogQuotaMap(namespace);
#### Remove backlog quota policies
-It removes backlog quota policies for a given namespace
+You can remove backlog quota policies for a given namespace.
<Tabs
defaultValue="pulsar-admin"
@@ -391,12 +400,6 @@ $ pulsar-admin namespaces remove-backlog-quota test-tenant/ns1
```
-```
-
-N/A
-
-```
-
</TabItem>
<TabItem value="REST API">
@@ -423,7 +426,7 @@ admin.namespaces().removeBacklogQuota(namespace, backlogQuotaType)
#### Set persistence policies
-Persistence policies allow to configure persistency-level for all topic messages under a given namespace.
+Persistence policies allow users to configure persistency-level for all topic messages under a given namespace.
- Bookkeeper-ack-quorum: Number of acks (guaranteed copies) to wait for each entry, default: 0
@@ -444,12 +447,6 @@ $ pulsar-admin namespaces set-persistence --bookkeeper-ack-quorum 2 --bookkeeper
```
-```
-
-N/A
-
-```
-
</TabItem>
<TabItem value="REST API">
@@ -474,7 +471,7 @@ admin.namespaces().setPersistence(namespace,new PersistencePolicies(bookkeeperEn
#### Get persistence policies
-It shows the configured persistence policies of a given namespace.
+You can get the configured persistence policies of a given namespace.
<Tabs
defaultValue="pulsar-admin"
@@ -537,12 +534,6 @@ $ pulsar-admin namespaces unload --bundle 0x00000000_0xffffffff test-tenant/ns1
```
-```
-
-N/A
-
-```
-
</TabItem>
<TabItem value="REST API">
@@ -567,8 +558,7 @@ admin.namespaces().unloadNamespaceBundle(namespace, bundle)
#### Split namespace bundles
-Each namespace bundle can contain multiple topics and each bundle can be served by only one broker.
-If a single bundle is creating an excessive load on a broker, an admin splits the bundle using this command permitting one or more of the new bundles to be unloaded thus spreading the load across the brokers.
+One namespace bundle can contain multiple topics but can be served by only one broker. If a single bundle is creating an excessive load on a broker, an admin can split the bundle using the command below, permitting one or more of the new bundles to be unloaded, thus balancing the load across the brokers.
<Tabs
defaultValue="pulsar-admin"
@@ -581,12 +571,6 @@ $ pulsar-admin namespaces split-bundle --bundle 0x00000000_0xffffffff test-tenan
```
-```
-
-N/A
-
-```
-
</TabItem>
<TabItem value="REST API">
@@ -613,7 +597,7 @@ admin.namespaces().splitNamespaceBundle(namespace, bundle)
#### Set message-ttl
-It configures message’s time to live (in seconds) duration.
+You can configure the time to live (in seconds) duration for messages. In the example below, the message-ttl is set as 100s.
<Tabs
defaultValue="pulsar-admin"
@@ -626,12 +610,6 @@ $ pulsar-admin namespaces set-message-ttl --messageTTL 100 test-tenant/ns1
```
-```
-
-N/A
-
-```
-
</TabItem>
<TabItem value="REST API">
@@ -656,7 +634,7 @@ admin.namespaces().setNamespaceMessageTTL(namespace, messageTTL)
#### Get message-ttl
-It gives a message ttl of configured namespace.
+When the message-ttl for a namespace is set, you can use the command below to get the configured value. This example comtinues the example of the command `set message-ttl`, so the returned value is 100(s).
<Tabs
defaultValue="pulsar-admin"
@@ -684,6 +662,12 @@ $ pulsar-admin namespaces get-message-ttl test-tenant/ns1
```
+```
+
+100
+
+```
+
</TabItem>
<TabItem value="Java">
@@ -693,6 +677,12 @@ admin.namespaces().getNamespaceMessageTTL(namespace)
```
+```
+
+100
+
+```
+
</TabItem>
</Tabs>
@@ -712,12 +702,6 @@ $ pulsar-admin namespaces remove-message-ttl test-tenant/ns1
```
-```
-
-100
-
-```
-
</TabItem>
<TabItem value="REST API">
@@ -758,12 +742,6 @@ $ pulsar-admin namespaces clear-backlog --sub my-subscription test-tenant/ns1
```
-```
-
-N/A
-
-```
-
</TabItem>
<TabItem value="REST API">
@@ -801,12 +779,6 @@ $ pulsar-admin namespaces clear-backlog --bundle 0x00000000_0xffffffff --sub m
```
-```
-
-N/A
-
-```
-
</TabItem>
<TabItem value="REST API">
@@ -842,13 +814,7 @@ Each namespace contains multiple topics and the retention size (storage size) of
```
-$ pulsar-admin set-retention --size 100 --time 10 test-tenant/ns1
-
-```
-
-```
-
-N/A
+$ pulsar-admin namespaces set-retention --size 100 --time 10 test-tenant/ns1
```
@@ -932,9 +898,7 @@ disables the throttling.
:::note
- If neither `clusterDispatchRate` nor `topicDispatchRate` is configured, dispatch throttling is disabled.
->
- If `topicDispatchRate` is not configured, `clusterDispatchRate` takes effect.
->
- If `topicDispatchRate` is configured, `topicDispatchRate` takes effect.
:::
diff --git a/site2/website-next/versioned_docs/version-2.4.1/admin-api-permissions.md b/site2/website-next/versioned_docs/version-2.4.1/admin-api-permissions.md
index e2ca469..faedbf1 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/admin-api-permissions.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/admin-api-permissions.md
@@ -2,7 +2,6 @@
id: admin-api-permissions
title: Managing permissions
sidebar_label: "Permissions"
-original_id: admin-api-permissions
---
import Tabs from '@theme/Tabs';
@@ -13,7 +12,7 @@ import TabItem from '@theme/TabItem';
>
> This page only shows **some frequently used operations**.
>
-> - For the latest and complete information about `Pulsar admin`, including commands, flags, descriptions, and more information, see [Pulsar admin doc](https://pulsar.apache.org/tools/pulsar-admin/).
+> - For the latest and complete information about `Pulsar admin`, including commands, flags, descriptions, and more, see [Pulsar admin doc](https://pulsar.apache.org/tools/pulsar-admin/)
>
> - For the latest and complete information about `REST API`, including parameters, responses, samples, and more, see {@inject: rest:REST:/} API doc.
>
diff --git a/site2/website-next/versioned_docs/version-2.4.1/admin-api-persistent-topics.md b/site2/website-next/versioned_docs/version-2.4.1/admin-api-persistent-topics.md
index b6d293b..8a7abae 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/admin-api-persistent-topics.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/admin-api-persistent-topics.md
@@ -2,7 +2,6 @@
id: admin-api-persistent-topics
title: Managing persistent topics
sidebar_label: "Persistent topics"
-original_id: admin-api-persistent-topics
---
For details of the content, refer to [manage topics](admin-api-topics).
\ No newline at end of file
diff --git a/site2/website-next/versioned_docs/version-2.4.1/admin-api-schemas.md b/site2/website-next/versioned_docs/version-2.4.1/admin-api-schemas.md
index 9ffe21f..8399a03 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/admin-api-schemas.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/admin-api-schemas.md
@@ -2,6 +2,5 @@
id: admin-api-schemas
title: Managing Schemas
sidebar_label: "Schemas"
-original_id: admin-api-schemas
---
diff --git a/site2/website-next/versioned_docs/version-2.4.1/admin-api-tenants.md b/site2/website-next/versioned_docs/version-2.4.1/admin-api-tenants.md
index fe68336..570ac31 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/admin-api-tenants.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/admin-api-tenants.md
@@ -2,7 +2,6 @@
id: admin-api-tenants
title: Managing Tenants
sidebar_label: "Tenants"
-original_id: admin-api-tenants
---
import Tabs from '@theme/Tabs';
@@ -13,7 +12,7 @@ import TabItem from '@theme/TabItem';
>
> This page only shows **some frequently used operations**.
>
-> - For the latest and complete information about `Pulsar admin`, including commands, flags, descriptions, and more information, see [Pulsar admin doc](https://pulsar.apache.org/tools/pulsar-admin/).
+> - For the latest and complete information about `Pulsar admin`, including commands, flags, descriptions, and more, see [Pulsar admin doc](https://pulsar.apache.org/tools/pulsar-admin/)
>
> - For the latest and complete information about `REST API`, including parameters, responses, samples, and more, see {@inject: rest:REST:/} API doc.
>
@@ -80,22 +79,26 @@ $ pulsar-admin tenants create my-tenant
```
-When creating a tenant, you can assign admin roles using the `-r`/`--admin-roles` flag. You can specify multiple roles as a comma-separated list. Here are some examples:
+When creating a tenant, you can optionally assign admin roles using the `-r`/`--admin-roles`
+flag, and clusters using the `-c`/`--allowed-clusters` flag. You can specify multiple values
+as a comma-separated list. Here are some examples:
```shell
$ pulsar-admin tenants create my-tenant \
- --admin-roles role1,role2,role3
+ --admin-roles role1,role2,role3 \
+ --allowed-clusters cluster1
$ pulsar-admin tenants create my-tenant \
-r role1
+ -c cluster1
```
</TabItem>
<TabItem value="REST API">
-{@inject: endpoint|POST|/admin/v2/tenants/:tenant|operation/createTenant?version=@pulsar:version_number@}
+{@inject: endpoint|PUT|/admin/v2/tenants/:tenant|operation/createTenant?version=@pulsar:version_number@}
</TabItem>
<TabItem value="JAVA">
@@ -140,7 +143,7 @@ $ pulsar-admin tenants get my-tenant
</TabItem>
<TabItem value="REST API">
-{@inject: endpoint|GET|/admin/v2/tenants/:cluster|operation/getTenant?version=@pulsar:version_number@}
+{@inject: endpoint|GET|/admin/v2/tenants/:tenant|operation/getTenant?version=@pulsar:version_number@}
</TabItem>
<TabItem value="JAVA">
@@ -175,7 +178,7 @@ $ pulsar-admin tenants delete my-tenant
</TabItem>
<TabItem value="REST API">
-{@inject: endpoint|DELETE|/admin/v2/tenants/:cluster|operation/deleteTenant?version=@pulsar:version_number@}
+{@inject: endpoint|DELETE|/admin/v2/tenants/:tenant|operation/deleteTenant?version=@pulsar:version_number@}
</TabItem>
<TabItem value="JAVA">
@@ -210,7 +213,7 @@ $ pulsar-admin tenants update my-tenant
</TabItem>
<TabItem value="REST API">
-{@inject: endpoint|DELETE|/admin/v2/tenants/:cluster|operation/updateTenant?version=@pulsar:version_number@}
+{@inject: endpoint|POST|/admin/v2/tenants/:tenant|operation/updateTenant?version=@pulsar:version_number@}
</TabItem>
<TabItem value="JAVA">
diff --git a/site2/website-next/versioned_docs/version-2.4.1/administration-load-balance.md b/site2/website-next/versioned_docs/version-2.4.1/administration-load-balance.md
index 3efba60..834b156 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/administration-load-balance.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/administration-load-balance.md
@@ -2,13 +2,11 @@
id: administration-load-balance
title: Pulsar load balance
sidebar_label: "Load balance"
-original_id: administration-load-balance
---
## Load balance across Pulsar brokers
-Pulsar is an horizontally scalable messaging system, so the traffic
-in a logical cluster must be spread across all the available Pulsar brokers as evenly as possible, which is a core requirement.
+Pulsar is an horizontally scalable messaging system, so the traffic in a logical cluster must be balanced across all the available Pulsar brokers as evenly as possible, which is a core requirement.
You can use multiple settings and tools to control the traffic distribution which require a bit of context to understand how the traffic is managed in Pulsar. Though, in most cases, the core requirement mentioned above is true out of the box and you should not worry about it.
@@ -36,11 +34,9 @@ Instead of individual topic or partition assignment, each broker takes ownership
The namespace is the "administrative" unit: many config knobs or operations are done at the namespace level.
-For assignment, a namespaces is sharded into a list of "bundles", with each bundle comprising
-a portion of overall hash range of the namespace.
+For assignment, a namespaces is sharded into a list of "bundles", with each bundle comprising a portion of overall hash range of the namespace.
-Topics are assigned to a particular bundle by taking the hash of the topic name and checking in which
-bundle the hash falls into.
+Topics are assigned to a particular bundle by taking the hash of the topic name and checking in which bundle the hash falls into.
Each bundle is independent of the others and thus is independently assigned to different brokers.
@@ -72,8 +68,7 @@ On the same note, it is beneficial to start with more bundles than the number of
### Unload topics and bundles
-You can "unload" a topic in Pulsar with admin operation. Unloading means to close the topics,
-release ownership and reassign the topics to a new broker, based on current load.
+You can "unload" a topic in Pulsar with admin operation. Unloading means to close the topics, release ownership and reassign the topics to a new broker, based on current load.
When unloading happens, the client experiences a small latency blip, typically in the order of tens of milliseconds, while the topic is reassigned.
@@ -97,9 +92,11 @@ pulsar-admin namespaces unload tenant/namespace
### Split namespace bundles
-Since the load for the topics in a bundle might change over time, or predicting upfront might just be hard, brokers can split bundles into two. The new smaller bundles can be reassigned to different brokers.
+Since the load for the topics in a bundle might change over time and predicting the load might be hard, bundle split is designed to deal with these issues. The broker splits a bundle into two and the new smaller bundles can be reassigned to different brokers.
-The splitting happens based on some tunable thresholds. Any existing bundle that exceeds any of the threshold is a candidate to be split. By default the newly split bundles are also immediately offloaded to other brokers, to facilitate the traffic distribution.
+The splitting is based on some tunable thresholds. Any existing bundle that exceeds any of the threshold is a candidate to be split. By default the newly split bundles are also immediately offloaded to other brokers, to facilitate the traffic distribution.
+
+You can split namespace bundles in two ways, by setting `supportedNamespaceBundleSplitAlgorithms` to `range_equally_divide` or `topic_count_equally_divide` in `broker.conf` file. The former splits the bundle into two parts with the same hash range size; the latter splits the bundle into two parts with the same number of topics. You can also configure other parameters for namespace bundles.
```properties
@@ -130,13 +127,11 @@ loadBalancerNamespaceMaximumBundles=128
The support for automatic load shedding is available in the load manager of Pulsar. This means that whenever the system recognizes a particular broker is overloaded, the system forces some traffic to be reassigned to less loaded brokers.
-When a broker is identified as overloaded, the broker forces to "unload" a subset of the bundles, the
-ones with higher traffic, that make up for the overload percentage.
+When a broker is identified as overloaded, the broker forces to "unload" a subset of the bundles, the ones with higher traffic, that make up for the overload percentage.
For example, the default threshold is 85% and if a broker is over quota at 95% CPU usage, then the broker unloads the percent difference plus a 5% margin: `(95% - 85%) + 5% = 15%`.
-Given the selection of bundles to offload is based on traffic (as a proxy measure for cpu, network
-and memory), broker unloads bundles for at least 15% of traffic.
+Given the selection of bundles to offload is based on traffic (as a proxy measure for cpu, network and memory), broker unloads bundles for at least 15% of traffic.
The automatic load shedding is enabled by default and you can disable the automatic load shedding with this setting:
@@ -160,6 +155,20 @@ loadBalancerSheddingGracePeriodMinutes=30
```
+Pulsar supports three types of shedding strategies:
+
+##### ThresholdShedder
+This strategy tends to shed the bundles if any broker's usage is above the configured threshold. It does this by first computing the average resource usage per broker for the whole cluster. The resource usage for each broker is calculated using the following method: LocalBrokerData#getMaxResourceUsageWithWeight). The weights for each resource are configurable. Historical observations are included in the running average based on the broker's setting for loadBalancerHistoryResourcePercenta [...]
+`loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder`
+
+##### OverloadShedder
+This strategy will attempt to shed exactly one bundle on brokers which are overloaded, that is, whose maximum system resource usage exceeds loadBalancerBrokerOverloadedThresholdPercentage. To see which resources are considered when determining the maximum system resource. A bundle is recommended for unloading off that broker if and only if the following conditions hold: The broker has at least two bundles assigned and the broker has at least one bundle that has not been unloaded recently [...]
+`loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.impl.OverloadShedder`
+
+##### UniformLoadShedder
+This strategy tends to distribute load uniformly across all brokers. This strategy checks laod difference between broker with highest load and broker with lowest load. If the difference is higher than configured thresholds `loadBalancerMsgRateDifferenceShedderThreshold` and `loadBalancerMsgThroughputMultiplierDifferenceShedderThreshold` then it finds out bundles which can be unloaded to distribute traffic evenly across all brokers. Configure broker with below value to use this strategy.
+`loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.impl.UniformLoadShedder`
+
#### Broker overload thresholds
The determinations of when a broker is overloaded is based on threshold of CPU, network and memory usage. Whenever either of those metrics reaches the threshold, the system triggers the shedding (if enabled).
@@ -175,9 +184,7 @@ loadBalancerBrokerOverloadedThresholdPercentage=85
Pulsar gathers the usage stats from the system metrics.
-In case of network utilization, in some cases the network interface speed that Linux reports is
-not correct and needs to be manually overridden. This is the case in AWS EC2 instances with 1Gbps
-NIC speed for which the OS reports 10Gbps speed.
+In case of network utilization, in some cases the network interface speed that Linux reports is not correct and needs to be manually overridden. This is the case in AWS EC2 instances with 1Gbps NIC speed for which the OS reports 10Gbps speed.
Because of the incorrect max speed, the Pulsar load manager might think the broker has not reached the NIC capacity, while in fact the broker already uses all the bandwidth and the traffic is slowed down.
diff --git a/site2/website-next/versioned_docs/version-2.4.1/administration-proxy.md b/site2/website-next/versioned_docs/version-2.4.1/administration-proxy.md
index c046ed3..3cef937 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/administration-proxy.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/administration-proxy.md
@@ -2,10 +2,9 @@
id: administration-proxy
title: Pulsar proxy
sidebar_label: "Pulsar proxy"
-original_id: administration-proxy
---
-Pulsar proxy is an optional gateway. Pulsar proxy is used when direction connections between clients and Pulsar brokers are either infeasible or undesirable. For example, when you run Pulsar in a cloud environment or on [Kubernetes](https://kubernetes.io) or an analogous platform, you can run Pulsar proxy.
+Pulsar proxy is an optional gateway. Pulsar proxy is used when direct connections between clients and Pulsar brokers are either infeasible or undesirable. For example, when you run Pulsar in a cloud environment or on [Kubernetes](https://kubernetes.io) or an analogous platform, you can run Pulsar proxy.
## Configure the proxy
diff --git a/site2/website-next/versioned_docs/version-2.4.1/administration-stats.md b/site2/website-next/versioned_docs/version-2.4.1/administration-stats.md
index ac0c036..2ccd73c 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/administration-stats.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/administration-stats.md
@@ -2,7 +2,6 @@
id: administration-stats
title: Pulsar stats
sidebar_label: "Pulsar statistics"
-original_id: administration-stats
---
## Partitioned topics
diff --git a/site2/website-next/versioned_docs/version-2.4.1/administration-zk-bk.md b/site2/website-next/versioned_docs/version-2.4.1/administration-zk-bk.md
index de10d50..e5f9688 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/administration-zk-bk.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/administration-zk-bk.md
@@ -2,7 +2,6 @@
id: administration-zk-bk
title: ZooKeeper and BookKeeper administration
sidebar_label: "ZooKeeper and BookKeeper"
-original_id: administration-zk-bk
---
Pulsar relies on two external systems for essential tasks:
diff --git a/site2/website-next/versioned_docs/version-2.4.1/client-libraries-cpp.md b/site2/website-next/versioned_docs/version-2.4.1/client-libraries-cpp.md
index 333ec67..958861a 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/client-libraries-cpp.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/client-libraries-cpp.md
@@ -2,7 +2,6 @@
id: client-libraries-cpp
title: Pulsar C++ client
sidebar_label: "C++"
-original_id: client-libraries-cpp
---
You can use Pulsar C++ client to create Pulsar producers and consumers in C++.
@@ -11,7 +10,7 @@ All the methods in producer, consumer, and reader of a C++ client are thread-saf
## Supported platforms
-Pulsar C++ client is supported on **Linux** and **MacOS** platforms.
+Pulsar C++ client is supported on **Linux** ,**MacOS** and **Windows** platforms.
[Doxygen](http://www.doxygen.nl/)-generated API docs for the C++ client are available [here](/api/cpp).
@@ -21,8 +20,8 @@ You need to install the following components before using the C++ client:
* [CMake](https://cmake.org/)
* [Boost](http://www.boost.org/)
-* [Protocol Buffers](https://developers.google.com/protocol-buffers/) 2.6
-* [libcurl](https://curl.haxx.se/libcurl/)
+* [Protocol Buffers](https://developers.google.com/protocol-buffers/) >= 3
+* [libcurl](https://curl.se/libcurl/)
* [Google Test](https://github.com/google/googletest)
## Linux
@@ -147,6 +146,12 @@ $ rpm -ivh apache-pulsar-client*.rpm
After you install RPM successfully, Pulsar libraries are in the `/usr/lib` directory.
+:::note
+
+If you get the error that `libpulsar.so: cannot open shared object file: No such file or directory` when starting Pulsar client, you may need to run `ldconfig` first.
+
+:::
+
### Install Debian
1. Download a Debian package from the links in the table.
@@ -236,10 +241,8 @@ $ export OPENSSL_INCLUDE_DIR=/usr/local/opt/openssl/include/
$ export OPENSSL_ROOT_DIR=/usr/local/opt/openssl/
# Protocol Buffers installation
-$ brew tap homebrew/versions
-$ brew install protobuf260
-$ brew install boost
-$ brew install log4cxx
+$ brew install protobuf boost boost-python log4cxx
+# If you are using python3, you need to install boost-python3
# Google Test installation
$ git clone https://github.com/google/googletest.git
@@ -269,6 +272,50 @@ brew install libpulsar
```
+## Windows (64-bit)
+
+### Compilation
+
+1. Clone the Pulsar repository.
+
+```shell
+
+$ git clone https://github.com/apache/pulsar
+
+```
+
+2. Install all necessary dependencies.
+
+```shell
+
+cd ${PULSAR_HOME}/pulsar-client-cpp
+vcpkg install --feature-flags=manifests --triplet x64-windows
+
+```
+
+3. Build C++ libraries.
+
+```shell
+
+cmake -B ./build -A x64 -DBUILD_PYTHON_WRAPPER=OFF -DBUILD_TESTS=OFF -DVCPKG_TRIPLET=x64-windows -DCMAKE_BUILD_TYPE=Release -S .
+cmake --build ./build --config Release
+
+```
+
+> **NOTE**
+>
+> 1. For Windows 32-bit, you need to use `-A Win32` and `-DVCPKG_TRIPLET=x86-windows`.
+> 2. For MSVC Debug mode, you need to replace `Release` with `Debug` for both `CMAKE_BUILD_TYPE` variable and `--config` option.
+
+4. Client libraries are available in the following places.
+
+```
+
+${PULSAR_HOME}/pulsar-client-cpp/build/lib/Release/pulsar.lib
+${PULSAR_HOME}/pulsar-client-cpp/build/lib/Release/pulsar.dll
+
+```
+
## Connection URLs
To connect Pulsar using client libraries, you need to specify a Pulsar protocol URL.
@@ -299,109 +346,361 @@ pulsar+ssl://pulsar.us-west.example.com:6651
## Create a consumer
-To use Pulsar as a consumer, you need to create a consumer on the C++ client. The following is an example.
+To use Pulsar as a consumer, you need to create a consumer on the C++ client. There are two main ways of using the consumer:
+- [Blocking style](#blocking-example): synchronously calling `receive(msg)`.
+- [Non-blocking](#consumer-with-a-message-listener) (event based) style: using a message listener.
+
+### Blocking example
+
+The benefit of this approach is that it is the simplest code. Simply keeps calling `receive(msg)` which blocks until a message is received.
+
+This example starts a subscription at the earliest offset and consumes 100 messages.
```c++
-Client client("pulsar://localhost:6650");
+#include <pulsar/Client.h>
+
+using namespace pulsar;
+
+int main() {
+ Client client("pulsar://localhost:6650");
+
+ Consumer consumer;
+ ConsumerConfiguration config;
+ config.setSubscriptionInitialPosition(InitialPositionEarliest);
+ Result result = client.subscribe("persistent://public/default/my-topic", "consumer-1", config, consumer);
+ if (result != ResultOk) {
+ std::cout << "Failed to subscribe: " << result << std::endl;
+ return -1;
+ }
-Consumer consumer;
-Result result = client.subscribe("my-topic", "my-subscription-name", consumer);
-if (result != ResultOk) {
- LOG_ERROR("Failed to subscribe: " << result);
- return -1;
+ Message msg;
+ int ctr = 0;
+ // consume 100 messages
+ while (ctr < 100) {
+ consumer.receive(msg);
+ std::cout << "Received: " << msg
+ << " with payload '" << msg.getDataAsString() << "'" << std::endl;
+
+ consumer.acknowledge(msg);
+ ctr++;
+ }
+
+ std::cout << "Finished consuming synchronously!" << std::endl;
+
+ client.close();
+ return 0;
}
-Message msg;
+```
+
+### Consumer with a message listener
+
+You can avoid running a loop with blocking calls with an event based style by using a message listener which is invoked for each message that is received.
+
+This example starts a subscription at the earliest offset and consumes 100 messages.
-while (true) {
- consumer.receive(msg);
- LOG_INFO("Received: " << msg
- << " with payload '" << msg.getDataAsString() << "'");
+```c++
+
+#include <pulsar/Client.h>
+#include <atomic>
+#include <thread>
+
+using namespace pulsar;
+
+std::atomic<uint32_t> messagesReceived;
+
+void handleAckComplete(Result res) {
+ std::cout << "Ack res: " << res << std::endl;
+}
- consumer.acknowledge(msg);
+void listener(Consumer consumer, const Message& msg) {
+ std::cout << "Got message " << msg << " with content '" << msg.getDataAsString() << "'" << std::endl;
+ messagesReceived++;
+ consumer.acknowledgeAsync(msg.getMessageId(), handleAckComplete);
}
-client.close();
+int main() {
+ Client client("pulsar://localhost:6650");
+
+ Consumer consumer;
+ ConsumerConfiguration config;
+ config.setMessageListener(listener);
+ config.setSubscriptionInitialPosition(InitialPositionEarliest);
+ Result result = client.subscribe("persistent://public/default/my-topic", "consumer-1", config, consumer);
+ if (result != ResultOk) {
+ std::cout << "Failed to subscribe: " << result << std::endl;
+ return -1;
+ }
+
+ // wait for 100 messages to be consumed
+ while (messagesReceived < 100) {
+ std::this_thread::sleep_for(std::chrono::milliseconds(100));
+ }
+
+ std::cout << "Finished consuming asynchronously!" << std::endl;
+
+ client.close();
+ return 0;
+}
```
## Create a producer
-To use Pulsar as a producer, you need to create a producer on the C++ client. The following is an example.
+To use Pulsar as a producer, you need to create a producer on the C++ client. There are two main ways of using a producer:
+- [Blocking style](#simple-blocking-example) : each call to `send` waits for an ack from the broker.
+- [Non-blocking asynchronous style](#non-blocking-example) : `sendAsync` is called instead of `send` and a callback is supplied for when the ack is received from the broker.
+
+### Simple blocking example
+
+This example sends 100 messages using the blocking style. While simple, it does not produce high throughput as it waits for each ack to come back before sending the next message.
```c++
-Client client("pulsar://localhost:6650");
+#include <pulsar/Client.h>
+#include <thread>
-Producer producer;
-Result result = client.createProducer("my-topic", producer);
-if (result != ResultOk) {
- LOG_ERROR("Error creating producer: " << result);
- return -1;
-}
+using namespace pulsar;
+
+int main() {
+ Client client("pulsar://localhost:6650");
+
+ Result result = client.createProducer("persistent://public/default/my-topic", producer);
+ if (result != ResultOk) {
+ std::cout << "Error creating producer: " << result << std::endl;
+ return -1;
+ }
+
+ // Send 100 messages synchronously
+ int ctr = 0;
+ while (ctr < 100) {
+ std::string content = "msg" + std::to_string(ctr);
+ Message msg = MessageBuilder().setContent(content).setProperty("x", "1").build();
+ Result result = producer.send(msg);
+ if (result != ResultOk) {
+ std::cout << "The message " << content << " could not be sent, received code: " << result << std::endl;
+ } else {
+ std::cout << "The message " << content << " sent successfully" << std::endl;
+ }
+
+ std::this_thread::sleep_for(std::chrono::milliseconds(100));
+ ctr++;
+ }
+
+ std::cout << "Finished producing synchronously!" << std::endl;
-// Publish 10 messages to the topic
-for (int i = 0; i < 10; i++){
- Message msg = MessageBuilder().setContent("my-message").build();
- Result res = producer.send(msg);
- LOG_INFO("Message sent: " << res);
+ client.close();
+ return 0;
}
-client.close();
```
-## Enable authentication in connection URLs
-If you use TLS authentication when connecting to Pulsar, you need to add `ssl` in the connection URLs, and the default port is `6651`. The following is an example.
+### Non-blocking example
-```cpp
+This example sends 100 messages using the non-blocking style calling `sendAsync` instead of `send`. This allows the producer to have multiple messages inflight at a time which increases throughput.
-ClientConfiguration config = ClientConfiguration();
-config.setUseTls(true);
-config.setTlsTrustCertsFilePath("/path/to/cacert.pem");
-config.setTlsAllowInsecureConnection(false);
-config.setAuth(pulsar::AuthTls::create(
- "/path/to/client-cert.pem", "/path/to/client-key.pem"););
+The producer configuration `blockIfQueueFull` is useful here to avoid `ResultProducerQueueIsFull` errors when the internal queue for outgoing send requests becomes full. Once the internal queue is full, `sendAsync` becomes blocking which can make your code simpler.
-Client client("pulsar+ssl://my-broker.com:6651", config);
+Without this configuration, the result code `ResultProducerQueueIsFull` is passed to the callback. You must decide how to deal with that (retry, discard etc).
+
+```c++
+
+#include <pulsar/Client.h>
+#include <thread>
+
+using namespace pulsar;
+
+std::atomic<uint32_t> acksReceived;
+
+void callback(Result code, const MessageId& msgId, std::string msgContent) {
+ // message processing logic here
+ std::cout << "Received ack for msg: " << msgContent << " with code: "
+ << code << " -- MsgID: " << msgId << std::endl;
+ acksReceived++;
+}
+
+int main() {
+ Client client("pulsar://localhost:6650");
+
+ ProducerConfiguration producerConf;
+ producerConf.setBlockIfQueueFull(true);
+ Producer producer;
+ Result result = client.createProducer("persistent://public/default/my-topic",
+ producerConf, producer);
+ if (result != ResultOk) {
+ std::cout << "Error creating producer: " << result << std::endl;
+ return -1;
+ }
+
+ // Send 100 messages asynchronously
+ int ctr = 0;
+ while (ctr < 100) {
+ std::string content = "msg" + std::to_string(ctr);
+ Message msg = MessageBuilder().setContent(content).setProperty("x", "1").build();
+ producer.sendAsync(msg, std::bind(callback,
+ std::placeholders::_1, std::placeholders::_2, content));
+
+ std::this_thread::sleep_for(std::chrono::milliseconds(100));
+ ctr++;
+ }
+
+ // wait for 100 messages to be acked
+ while (acksReceived < 100) {
+ std::this_thread::sleep_for(std::chrono::milliseconds(100));
+ }
+
+ std::cout << "Finished producing asynchronously!" << std::endl;
+
+ client.close();
+ return 0;
+}
```
-For complete examples, refer to [C++ client examples](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/examples).
+### Partitioned topics and lazy producers
-## Schema
+When scaling out a Pulsar topic, you may configure a topic to have hundreds of partitions. Likewise, you may have also scaled out your producers so there are hundreds or even thousands of producers. This can put some strain on the Pulsar brokers as when you create a producer on a partitioned topic, internally it creates one internal producer per partition which involves communications to the brokers for each one. So for a topic with 1000 partitions and 1000 producers, it ends up creating [...]
-This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started).
+You can reduce the load caused by this combination of a large number of partitions and many producers by doing the following:
+- use SinglePartition partition routing mode (this ensures that all messages are only sent to a single, randomly selected partition)
+- use non-keyed messages (when messages are keyed, routing is based on the hash of the key and so messages will end up being sent to multiple partitions)
+- use lazy producers (this ensures that an internal producer is only created on demand when a message needs to be routed to a partition)
-### Create producer with Avro schema
+With our example above, that reduces the number of internal producers spread out over the 1000 producer apps from 1,000,000 to just 1000.
-The following example shows how to create a producer with an Avro schema.
+Note that there can be extra latency for the first message sent. If you set a low send timeout, this timeout could be reached if the initial connection handshake is slow to complete.
-```cpp
+```c++
-static const std::string exampleSchema =
- "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
- "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
-Producer producer;
ProducerConfiguration producerConf;
-producerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
-client.createProducer("topic-avro", producerConf, producer);
+producerConf.setPartitionsRoutingMode(ProducerConfiguration::UseSinglePartition);
+producerConf.setLazyStartPartitionedProducers(true);
```
-### Create consumer with Avro schema
-
-The following example shows how to create a consumer with an Avro schema.
+## Enable authentication in connection URLs
+If you use TLS authentication when connecting to Pulsar, you need to add `ssl` in the connection URLs, and the default port is `6651`. The following is an example.
```cpp
-static const std::string exampleSchema =
- "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
- "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
-ConsumerConfiguration consumerConf;
-Consumer consumer;
-consumerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
-client.subscribe("topic-avro", "sub-2", consumerConf, consumer)
+ClientConfiguration config = ClientConfiguration();
+config.setUseTls(true);
+config.setTlsTrustCertsFilePath("/path/to/cacert.pem");
+config.setTlsAllowInsecureConnection(false);
+config.setAuth(pulsar::AuthTls::create(
+ "/path/to/client-cert.pem", "/path/to/client-key.pem"););
+
+Client client("pulsar+ssl://my-broker.com:6651", config);
```
+For complete examples, refer to [C++ client examples](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/examples).
+
+## Schema
+
+This section describes some examples about schema. For more information about
+schema, see [Pulsar schema](schema-get-started).
+
+### Avro schema
+
+- The following example shows how to create a producer with an Avro schema.
+
+ ```cpp
+
+ static const std::string exampleSchema =
+ "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
+ "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
+ Producer producer;
+ ProducerConfiguration producerConf;
+ producerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
+ client.createProducer("topic-avro", producerConf, producer);
+
+ ```
+
+- The following example shows how to create a consumer with an Avro schema.
+
+ ```cpp
+
+ static const std::string exampleSchema =
+ "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
+ "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
+ ConsumerConfiguration consumerConf;
+ Consumer consumer;
+ consumerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
+ client.subscribe("topic-avro", "sub-2", consumerConf, consumer)
+
+ ```
+
+### ProtobufNative schema
+
+The following example shows how to create a producer and a consumer with a ProtobufNative schema.
+
+1. Generate the `User` class using Protobuf3.
+
+ :::note
+
+ You need to use Protobuf3 or later versions.
+
+ :::
+
+
+
+ ```protobuf
+
+ syntax = "proto3";
+
+ message User {
+ string name = 1;
+ int32 age = 2;
+ }
+
+ ```
+
+
+2. Include the `ProtobufNativeSchema.h` in your source code. Ensure the Protobuf dependency has been added to your project.
+
+
+ ```c++
+
+ #include <pulsar/ProtobufNativeSchema.h>
+
+ ```
+
+
+3. Create a producer to send a `User` instance.
+
+
+ ```c++
+
+ ProducerConfiguration producerConf;
+ producerConf.setSchema(createProtobufNativeSchema(User::GetDescriptor()));
+ Producer producer;
+ client.createProducer("topic-protobuf", producerConf, producer);
+ User user;
+ user.set_name("my-name");
+ user.set_age(10);
+ std::string content;
+ user.SerializeToString(&content);
+ producer.send(MessageBuilder().setContent(content).build());
+
+ ```
+
+
+4. Create a consumer to receive a `User` instance.
+
+
+ ```c++
+
+ ConsumerConfiguration consumerConf;
+ consumerConf.setSchema(createProtobufNativeSchema(User::GetDescriptor()));
+ consumerConf.setSubscriptionInitialPosition(InitialPositionEarliest);
+ Consumer consumer;
+ client.subscribe("topic-protobuf", "my-sub", consumerConf, consumer);
+ Message msg;
+ consumer.receive(msg);
+ User user2;
+ user2.ParseFromArray(msg.getData(), msg.getLength());
+
+ ```
+
diff --git a/site2/website-next/versioned_docs/version-2.4.1/client-libraries-go.md b/site2/website-next/versioned_docs/version-2.4.1/client-libraries-go.md
index 7e797bb..22df463 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/client-libraries-go.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/client-libraries-go.md
@@ -2,7 +2,6 @@
id: client-libraries-go
title: Pulsar Go client
sidebar_label: "Go"
-original_id: client-libraries-go
---
> Tips: Currently, the CGo client will be deprecated, if you want to know more about the CGo client, please refer to [CGo client docs](client-libraries-cgo)
@@ -286,7 +285,8 @@ defer client.Close()
topicName := newTopicName()
producer, err := client.CreateProducer(pulsar.ProducerOptions{
- Topic: topicName,
+ Topic: topicName,
+ DisableBatching: true,
})
if err != nil {
log.Fatal(err)
@@ -330,6 +330,85 @@ canc()
```
+#### How to use Prometheus metrics in producer
+
+Pulsar Go client registers client metrics using Prometheus. This section demonstrates how to create a simple Pulsar producer application that exposes Prometheus metrics via HTTP.
+
+1. Write a simple producer application.
+
+```go
+
+// Create a Pulsar client
+client, err := pulsar.NewClient(pulsar.ClientOptions{
+ URL: "pulsar://localhost:6650",
+})
+if err != nil {
+ log.Fatal(err)
+}
+
+defer client.Close()
+
+// Start a separate goroutine for Prometheus metrics
+// In this case, Prometheus metrics can be accessed via http://localhost:2112/metrics
+go func() {
+ prometheusPort := 2112
+ log.Printf("Starting Prometheus metrics at http://localhost:%v/metrics\n", prometheusPort)
+ http.Handle("/metrics", promhttp.Handler())
+ err = http.ListenAndServe(":"+strconv.Itoa(prometheusPort), nil)
+ if err != nil {
+ log.Fatal(err)
+ }
+}()
+
+// Create a producer
+producer, err := client.CreateProducer(pulsar.ProducerOptions{
+ Topic: "topic-1",
+})
+if err != nil {
+ log.Fatal(err)
+}
+
+defer producer.Close()
+
+ctx := context.Background()
+
+// Write your business logic here
+// In this case, you build a simple Web server. You can produce messages by requesting http://localhost:8082/produce
+webPort := 8082
+http.HandleFunc("/produce", func(w http.ResponseWriter, r *http.Request) {
+ msgId, err := producer.Send(ctx, &pulsar.ProducerMessage{
+ Payload: []byte(fmt.Sprintf("hello world")),
+ })
+ if err != nil {
+ log.Fatal(err)
+ } else {
+ log.Printf("Published message: %v", msgId)
+ fmt.Fprintf(w, "Published message: %v", msgId)
+ }
+})
+
+err = http.ListenAndServe(":"+strconv.Itoa(webPort), nil)
+if err != nil {
+ log.Fatal(err)
+}
+
+```
+
+2. To scrape metrics from applications, configure a local running Prometheus instance using a configuration file (`prometheus.yml`).
+
+```yaml
+
+scrape_configs:
+- job_name: pulsar-client-go-metrics
+ scrape_interval: 10s
+ static_configs:
+ - targets:
+ - localhost:2112
+
+```
+
+Now you can query Pulsar client metrics on Prometheus.
+
### Producer configuration
Name | Description | Default
@@ -607,6 +686,85 @@ defer consumer.Close()
```
+#### How to use Prometheus metrics in consumer
+
+In this guide, This section demonstrates how to create a simple Pulsar consumer application that exposes Prometheus metrics via HTTP.
+1. Write a simple consumer application.
+
+```go
+
+// Create a Pulsar client
+client, err := pulsar.NewClient(pulsar.ClientOptions{
+ URL: "pulsar://localhost:6650",
+})
+if err != nil {
+ log.Fatal(err)
+}
+
+defer client.Close()
+
+// Start a separate goroutine for Prometheus metrics
+// In this case, Prometheus metrics can be accessed via http://localhost:2112/metrics
+go func() {
+ prometheusPort := 2112
+ log.Printf("Starting Prometheus metrics at http://localhost:%v/metrics\n", prometheusPort)
+ http.Handle("/metrics", promhttp.Handler())
+ err = http.ListenAndServe(":"+strconv.Itoa(prometheusPort), nil)
+ if err != nil {
+ log.Fatal(err)
+ }
+}()
+
+// Create a consumer
+consumer, err := client.Subscribe(pulsar.ConsumerOptions{
+ Topic: "topic-1",
+ SubscriptionName: "sub-1",
+ Type: pulsar.Shared,
+})
+if err != nil {
+ log.Fatal(err)
+}
+
+defer consumer.Close()
+
+ctx := context.Background()
+
+// Write your business logic here
+// In this case, you build a simple Web server. You can consume messages by requesting http://localhost:8083/consume
+webPort := 8083
+http.HandleFunc("/consume", func(w http.ResponseWriter, r *http.Request) {
+ msg, err := consumer.Receive(ctx)
+ if err != nil {
+ log.Fatal(err)
+ } else {
+ log.Printf("Received message msgId: %v -- content: '%s'\n", msg.ID(), string(msg.Payload()))
+ fmt.Fprintf(w, "Received message msgId: %v -- content: '%s'\n", msg.ID(), string(msg.Payload()))
+ consumer.Ack(msg)
+ }
+})
+
+err = http.ListenAndServe(":"+strconv.Itoa(webPort), nil)
+if err != nil {
+ log.Fatal(err)
+}
+
+```
+
+2. To scrape metrics from applications, configure a local running Prometheus instance using a configuration file (`prometheus.yml`).
+
+```yaml
+
+scrape_configs:
+- job_name: pulsar-client-go-metrics
+ scrape_interval: 10s
+ static_configs:
+ - targets:
+ - localhost:2112
+
+```
+
+Now you can query Pulsar client metrics on Prometheus.
+
### Consumer configuration
Name | Description | Default
diff --git a/site2/website-next/versioned_docs/version-2.4.1/client-libraries-websocket.md b/site2/website-next/versioned_docs/version-2.4.1/client-libraries-websocket.md
index bc13b43..c663f97 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/client-libraries-websocket.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/client-libraries-websocket.md
@@ -2,7 +2,6 @@
id: client-libraries-websocket
title: Pulsar WebSocket API
sidebar_label: "WebSocket"
-original_id: client-libraries-websocket
---
Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page.
@@ -190,7 +189,7 @@ Key | Type | Required? | Explanation
`maxRedeliverCount` | int | no | Define a [maxRedeliverCount](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html#deadLetterPolicy-org.apache.pulsar.client.api.DeadLetterPolicy-) for the consumer (default: 0). Activates [Dead Letter Topic](https://github.com/apache/pulsar/wiki/PIP-22%3A-Pulsar-Dead-Letter-Topic) feature.
`deadLetterTopic` | string | no | Define a [deadLetterTopic](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html#deadLetterPolicy-org.apache.pulsar.client.api.DeadLetterPolicy-) for the consumer (default: {topic}-{subscription}-DLQ). Activates [Dead Letter Topic](https://github.com/apache/pulsar/wiki/PIP-22%3A-Pulsar-Dead-Letter-Topic) feature.
`pullMode` | boolean | no | Enable pull mode (default: false). See "Flow Control" below.
-`negativeAckRedeliveryDelay` | int | no | When a message is negatively acknowledged, it will be redelivered to the DLQ.
+`negativeAckRedeliveryDelay` | int | no | When a message is negatively acknowledged, the delay time before the message is redelivered (in milliseconds). The default value is 60000.
`token` | string | no | Authentication token, this is used for the browser javascript client
NB: these parameter (except `pullMode`) apply to the internal consumer of the WebSocket service.
@@ -204,23 +203,60 @@ Server will push messages on the WebSocket session:
```json
{
- "messageId": "CAAQAw==",
- "payload": "SGVsbG8gV29ybGQ=",
- "properties": {"key1": "value1", "key2": "value2"},
- "publishTime": "2016-08-30 16:45:57.785",
- "redeliveryCount": 4
+ "messageId": "CAMQADAA",
+ "payload": "hvXcJvHW7kOSrUn17P2q71RA5SdiXwZBqw==",
+ "properties": {},
+ "publishTime": "2021-10-29T16:01:38.967-07:00",
+ "redeliveryCount": 0,
+ "encryptionContext": {
+ "keys": {
+ "client-rsa.pem": {
+ "keyValue": "jEuwS+PeUzmCo7IfLNxqoj4h7txbLjCQjkwpaw5AWJfZ2xoIdMkOuWDkOsqgFmWwxiecakS6GOZHs94x3sxzKHQx9Oe1jpwBg2e7L4fd26pp+WmAiLm/ArZJo6JotTeFSvKO3u/yQtGTZojDDQxiqFOQ1ZbMdtMZA8DpSMuq+Zx7PqLo43UdW1+krjQfE5WD+y+qE3LJQfwyVDnXxoRtqWLpVsAROlN2LxaMbaftv5HckoejJoB4xpf/dPOUqhnRstwQHf6klKT5iNhjsY4usACt78uILT0pEPd14h8wEBidBz/vAlC/zVMEqiDVzgNS7dqEYS4iHbf7cnWVCn3Hxw==",
+ "metadata": {}
+ }
+ },
+ "param": "Tfu1PxVm6S9D3+Hk",
+ "compressionType": "NONE",
+ "uncompressedMessageSize": 0,
+ "batchSize": {
+ "empty": false,
+ "present": true
+ }
+ }
}
```
-Key | Type | Required? | Explanation
-:---|:-----|:----------|:-----------
-`messageId` | string | yes | Message ID
-`payload` | string | yes | Base-64 encoded payload
-`publishTime` | string | yes | Publish timestamp
-`redeliveryCount` | number | yes | Number of times this message was already delivered
-`properties` | key-value pairs | no | Application-defined properties
-`key` | string | no | Original routing key set by producer
+Below are the parameters in the WebSocket consumer response.
+
+- General parameters
+
+ Key | Type | Required? | Explanation
+ :---|:-----|:----------|:-----------
+ `messageId` | string | yes | Message ID
+ `payload` | string | yes | Base-64 encoded payload
+ `publishTime` | string | yes | Publish timestamp
+ `redeliveryCount` | number | yes | Number of times this message was already delivered
+ `properties` | key-value pairs | no | Application-defined properties
+ `key` | string | no | Original routing key set by producer
+ `encryptionContext` | EncryptionContext | no | Encryption context that consumers can use to decrypt received messages
+ `param` | string | no | Initialization vector for cipher (Base64 encoding)
+ `batchSize` | string | no | Number of entries in a message (if it is a batch message)
+ `uncompressedMessageSize` | string | no | Message size before compression
+ `compressionType` | string | no | Algorithm used to compress the message payload
+
+- `encryptionContext` related parameter
+
+ Key | Type | Required? | Explanation
+ :---|:-----|:----------|:-----------
+ `keys` |key-EncryptionKey pairs | yes | Key in `key-EncryptionKey` pairs is an encryption key name. Value in `key-EncryptionKey` pairs is an encryption key object.
+
+- `encryptionKey` related parameters
+
+ Key | Type | Required? | Explanation
+ :---|:-----|:----------|:-----------
+ `keyValue` | string | yes | Encryption key (Base64 encoding)
+ `metadata` | key-value pairs | no | Application-defined metadata
#### Acknowledging the message
@@ -454,9 +490,15 @@ TOPIC = scheme + '://localhost:8080/ws/v2/producer/persistent/public/default/my-
ws = websocket.create_connection(TOPIC)
+# encode message
+s = "Hello World"
+firstEncoded = s.encode("UTF-8")
+binaryEncoded = base64.b64encode(firstEncoded)
+payloadString = binaryEncoded.decode('UTF-8')
+
# Send one message as JSON
ws.send(json.dumps({
- 'payload' : base64.b64encode('Hello World'),
+ 'payload' : payloadString,
'properties': {
'key1' : 'value1',
'key2' : 'value2'
@@ -466,9 +508,9 @@ ws.send(json.dumps({
response = json.loads(ws.recv())
if response['result'] == 'ok':
- print 'Message published successfully'
+ print( 'Message published successfully')
else:
- print 'Failed to publish message:', response
+ print('Failed to publish message:', response)
ws.close()
```
@@ -495,7 +537,7 @@ while True:
msg = json.loads(ws.recv())
if not msg: break
- print "Received: {} - payload: {}".format(msg, base64.b64decode(msg['payload']))
+ print( "Received: {} - payload: {}".format(msg, base64.b64decode(msg['payload'])))
# Acknowledge successful processing
ws.send(json.dumps({'messageId' : msg['messageId']}))
@@ -525,7 +567,7 @@ while True:
msg = json.loads(ws.recv())
if not msg: break
- print "Received: {} - payload: {}".format(msg, base64.b64decode(msg['payload']))
+ print ( "Received: {} - payload: {}".format(msg, base64.b64decode(msg['payload'])))
# Acknowledge successful processing
ws.send(json.dumps({'messageId' : msg['messageId']}))
diff --git a/site2/website-next/versioned_docs/version-2.4.1/client-libraries.md b/site2/website-next/versioned_docs/version-2.4.1/client-libraries.md
index 23e5a06..ab5b7c4 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/client-libraries.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/client-libraries.md
@@ -2,7 +2,6 @@
id: client-libraries
title: Pulsar client libraries
sidebar_label: "Overview"
-original_id: client-libraries
---
Pulsar supports the following client libraries:
@@ -16,7 +15,7 @@ Pulsar supports the following client libraries:
- [C# client](client-libraries-dotnet)
## Feature matrix
-Pulsar client feature matrix for different languages is listed on [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page.
+Pulsar client feature matrix for different languages is listed on [Pulsar Feature Matrix (Client and Function)](https://github.com/apache/pulsar/wiki/PIP-108%3A-Pulsar-Feature-Matrix-%28Client-and-Function%29) page.
## Third-party clients
@@ -33,3 +32,4 @@ Besides the official released clients, multiple projects on developing Pulsar cl
| Scala | [pulsar4s](https://github.com/sksamuel/pulsar4s) | [sksamuel](https://github.com/sksamuel) | [](https://opensource.org/licenses/Apache-2.0) | Idomatic, typesafe, and reactive Scala client for Apache Pulsar |
| Rust | [pulsar-rs](https://github.com/wyyerd/pulsar-rs) | [Wyyerd Group](https://github.com/wyyerd) | [](https://opensource.org/licenses/Apache-2.0) | Future-based Rust bindings for Apache Pulsar |
| .NET | [pulsar-client-dotnet](https://github.com/fsharplang-ru/pulsar-client-dotnet) | [Lanayx](https://github.com/Lanayx) | [](https://opensource.org/licenses/MIT) | Native .NET client for C#/F#/VB |
+| Node.js | [pulsar-flex](https://github.com/ayeo-flex-org/pulsar-flex) | [Daniel Sinai](https://github.com/danielsinai), [Ron Farkash](https://github.com/ronfarkash), [Gal Rosenberg](https://github.com/galrose)| [](https://opensource.org/licenses/MIT) | Native Nodejs client |
diff --git a/site2/website-next/versioned_docs/version-2.4.1/concepts-architecture-overview.md b/site2/website-next/versioned_docs/version-2.4.1/concepts-architecture-overview.md
index 6a501d2..8fe0717 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/concepts-architecture-overview.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/concepts-architecture-overview.md
@@ -2,7 +2,6 @@
id: concepts-architecture-overview
title: Architecture Overview
sidebar_label: "Architecture"
-original_id: concepts-architecture-overview
---
At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves.
@@ -146,7 +145,7 @@ Some important things to know about the Pulsar proxy:
## Service discovery
-[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide.
+[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL.
You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means.
diff --git a/site2/website-next/versioned_docs/version-2.4.1/concepts-authentication.md b/site2/website-next/versioned_docs/version-2.4.1/concepts-authentication.md
index b375ecb..335da8d 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/concepts-authentication.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/concepts-authentication.md
@@ -2,7 +2,6 @@
id: concepts-authentication
title: Authentication and Authorization
sidebar_label: "Authentication and Authorization"
-original_id: concepts-authentication
---
Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants.
diff --git a/site2/website-next/versioned_docs/version-2.4.1/concepts-clients.md b/site2/website-next/versioned_docs/version-2.4.1/concepts-clients.md
index b68f76a..65201b5 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/concepts-clients.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/concepts-clients.md
@@ -2,7 +2,6 @@
id: concepts-clients
title: Pulsar Clients
sidebar_label: "Clients"
-original_id: concepts-clients
---
Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications.
diff --git a/site2/website-next/versioned_docs/version-2.4.1/concepts-multi-tenancy.md b/site2/website-next/versioned_docs/version-2.4.1/concepts-multi-tenancy.md
index be752cc..8a17e72 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/concepts-multi-tenancy.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/concepts-multi-tenancy.md
@@ -2,7 +2,6 @@
id: concepts-multi-tenancy
title: Multi Tenancy
sidebar_label: "Multi Tenancy"
-original_id: concepts-multi-tenancy
---
Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed.
diff --git a/site2/website-next/versioned_docs/version-2.4.1/concepts-overview.md b/site2/website-next/versioned_docs/version-2.4.1/concepts-overview.md
index b903fa4..c76032c 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/concepts-overview.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/concepts-overview.md
@@ -2,7 +2,6 @@
id: concepts-overview
title: Pulsar Overview
sidebar_label: "Overview"
-original_id: concepts-overview
---
Pulsar is a multi-tenant, high-performance solution for server-to-server messaging. Pulsar was originally developed by Yahoo, it is under the stewardship of the [Apache Software Foundation](https://www.apache.org/).
diff --git a/site2/website-next/versioned_docs/version-2.4.1/concepts-replication.md b/site2/website-next/versioned_docs/version-2.4.1/concepts-replication.md
index 6e23962..11677cc 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/concepts-replication.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/concepts-replication.md
@@ -2,7 +2,6 @@
id: concepts-replication
title: Geo Replication
sidebar_label: "Geo Replication"
-original_id: concepts-replication
---
Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo) in Pulsar enables you to do that.
diff --git a/site2/website-next/versioned_docs/version-2.4.1/concepts-topic-compaction.md b/site2/website-next/versioned_docs/version-2.4.1/concepts-topic-compaction.md
index c85e703..3356298 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/concepts-topic-compaction.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/concepts-topic-compaction.md
@@ -2,7 +2,6 @@
id: concepts-topic-compaction
title: Topic Compaction
sidebar_label: "Topic Compaction"
-original_id: concepts-topic-compaction
---
Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time int [...]
diff --git a/site2/website-next/versioned_docs/version-2.4.1/cookbooks-bookkeepermetadata.md b/site2/website-next/versioned_docs/version-2.4.1/cookbooks-bookkeepermetadata.md
index b0fa98d..187cb65 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/cookbooks-bookkeepermetadata.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/cookbooks-bookkeepermetadata.md
@@ -1,7 +1,6 @@
---
id: cookbooks-bookkeepermetadata
title: BookKeeper Ledger Metadata
-original_id: cookbooks-bookkeepermetadata
---
Pulsar stores data on BookKeeper ledgers, you can understand the contents of a ledger by inspecting the metadata attached to the ledger.
diff --git a/site2/website-next/versioned_docs/version-2.4.1/cookbooks-deduplication.md b/site2/website-next/versioned_docs/version-2.4.1/cookbooks-deduplication.md
index 1669afa..307fe03 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/cookbooks-deduplication.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/cookbooks-deduplication.md
@@ -1,8 +1,7 @@
---
id: cookbooks-deduplication
title: Message deduplication
-sidebar_label: "Message deduplication"
-original_id: cookbooks-deduplication
+sidebar_label: "Message deduplication "
---
import Tabs from '@theme/Tabs';
diff --git a/site2/website-next/versioned_docs/version-2.4.1/cookbooks-encryption.md b/site2/website-next/versioned_docs/version-2.4.1/cookbooks-encryption.md
index f0d8fb8..fbd1c97 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/cookbooks-encryption.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/cookbooks-encryption.md
@@ -1,8 +1,7 @@
---
id: cookbooks-encryption
title: Pulsar Encryption
-sidebar_label: "Encryption"
-original_id: cookbooks-encryption
+sidebar_label: "Encryption "
---
Pulsar encryption allows applications to encrypt messages at the producer and decrypt at the consumer. Encryption is performed using the public/private key pair configured by the application. Encrypted messages can only be decrypted by consumers with a valid key.
diff --git a/site2/website-next/versioned_docs/version-2.4.1/cookbooks-message-queue.md b/site2/website-next/versioned_docs/version-2.4.1/cookbooks-message-queue.md
index eb43cbd..9b93a94 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/cookbooks-message-queue.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/cookbooks-message-queue.md
@@ -2,7 +2,6 @@
id: cookbooks-message-queue
title: Using Pulsar as a message queue
sidebar_label: "Message queue"
-original_id: cookbooks-message-queue
---
Message queues are essential components of many large-scale data architectures. If every single work object that passes through your system absolutely *must* be processed in spite of the slowness or downright failure of this or that system component, there's a good chance that you'll need a message queue to step in and ensure that unprocessed data is retained---with correct ordering---until the required actions are taken.
diff --git a/site2/website-next/versioned_docs/version-2.4.1/cookbooks-non-persistent.md b/site2/website-next/versioned_docs/version-2.4.1/cookbooks-non-persistent.md
index 391569a..d40c4fb 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/cookbooks-non-persistent.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/cookbooks-non-persistent.md
@@ -2,7 +2,6 @@
id: cookbooks-non-persistent
title: Non-persistent messaging
sidebar_label: "Non-persistent messaging"
-original_id: cookbooks-non-persistent
---
**Non-persistent topics** are Pulsar topics in which message data is *never* [persistently stored](concepts-architecture-overview.md#persistent-storage) and kept only in memory. This cookbook provides:
diff --git a/site2/website-next/versioned_docs/version-2.4.1/cookbooks-partitioned.md b/site2/website-next/versioned_docs/version-2.4.1/cookbooks-partitioned.md
index 7882fb9..2589693 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/cookbooks-partitioned.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/cookbooks-partitioned.md
@@ -2,6 +2,5 @@
id: cookbooks-partitioned
title: Partitioned topics
sidebar_label: "Partitioned Topics"
-original_id: cookbooks-partitioned
---
For details of the content, refer to [manage topics](admin-api-topics).
\ No newline at end of file
diff --git a/site2/website-next/versioned_docs/version-2.4.1/cookbooks-retention-expiry.md b/site2/website-next/versioned_docs/version-2.4.1/cookbooks-retention-expiry.md
index b9353b5..738cf42 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/cookbooks-retention-expiry.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/cookbooks-retention-expiry.md
@@ -2,7 +2,6 @@
id: cookbooks-retention-expiry
title: Message retention and expiry
sidebar_label: "Message retention and expiry"
-original_id: cookbooks-retention-expiry
---
import Tabs from '@theme/Tabs';
@@ -36,7 +35,7 @@ By default, when a Pulsar message arrives at a broker, the message is stored unt
Retention policies are useful when you use the Reader interface. The Reader interface does not use acknowledgements, and messages do not exist within backlogs. It is required to configure retention for Reader-only use cases.
-When you set a retention policy on topics in a namespace, you must set **both** a *size limit* and a *time limit*. You can refer to the following table to set retention policies in `pulsar-admin` and Java.
+When you set a retention policy on topics in a namespace, you must set **both** a *size limit* (via `defaultRetentionSizeInMB`) and a *time limit* (via `defaultRetentionTimeInMinutes`) . You can refer to the following table to set retention policies in `pulsar-admin` and Java.
|Time limit|Size limit| Message retention |
|----------|----------|------------------------|
@@ -152,7 +151,10 @@ admin.namespaces().setRetention(namespace, policies);
You can fetch the retention policy for a namespace by specifying the namespace. The output will be a JSON object with two keys: `retentionTimeInMinutes` and `retentionSizeInMB`.
-#### pulsar-admin
+<Tabs
+ defaultValue="pulsar-admin"
+ values={[{"label":"pulsar-admin","value":"pulsar-admin"},{"label":"REST API","value":"REST API"},{"label":"Java","value":"Java"}]}>
+<TabItem value="pulsar-admin">
Use the [`get-retention`](reference-pulsar-admin.md#namespaces) subcommand and specify the namespace.
@@ -168,11 +170,13 @@ $ pulsar-admin namespaces get-retention my-tenant/my-ns
```
-#### REST API
+</TabItem>
+<TabItem value="REST API">
{@inject: endpoint|GET|/admin/v2/namespaces/:tenant/:namespace/retention|operation/getRetention?version=@pulsar:version_number@}
-#### Java
+</TabItem>
+<TabItem value="Java">
```java
@@ -180,15 +184,17 @@ admin.namespaces().getRetention(namespace);
```
+</TabItem>
+
+</Tabs>
+
## Backlog quotas
*Backlogs* are sets of unacknowledged messages for a topic that have been stored by bookies. Pulsar stores all unacknowledged messages in backlogs until they are processed and acknowledged.
-You can control the allowable size of backlogs, at the namespace level, using *backlog quotas*. Setting a backlog quota involves setting:
+You can control the allowable size and/or time of backlogs, at the namespace level, using *backlog quotas*. Setting a backlog quota involves setting:
-TODO: Expand on is this per backlog or per topic?
-
-* an allowable *size threshold* for each topic in the namespace
+* an allowable *size and/or time threshold* for each topic in the namespace
* a *retention policy* that determines which action the [broker](reference-terminology.md#broker) takes if the threshold is exceeded.
The following retention policies are available:
@@ -210,9 +216,12 @@ Backlog quotas are handled at the namespace level. They can be managed via:
You can set a size and/or time threshold and backlog retention policy for all of the topics in a [namespace](reference-terminology.md#namespace) by specifying the namespace, a size limit and/or a time limit in second, and a policy by name.
-#### pulsar-admin
+<Tabs
+ defaultValue="pulsar-admin"
+ values={[{"label":"pulsar-admin","value":"pulsar-admin"},{"label":"REST API","value":"REST API"},{"label":"Java","value":"Java"}]}>
+<TabItem value="pulsar-admin">
-Use the [`set-backlog-quota`](reference-pulsar-admin.md#namespaces) subcommand and specify a namespace, a size limit using the `-l`/`--limit` flag, and a retention policy using the `-p`/`--policy` flag.
+Use the [`set-backlog-quota`](reference-pulsar-admin.md#namespaces) subcommand and specify a namespace, a size limit using the `-l`/`--limit` , `-lt`/`--limitTime` flag to limit backlog, a retention policy using the `-p`/`--policy` flag and a policy type using `-t`/`--type` (default is destination_storage).
##### Example
@@ -220,16 +229,26 @@ Use the [`set-backlog-quota`](reference-pulsar-admin.md#namespaces) subcommand a
$ pulsar-admin namespaces set-backlog-quota my-tenant/my-ns \
--limit 2G \
- --limitTime 36000 \
--policy producer_request_hold
```
-#### REST API
+```shell
+
+$ pulsar-admin namespaces set-backlog-quota my-tenant/my-ns/my-topic \
+--limitTime 3600 \
+--policy producer_request_hold \
+--type message_age
+
+```
+
+</TabItem>
+<TabItem value="REST API">
{@inject: endpoint|POST|/admin/v2/namespaces/:tenant/:namespace/backlogQuota|operation/getBacklogQuotaMap?version=@pulsar:version_number@}
-#### Java
+</TabItem>
+<TabItem value="Java">
```java
@@ -240,11 +259,18 @@ admin.namespaces().setBacklogQuota(namespace, quota);
```
+</TabItem>
+
+</Tabs>
+
### Get backlog threshold and backlog retention policy
You can see which size threshold and backlog retention policy has been applied to a namespace.
-#### pulsar-admin
+<Tabs
+ defaultValue="pulsar-admin"
+ values={[{"label":"pulsar-admin","value":"pulsar-admin"},{"label":"REST API","value":"REST API"},{"label":"Java","value":"Java"}]}>
+<TabItem value="pulsar-admin">
Use the [`get-backlog-quotas`](reference-pulsar-admin.md#pulsar-admin-namespaces-get-backlog-quotas) subcommand and specify a namespace. Here's an example:
@@ -260,11 +286,13 @@ $ pulsar-admin namespaces get-backlog-quotas my-tenant/my-ns
```
-#### REST API
+</TabItem>
+<TabItem value="REST API">
{@inject: endpoint|GET|/admin/v2/namespaces/:tenant/:namespace/backlogQuotaMap|operation/getBacklogQuotaMap?version=@pulsar:version_number@}
-#### Java
+</TabItem>
+<TabItem value="Java">
```java
@@ -273,11 +301,18 @@ Map<BacklogQuota.BacklogQuotaType,BacklogQuota> quotas =
```
+</TabItem>
+
+</Tabs>
+
### Remove backlog quotas
-#### pulsar-admin
+<Tabs
+ defaultValue="pulsar-admin"
+ values={[{"label":"pulsar-admin","value":"pulsar-admin"},{"label":"REST API","value":"REST API"},{"label":"Java","value":"Java"}]}>
+<TabItem value="pulsar-admin">
-Use the [`remove-backlog-quota`](reference-pulsar-admin.md#pulsar-admin-namespaces-remove-backlog-quota) subcommand and specify a namespace. Here's an example:
+Use the [`remove-backlog-quota`](reference-pulsar-admin.md#pulsar-admin-namespaces-remove-backlog-quota) subcommand and specify a namespace, use `t`/`--type` to specify backlog type to remove(default is destination_storage). Here's an example:
```shell
@@ -285,11 +320,13 @@ $ pulsar-admin namespaces remove-backlog-quota my-tenant/my-ns
```
-#### REST API
+</TabItem>
+<TabItem value="REST API">
{@inject: endpoint|DELETE|/admin/v2/namespaces/:tenant/:namespace/backlogQuota|operation/removeBacklogQuota?version=@pulsar:version_number@}
-#### Java
+</TabItem>
+<TabItem value="Java">
```java
@@ -297,6 +334,10 @@ admin.namespaces().removeBacklogQuota(namespace);
```
+</TabItem>
+
+</Tabs>
+
### Clear backlog
#### pulsar-admin
@@ -319,7 +360,10 @@ By default, Pulsar stores all unacknowledged messages forever. This can lead to
### Set the TTL for a namespace
-#### pulsar-admin
+<Tabs
+ defaultValue="pulsar-admin"
+ values={[{"label":"pulsar-admin","value":"pulsar-admin"},{"label":"REST API","value":"REST API"},{"label":"Java","value":"Java"}]}>
+<TabItem value="pulsar-admin">
Use the [`set-message-ttl`](reference-pulsar-admin.md#pulsar-admin-namespaces-set-message-ttl) subcommand and specify a namespace and a TTL (in seconds) using the `-ttl`/`--messageTTL` flag.
@@ -332,11 +376,13 @@ $ pulsar-admin namespaces set-message-ttl my-tenant/my-ns \
```
-#### REST API
+</TabItem>
+<TabItem value="REST API">
{@inject: endpoint|POST|/admin/v2/namespaces/:tenant/:namespace/messageTTL|operation/setNamespaceMessageTTL?version=@pulsar:version_number@}
-#### Java
+</TabItem>
+<TabItem value="Java">
```java
@@ -344,9 +390,16 @@ admin.namespaces().setNamespaceMessageTTL(namespace, ttlInSeconds);
```
+</TabItem>
+
+</Tabs>
+
### Get the TTL configuration for a namespace
-#### pulsar-admin
+<Tabs
+ defaultValue="pulsar-admin"
+ values={[{"label":"pulsar-admin","value":"pulsar-admin"},{"label":"REST API","value":"REST API"},{"label":"Java","value":"Java"}]}>
+<TabItem value="pulsar-admin">
Use the [`get-message-ttl`](reference-pulsar-admin.md#pulsar-admin-namespaces-get-message-ttl) subcommand and specify a namespace.
@@ -359,11 +412,13 @@ $ pulsar-admin namespaces get-message-ttl my-tenant/my-ns
```
-#### REST API
+</TabItem>
+<TabItem value="REST API">
{@inject: endpoint|GET|/admin/v2/namespaces/:tenant/:namespace/messageTTL|operation/getNamespaceMessageTTL?version=@pulsar:version_number@}
-#### Java
+</TabItem>
+<TabItem value="Java">
```java
@@ -371,9 +426,16 @@ admin.namespaces().getNamespaceMessageTTL(namespace)
```
+</TabItem>
+
+</Tabs>
+
### Remove the TTL configuration for a namespace
-#### pulsar-admin
+<Tabs
+ defaultValue="pulsar-admin"
+ values={[{"label":"pulsar-admin","value":"pulsar-admin"},{"label":"REST API","value":"REST API"},{"label":"Java","value":"Java"}]}>
+<TabItem value="pulsar-admin">
Use the [`remove-message-ttl`](reference-pulsar-admin.md#pulsar-admin-namespaces-remove-message-ttl) subcommand and specify a namespace.
@@ -385,11 +447,13 @@ $ pulsar-admin namespaces remove-message-ttl my-tenant/my-ns
```
-#### REST API
+</TabItem>
+<TabItem value="REST API">
{@inject: endpoint|DELETE|/admin/v2/namespaces/:tenant/:namespace/messageTTL|operation/removeNamespaceMessageTTL?version=@pulsar:version_number@}
-#### Java
+</TabItem>
+<TabItem value="Java">
```java
@@ -397,6 +461,10 @@ admin.namespaces().removeNamespaceMessageTTL(namespace)
```
+</TabItem>
+
+</Tabs>
+
## Delete messages from namespaces
If you do not have any retention period and that you never have much of a backlog, the upper limit for retaining messages, which are acknowledged, equals to the Pulsar segment rollover period + entry log rollover period + (garbage collection interval * garbage collection ratios).
diff --git a/site2/website-next/versioned_docs/version-2.4.1/cookbooks-tiered-storage.md b/site2/website-next/versioned_docs/version-2.4.1/cookbooks-tiered-storage.md
index 8f6a7fb..f2ea50d 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/cookbooks-tiered-storage.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/cookbooks-tiered-storage.md
@@ -2,14 +2,15 @@
id: cookbooks-tiered-storage
title: Tiered Storage
sidebar_label: "Tiered Storage"
-original_id: cookbooks-tiered-storage
---
Pulsar's **Tiered Storage** feature allows older backlog data to be offloaded to long term storage, thereby freeing up space in BookKeeper and reducing storage costs. This cookbook walks you through using tiered storage in your Pulsar cluster.
-* Tiered storage uses [Apache jclouds](https://jclouds.apache.org) to support [Amazon S3](https://aws.amazon.com/s3/) and [Google Cloud Storage](https://cloud.google.com/storage/)(GCS for short) for long term storage. With Jclouds, it is easy to add support for more [cloud storage providers](https://jclouds.apache.org/reference/providers/#blobstore-providers) in the future.
+* Tiered storage uses [Apache jclouds](https://jclouds.apache.org) to support [Amazon S3](https://aws.amazon.com/s3/) and [Google Cloud Storage](https://cloud.google.com/storage/)(GCS for short)
+for long term storage. With Jclouds, it is easy to add support for more [cloud storage providers](https://jclouds.apache.org/reference/providers/#blobstore-providers) in the future.
-* Tiered storage uses [Apache Hadoop](http://hadoop.apache.org/) to support filesystem for long term storage. With Hadoop, it is easy to add support for more filesystem in the future.
+* Tiered storage uses [Apache Hadoop](http://hadoop.apache.org/) to support filesystem for long term storage.
+With Hadoop, it is easy to add support for more filesystem in the future.
## When should I use Tiered Storage?
diff --git a/site2/website-next/versioned_docs/version-2.4.1/deploy-aws.md b/site2/website-next/versioned_docs/version-2.4.1/deploy-aws.md
index 6323051..2034749 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/deploy-aws.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/deploy-aws.md
@@ -2,7 +2,6 @@
id: deploy-aws
title: Deploying a Pulsar cluster on AWS using Terraform and Ansible
sidebar_label: "Amazon Web Services"
-original_id: deploy-aws
---
> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster).
@@ -148,7 +147,7 @@ Variable name | Description | Default
When you run the Ansible playbook, the following AWS resources are used:
* 9 total [Elastic Compute Cloud](https://aws.amazon.com/ec2) (EC2) instances running the [ami-9fa343e7](https://access.redhat.com/articles/3135091) Amazon Machine Image (AMI), which runs [Red Hat Enterprise Linux (RHEL) 7.4](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/7.4_release_notes/index). By default, that includes:
- * 3 small VMs for ZooKeeper ([t2.small](https://www.ec2instances.info/?selected=t2.small) instances)
+ * 3 small VMs for ZooKeeper ([t3.small](https://www.ec2instances.info/?selected=t3.small) instances)
* 3 larger VMs for BookKeeper [bookies](reference-terminology.md#bookie) ([i3.xlarge](https://www.ec2instances.info/?selected=i3.xlarge) instances)
* 2 larger VMs for Pulsar [brokers](reference-terminology.md#broker) ([c5.2xlarge](https://www.ec2instances.info/?selected=c5.2xlarge) instances)
* 1 larger VMs for Pulsar [proxy](reference-terminology.md#proxy) ([c5.2xlarge](https://www.ec2instances.info/?selected=c5.2xlarge) instances)
diff --git a/site2/website-next/versioned_docs/version-2.4.1/deploy-bare-metal-multi-cluster.md b/site2/website-next/versioned_docs/version-2.4.1/deploy-bare-metal-multi-cluster.md
index 643c122..9dd2526 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/deploy-bare-metal-multi-cluster.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/deploy-bare-metal-multi-cluster.md
@@ -2,38 +2,30 @@
id: deploy-bare-metal-multi-cluster
title: Deploying a multi-cluster on bare metal
sidebar_label: "Bare metal multi-cluster"
-original_id: deploy-bare-metal-multi-cluster
---
:::tip
-1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with
-Pulsar or using it in a startup or on a single team, you had better opt for a single cluster. For instructions on deploying a single cluster,
-see the guide [here](deploy-bare-metal).
-2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors`
-package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you
-run a separate cluster of function workers for [Pulsar Functions](functions-overview).
-3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`
-package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure
-this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage).
+1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster).
+2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview).
+3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage).
:::
-A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo). Deploying a multi-cluster Pulsar instance involves the following basic steps:
+A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo).Deploying a multi-cluster Pulsar instance consists of the following steps:
-* Deploying two separate [ZooKeeper](#deploy-zookeeper) quorums: a [local](#deploy-local-zookeeper) quorum for each cluster in the instance and a [configuration store](#configuration-store) quorum for instance-wide tasks
-* Initializing [cluster metadata](#cluster-metadata-initialization) for each cluster
-* Deploying a [BookKeeper cluster](#deploy-bookkeeper) of bookies in each Pulsar cluster
-* Deploying [brokers](#deploy-brokers) in each Pulsar cluster
+1. Deploying two separate ZooKeeper quorums: a local quorum for each cluster in the instance and a configuration store quorum for instance-wide tasks
+2. Initializing cluster metadata for each cluster
+3. Deploying a BookKeeper cluster of bookies in each Pulsar cluster
+4. Deploying brokers in each Pulsar cluster
-If you want to deploy a single Pulsar cluster, see [Clusters and Brokers](getting-started-standalone.md#start-the-cluster).
> #### Run Pulsar locally or on Kubernetes?
-> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar [...]
+> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services.
## System requirement
-Currently, Pulsar is available for 64-bit **macOS**, **Linux**, and **Windows**. To use Pulsar, you need to install 64-bit JRE/JDK 8 or later versions.
+Currently, Pulsar is available for 64-bit **macOS**, **Linux**, and **Windows**. You need to install 64-bit JRE/JDK 8 or later versions.
:::note
@@ -68,8 +60,6 @@ $ cd apache-pulsar-@pulsar:version@
```
-## What your package contains
-
The Pulsar binary package initially contains the following directories:
Directory | Contains
@@ -93,17 +83,17 @@ Directory | Contains
Each Pulsar instance relies on two separate ZooKeeper quorums.
-* [Local ZooKeeper](#deploy-local-zookeeper) operates at the cluster level and provides cluster-specific configuration management and coordination. Each Pulsar cluster needs to have a dedicated ZooKeeper cluster.
-* [Configuration Store](#deploy-the-configuration-store) operates at the instance level and provides configuration management for the entire system (and thus across clusters). An independent cluster of machines or the same machines that local ZooKeeper uses can provide the configuration store quorum.
+* Local ZooKeeper operates at the cluster level and provides cluster-specific configuration management and coordination. Each Pulsar cluster needs a dedicated ZooKeeper cluster.
+* Configuration Store operates at the instance level and provides configuration management for the entire system (and thus across clusters). An independent cluster of machines or the same machines that local ZooKeeper uses can provide the configuration store quorum.
-The configuration store quorum can be provided by an independent cluster of machines or by the same machines used by local ZooKeeper.
+You can use an independent cluster of machines or the same machines used by local ZooKeeper to provide the configuration store quorum.
### Deploy local ZooKeeper
ZooKeeper manages a variety of essential coordination-related and configuration-related tasks for Pulsar.
-You need to stand up one local ZooKeeper cluster *per Pulsar cluster* for deploying a Pulsar instance.
+You need to stand up one local ZooKeeper cluster per Pulsar cluster for deploying a Pulsar instance.
To begin, add all ZooKeeper servers to the quorum configuration specified in the [`conf/zookeeper.conf`](reference-configuration.md#zookeeper) file. Add a `server.N` line for each node in the cluster to the configuration, where `N` is the number of the ZooKeeper node. The following is an example for a three-node cluster:
@@ -117,7 +107,11 @@ server.3=zk3.us-west.example.com:2888:3888
On each host, you need to specify the ID of the node in the `myid` file of each node, which is in `data/zookeeper` folder of each server by default (you can change the file location via the [`dataDir`](reference-configuration.md#zookeeper-dataDir) parameter).
-> See the [Multi-server setup guide](https://zookeeper.apache.org/doc/r3.4.10/zookeeperAdmin.html#sc_zkMulitServerSetup) in the ZooKeeper documentation for detailed information on `myid` and more.
+:::tip
+
+See the [Multi-server setup guide](https://zookeeper.apache.org/doc/r3.4.10/zookeeperAdmin.html#sc_zkMulitServerSetup) in the ZooKeeper documentation for detailed information on `myid` and more.
+
+:::
On a ZooKeeper server at `zk1.us-west.example.com`, for example, you could set the `myid` value like this:
@@ -140,15 +134,15 @@ $ bin/pulsar-daemon start zookeeper
### Deploy the configuration store
-The ZooKeeper cluster that is configured and started up in the section above is a *local* ZooKeeper cluster that you can use to manage a single Pulsar cluster. In addition to a local cluster, however, a full Pulsar instance also requires a configuration store for handling some instance-level configuration and coordination tasks.
+The ZooKeeper cluster configured and started up in the section above is a local ZooKeeper cluster that you can use to manage a single Pulsar cluster. In addition to a local cluster, however, a full Pulsar instance also requires a configuration store for handling some instance-level configuration and coordination tasks.
-If you deploy a [single-cluster](#single-cluster-pulsar-instance) instance, you do not need a separate cluster for the configuration store. If, however, you deploy a [multi-cluster](#multi-cluster-pulsar-instance) instance, you should stand up a separate ZooKeeper cluster for configuration tasks.
+If you deploy a single-cluster instance, you do not need a separate cluster for the configuration store. If, however, you deploy a multi-cluster instance, you should stand up a separate ZooKeeper cluster for configuration tasks.
#### Single-cluster Pulsar instance
If your Pulsar instance consists of just one cluster, then you can deploy a configuration store on the same machines as the local ZooKeeper quorum but run on different TCP ports.
-To deploy a ZooKeeper configuration store in a single-cluster instance, add the same ZooKeeper servers that the local quorum uses to the configuration file in [`conf/global_zookeeper.conf`](reference-configuration.md#configuration-store) using the same method for [local ZooKeeper](#local-zookeeper), but make sure to use a different port (2181 is the default for ZooKeeper). The following is an example that uses port 2184 for a three-node ZooKeeper cluster:
+To deploy a ZooKeeper configuration store in a single-cluster instance, add the same ZooKeeper servers that the local quorum. You need to use the configuration file in [`conf/global_zookeeper.conf`](reference-configuration.md#configuration-store) using the same method for [local ZooKeeper](#local-zookeeper), but make sure to use a different port (2181 is the default for ZooKeeper). The following is an example that uses port 2184 for a three-node ZooKeeper cluster:
```properties
@@ -165,13 +159,11 @@ As before, create the `myid` files for each server on `data/global-zookeeper/myi
When you deploy a global Pulsar instance, with clusters distributed across different geographical regions, the configuration store serves as a highly available and strongly consistent metadata store that can tolerate failures and partitions spanning whole regions.
-The key here is to make sure the ZK quorum members are spread across at least 3 regions and that other regions run as observers.
+The key here is to make sure the ZK quorum members are spread across at least 3 regions, and other regions run as observers.
-Again, given the very low expected load on the configuration store servers, you can
-share the same hosts used for the local ZooKeeper quorum.
+Again, given the very low expected load on the configuration store servers, you can share the same hosts used for the local ZooKeeper quorum.
-For example, assume a Pulsar instance with the following clusters `us-west`,
-`us-east`, `us-central`, `eu-central`, `ap-south`. Also assume, each cluster has its own local ZK servers named such as the following:
+For example, assume a Pulsar instance with the following clusters `us-west`, `us-east`, `us-central`, `eu-central`, `ap-south`. Also assume, each cluster has its own local ZK servers named such as the following:
```
@@ -179,8 +171,7 @@ zk[1-3].${CLUSTER}.example.com
```
-In this scenario if you want to pick the quorum participants from few clusters and
-let all the others be ZK observers. For example, to form a 7 servers quorum, you can pick 3 servers from `us-west`, 2 from `us-central` and 2 from `us-east`.
+In this scenario if you want to pick the quorum participants from few clusters and let all the others be ZK observers. For example, to form a 7 servers quorum, you can pick 3 servers from `us-west`, 2 from `us-central` and 2 from `us-east`.
This method guarantees that writes to configuration store is possible even if one of these regions is unreachable.
@@ -227,7 +218,7 @@ $ bin/pulsar-daemon start configuration-store
## Cluster metadata initialization
-Once you set up the cluster-specific ZooKeeper and configuration store quorums for your instance, you need to write some metadata to ZooKeeper for each cluster in your instance. **you only needs to write these metadata once**.
+Once you set up the cluster-specific ZooKeeper and configuration store quorums for your instance, you need to write some metadata to ZooKeeper for each cluster in your instance. **you only need to write these metadata once**.
You can initialize this metadata using the [`initialize-cluster-metadata`](reference-cli-tools.md#pulsar-initialize-cluster-metadata) command of the [`pulsar`](reference-cli-tools.md#pulsar) CLI tool. The following is an example:
@@ -260,7 +251,7 @@ Make sure to run `initialize-cluster-metadata` for each cluster in your instance
BookKeeper provides [persistent message storage](concepts-architecture-overview.md#persistent-storage) for Pulsar.
-Each Pulsar broker needs to have its own cluster of bookies. The BookKeeper cluster shares a local ZooKeeper quorum with the Pulsar cluster.
+Each Pulsar broker needs its own cluster of bookies. The BookKeeper cluster shares a local ZooKeeper quorum with the Pulsar cluster.
### Configure bookies
@@ -280,7 +271,7 @@ $ bin/pulsar-daemon start bookie
You can verify that the bookie works properly using the `bookiesanity` command for the [BookKeeper shell](reference-cli-tools.md#bookkeeper-shell):
-```shell
+```bash
$ bin/bookkeeper shell bookiesanity
@@ -304,7 +295,7 @@ Bookie hosts are responsible for storing message data on disk. In order for book
Message entries written to bookies are always synced to disk before returning an acknowledgement to the Pulsar broker. To ensure low write latency, BookKeeper is
designed to use multiple devices:
-* A **journal** to ensure durability. For sequential writes, having fast [fsync](https://linux.die.net/man/2/fsync) operations on bookie hosts is critical. Typically, small and fast [solid-state drives](https://en.wikipedia.org/wiki/Solid-state_drive) (SSDs) should suffice, or [hard disk drives](https://en.wikipedia.org/wiki/Hard_disk_drive) (HDDs) with a [RAID](https://en.wikipedia.org/wiki/RAID)s controller and a battery-backed write cache. Both solutions can reach fsync latency of ~0.4 ms.
+* A **journal** to ensure durability. For sequential writes, having fast [fsync](https://linux.die.net/man/2/fsync) operations on bookie hosts is critical. Typically, small and fast [solid-state drives](https://en.wikipedia.org/wiki/Solid-state_drive) (SSDs) should suffice, or [hard disk drives](https://en.wikipedia.org/wiki/Hard_disk_drive) (HDDs) with a [RAID](https://en.wikipedia.org/wiki/RAID) controller and a battery-backed write cache. Both solutions can reach fsync latency of ~0.4 ms.
* A **ledger storage device** is where data is stored until all consumers acknowledge the message. Writes happen in the background, so write I/O is not a big concern. Reads happen sequentially most of the time and the backlog is drained only in case of consumer drain. To store large amounts of data, a typical configuration involves multiple HDDs with a RAID controller.
@@ -371,39 +362,13 @@ $ bin/pulsar broker
## Service discovery
-[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup).
+[Clients](getting-started-clients) connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL.
-You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means.
+You can use your own service discovery system. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means.
-> #### Service discovery already provided by many scheduling systems
+> **Service discovery already provided by many scheduling systems**
> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism.
-
-### Service discovery setup
-
-The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol) of Pulsar.
-
-To get started setting up the built-in service of discovery of Pulsar, you need to change a few parameters in the [`conf/discovery.conf`](reference-configuration.md#service-discovery) configuration file. Set the [`zookeeperServers`](reference-configuration.md#service-discovery-zookeeperServers) parameter to the ZooKeeper quorum connection string of the cluster and the [`configurationStoreServers`](reference-configuration.md#service-discovery-configurationStoreServers) setting to the [con [...]
-store](reference-terminology.md#configuration-store) quorum connection string.
-
-```properties
-
-# Zookeeper quorum connection string
-zookeeperServers=zk1.us-west.example.com:2181,zk2.us-west.example.com:2181,zk3.us-west.example.com:2181
-
-# Global configuration store connection string
-configurationStoreServers=zk1.us-west.example.com:2184,zk2.us-west.example.com:2184,zk3.us-west.example.com:2184
-
-```
-
-To start the discovery service:
-
-```shell
-
-$ bin/pulsar-daemon start discovery
-
-```
-
## Admin client and verification
At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients.
diff --git a/site2/website-next/versioned_docs/version-2.4.2/develop-bare-metal.md b/site2/website-next/versioned_docs/version-2.4.1/deploy-bare-metal.md
similarity index 100%
rename from site2/website-next/versioned_docs/version-2.4.2/develop-bare-metal.md
rename to site2/website-next/versioned_docs/version-2.4.1/deploy-bare-metal.md
diff --git a/site2/website-next/versioned_docs/version-2.4.1/deploy-dcos.md b/site2/website-next/versioned_docs/version-2.4.1/deploy-dcos.md
index f5f8d1f..07f446e 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/deploy-dcos.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/deploy-dcos.md
@@ -7,18 +7,17 @@ original_id: deploy-dcos
:::tip
-If you want to enable all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of
-`apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors).
+To enable all built-in [Pulsar IO](io-overview) connectors in your Pulsar deploymente, we recommend you use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image; the former has already bundled [all built-in connectors](io-overview.md#working-with-connectors).
:::
-[DC/OS](https://dcos.io/) (the <strong>D</strong>ata<strong>C</strong>enter <strong>O</strong>perating <strong>S</strong>ystem) is a distributed operating system used for deploying and managing applications and systems on [Apache Mesos](http://mesos.apache.org/). DC/OS is an open-source tool that [Mesosphere](https://mesosphere.com/) creates and maintains .
+[DC/OS](https://dcos.io/) (the <strong>D</strong>ata<strong>C</strong>enter <strong>O</strong>perating <strong>S</strong>ystem) is a distributed operating system for deploying and managing applications and systems on [Apache Mesos](http://mesos.apache.org/). DC/OS is an open-source tool created and maintained by [Mesosphere](https://mesosphere.com/).
Apache Pulsar is available as a [Marathon Application Group](https://mesosphere.github.io/marathon/docs/application-groups.html), which runs multiple applications as manageable sets.
## Prerequisites
-In order to run Pulsar on DC/OS, you need the following:
+You need to prepare your environment before running Pulsar on DC/OS.
* DC/OS version [1.9](https://docs.mesosphere.com/1.9/) or higher
* A [DC/OS cluster](https://docs.mesosphere.com/1.9/installing/) with at least three agent nodes
@@ -37,7 +36,7 @@ Each node in the DC/OS-managed Mesos cluster must have at least:
* 4 GB of memory
* 60 GB of total persistent disk
-Alternatively, you can change the configuration in `PulsarGroups.json` according to match your resources of DC/OS cluster.
+Alternatively, you can change the configuration in `PulsarGroups.json` accordingly to match your resources of the DC/OS cluster.
## Deploy Pulsar using the DC/OS command interface
@@ -56,9 +55,9 @@ This command deploys Docker container instances in three groups, which together
* 1 [Prometheus](http://prometheus.io/) instance and 1 [Grafana](https://grafana.com/) instance
-> When you run DC/OS, a ZooKeeper cluster already runs at `master.mesos:2181`, thus you do not have to install or start up ZooKeeper separately.
+> When you run DC/OS, a ZooKeeper cluster will be running at `master.mesos:2181`, thus you do not have to install or start up ZooKeeper separately.
-After executing the `dcos` command above, click on the **Services** tab in the DC/OS [GUI interface](https://docs.mesosphere.com/latest/gui/), which you can access at [http://m1.dcos](http://m1.dcos) in this example. You should see several applications in the process of deploying.
+After executing the `dcos` command above, click the **Services** tab in the DC/OS [GUI interface](https://docs.mesosphere.com/latest/gui/), which you can access at [http://m1.dcos](http://m1.dcos) in this example. You should see several applications during the deployment.
@@ -66,15 +65,15 @@ After executing the `dcos` command above, click on the **Services** tab in the D
## The BookKeeper group
-To monitor the status of the BookKeeper cluster deployment, click on the **bookkeeper** group in the parent **pulsar** group.
+To monitor the status of the BookKeeper cluster deployment, click the **bookkeeper** group in the parent **pulsar** group.
-At this point, 3 [bookies](reference-terminology.md#bookie) should be shown as green, which means that the bookies have been deployed successfully and are now running.
+At this point, the status of the 3 [bookies](reference-terminology.md#bookie) are green, which means that the bookies have been deployed successfully and are running.
-You can also click into each bookie instance to get more detailed information, such as the bookie running log.
+You can also click each bookie instance to get more detailed information, such as the bookie running log.
@@ -82,23 +81,23 @@ To display information about the BookKeeper in ZooKeeper, you can visit [http://
-## The Pulsar broker Group
+## The Pulsar broker group
-Similar to the BookKeeper group above, click into the **brokers** to check the status of the Pulsar brokers.
+Similar to the BookKeeper group above, click **brokers** to check the status of the Pulsar brokers.
-You can also click into each broker instance to get more detailed information, such as the broker running log.
+You can also click each broker instance to get more detailed information, such as the broker running log.
-Broker cluster information in Zookeeper is also available through the web UI. In this example, you can see that the `loadbalance` and `managed-ledgers` directories have been created.
+Broker cluster information in ZooKeeper is also available through the web UI. In this example, you can see that the `loadbalance` and `managed-ledgers` directories have been created.
-## Monitor Group
+## Monitor group
The **monitory** group consists of Prometheus and Grafana.
@@ -106,17 +105,17 @@ The **monitory** group consists of Prometheus and Grafana.
### Prometheus
-Click into the instance of `prom` to get the endpoint of Prometheus, which is `192.168.65.121:9090` in this example.
+Click the instance of `prom` to get the endpoint of Prometheus, which is `192.168.65.121:9090` in this example.
-If you click that endpoint, you can see the Prometheus dashboard. The [http://192.168.65.121:9090/targets](http://192.168.65.121:9090/targets) URL display all the bookies and brokers.
+If you click that endpoint, you can see the Prometheus dashboard. All the bookies and brokers are listed on [http://192.168.65.121:9090/targets](http://192.168.65.121:9090/targets).
### Grafana
-Click into `grafana` to get the endpoint for Grafana, which is `192.168.65.121:3000` in this example.
+Click `grafana` to get the endpoint for Grafana, which is `192.168.65.121:3000` in this example.
@@ -130,7 +129,7 @@ Now that you have a fully deployed Pulsar cluster, you can run a simple consumer
### Download and prepare the Pulsar Java tutorial
-You can clone a [Pulsar Java tutorial](https://github.com/streamlio/pulsar-java-tutorial) repo. This repo contains a simple Pulsar consumer and producer (you can find more information in the `README` file of the repo).
+You can clone a [Pulsar Java tutorial](https://github.com/streamlio/pulsar-java-tutorial) repo. This repo contains a simple Pulsar consumer and producer (you can find more information in the `README` file in this repo).
```bash
@@ -138,12 +137,13 @@ $ git clone https://github.com/streamlio/pulsar-java-tutorial
```
-Change the `SERVICE_URL` from `pulsar://localhost:6650` to `pulsar://a1.dcos:6650` in both [`ConsumerTutorial.java`](https://github.com/streamlio/pulsar-java-tutorial/blob/master/src/main/java/tutorial/ConsumerTutorial.java) and [`ProducerTutorial.java`](https://github.com/streamlio/pulsar-java-tutorial/blob/master/src/main/java/tutorial/ProducerTutorial.java).
-The `pulsar://a1.dcos:6650` endpoint is for the broker service. You can fetch the endpoint details for each broker instance from the DC/OS GUI. `a1.dcos` is a DC/OS client agent, which runs a broker. The client agent IP address can also replace this.
+Change the `SERVICE_URL` from `pulsar://localhost:6650` to `pulsar://a1.dcos:6650` in both [`ConsumerTutorial.java`](https://github.com/streamlio/pulsar-java-tutorial/blob/master/src/main/java/tutorial/ConsumerTutorial.java) file and [`ProducerTutorial.java`](https://github.com/streamlio/pulsar-java-tutorial/blob/master/src/main/java/tutorial/ProducerTutorial.java) file.
-Now, change the message number from 10 to 10000000 in main method of [`ProducerTutorial.java`](https://github.com/streamlio/pulsar-java-tutorial/blob/master/src/main/java/tutorial/ProducerTutorial.java) so that it can produce more messages.
+The `pulsar://a1.dcos:6650` endpoint is for the broker service. You can fetch the endpoint details for each broker instance from the DC/OS GUI. `a1.dcos` is a DC/OS client agent that runs a broker, and you can replace it with the client agent IP address.
-Now compile the project code using the command below:
+Now, you can change the message number from 10 to 10000000 in the main method in [`ProducerTutorial.java`](https://github.com/streamlio/pulsar-java-tutorial/blob/master/src/main/java/tutorial/ProducerTutorial.java) file to produce more messages.
+
+Then, you can compile the project code using the command below:
```bash
@@ -169,7 +169,7 @@ $ mvn exec:java -Dexec.mainClass="tutorial.ProducerTutorial"
```
-You can see the producer producing messages and the consumer consuming messages through the DC/OS GUI.
+You see that the producer is producing messages and the consumer is consuming messages through the DC/OS GUI.
@@ -177,20 +177,20 @@ You can see the producer producing messages and the consumer consuming messages
### View Grafana metric output
-While the producer and consumer run, you can access running metrics information from Grafana.
+While the producer and consumer are running, you can access the running metrics from Grafana.
## Uninstall Pulsar
-You can shut down and uninstall the `pulsar` application from DC/OS at any time in the following two ways:
+You can shut down and uninstall the `pulsar` application from DC/OS at any time in one of the following two ways:
-1. Using the DC/OS GUI, you can choose **Delete** at the right end of Pulsar group.
+1. Click the three dots at the right end of Pulsar group and choose **Delete** on the DC/OS GUI.
-2. You can use the following command:
+2. Use the command below.
```bash
diff --git a/site2/website-next/versioned_docs/version-2.4.1/deploy-kubernetes.md b/site2/website-next/versioned_docs/version-2.4.1/deploy-kubernetes.md
index dc7123d..4e170dc 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/deploy-kubernetes.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/deploy-kubernetes.md
@@ -2,7 +2,6 @@
id: deploy-kubernetes
title: Deploy Pulsar on Kubernetes
sidebar_label: "Kubernetes"
-original_id: deploy-kubernetes
---
To get up and running with these charts as fast as possible, in a **non-production** use case, we provide
diff --git a/site2/website-next/versioned_docs/version-2.4.1/deploy-monitoring.md b/site2/website-next/versioned_docs/version-2.4.1/deploy-monitoring.md
index 074ce3f..95ccdd6 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/deploy-monitoring.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/deploy-monitoring.md
@@ -2,7 +2,6 @@
id: deploy-monitoring
title: Monitor
sidebar_label: "Monitor"
-original_id: deploy-monitoring
---
You can use different ways to monitor a Pulsar cluster, exposing both metrics related to the usage of topics and the overall health of the individual components of the cluster.
@@ -127,17 +126,7 @@ The per-topic dashboard instructions are available at [Pulsar manager](administr
You can use grafana to create dashboard driven by the data that is stored in Prometheus.
-When you deploy Pulsar on Kubernetes, a `pulsar-grafana` Docker image is enabled by default. You can use the docker image with the principal dashboards.
-
-Enter the command below to use the dashboard manually:
-
-```shell
-
-docker run -p3000:3000 \
- -e PROMETHEUS_URL=http://$PROMETHEUS_HOST:9090/ \
- apachepulsar/pulsar-grafana:latest
-
-```
+When you deploy Pulsar on Kubernetes with the Pulsar Helm Chart, a `pulsar-grafana` Docker image is enabled by default. You can use the docker image with the principal dashboards.
The following are some Grafana dashboards examples:
@@ -145,4 +134,4 @@ The following are some Grafana dashboards examples:
- [apache-pulsar-grafana-dashboard](https://github.com/streamnative/apache-pulsar-grafana-dashboard): a collection of Grafana dashboard templates for different Pulsar components running on both Kubernetes and on-premise machines.
## Alerting rules
-You can set alerting rules according to your Pulsar environment. To configure alerting rules for Apache Pulsar, refer to [alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/).
+You can set alerting rules according to your Pulsar environment. To configure alerting rules for Apache Pulsar, refer to [alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/).
\ No newline at end of file
diff --git a/site2/website-next/versioned_docs/version-2.4.1/develop-binary-protocol.md b/site2/website-next/versioned_docs/version-2.4.1/develop-binary-protocol.md
index b233f10..fa03383 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/develop-binary-protocol.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/develop-binary-protocol.md
@@ -2,7 +2,6 @@
id: develop-binary-protocol
title: Pulsar binary protocol specification
sidebar_label: "Binary protocol"
-original_id: develop-binary-protocol
---
Pulsar uses a custom binary protocol for communications between producers/consumers and brokers. This protocol is designed to support required features, such as acknowledgements and flow control, while ensuring maximum transport and implementation efficiency.
@@ -29,42 +28,63 @@ The Pulsar protocol allows for two types of commands:
Simple (payload-free) commands have this basic structure:
-| Component | Description | Size (in bytes) |
-|:------------|:----------------------------------------------------------------------------------------|:----------------|
-| totalSize | The size of the frame, counting everything that comes after it (in bytes) | 4 |
-| commandSize | The size of the protobuf-serialized command | 4 |
-| message | The protobuf message serialized in a raw binary format (rather than in protobuf format) | |
+| Component | Description | Size (in bytes) |
+|:--------------|:----------------------------------------------------------------------------------------|:----------------|
+| `totalSize` | The size of the frame, counting everything that comes after it (in bytes) | 4 |
+| `commandSize` | The size of the protobuf-serialized command | 4 |
+| `message` | The protobuf message serialized in a raw binary format (rather than in protobuf format) | |
### Payload commands
Payload commands have this basic structure:
-| Component | Description | Size (in bytes) |
-|:-------------|:--------------------------------------------------------------------------------------------|:----------------|
-| totalSize | The size of the frame, counting everything that comes after it (in bytes) | 4 |
-| commandSize | The size of the protobuf-serialized command | 4 |
-| message | The protobuf message serialized in a raw binary format (rather than in protobuf format) | |
-| magicNumber | A 2-byte byte array (`0x0e01`) identifying the current format | 2 |
-| checksum | A [CRC32-C checksum](http://www.evanjones.ca/crc32c.html) of everything that comes after it | 4 |
-| metadataSize | The size of the message [metadata](#message-metadata) | 4 |
-| metadata | The message [metadata](#message-metadata) stored as a binary protobuf message | |
-| payload | Anything left in the frame is considered the payload and can include any sequence of bytes | |
+| Component | Required or optional| Description | Size (in bytes) |
+|:-----------------------------------|:----------|:--------------------------------------------------------------------------------------------|:----------------|
+| `totalSize` | Required | The size of the frame, counting everything that comes after it (in bytes) | 4 |
+| `commandSize` | Required | The size of the protobuf-serialized command | 4 |
+| `message` | Required | The protobuf message serialized in a raw binary format (rather than in protobuf format) | |
+| `magicNumberOfBrokerEntryMetadata` | Optional | A 2-byte byte array (`0x0e02`) identifying the broker entry metadata <br /> **Note**: `magicNumberOfBrokerEntryMetadata` , `brokerEntryMetadataSize`, and `brokerEntryMetadata` should be used **together**. | 2 |
+| `brokerEntryMetadataSize` | Optional | The size of the broker entry metadata | 4 |
+| `brokerEntryMetadata` | Optional | The broker entry metadata stored as a binary protobuf message | |
+| `magicNumber` | Required | A 2-byte byte array (`0x0e01`) identifying the current format | 2 |
+| `checksum` | Required | A [CRC32-C checksum](http://www.evanjones.ca/crc32c.html) of everything that comes after it | 4 |
+| `metadataSize` | Required | The size of the message [metadata](#message-metadata) | 4 |
+| `metadata` | Required | The message [metadata](#message-metadata) stored as a binary protobuf message | |
+| `payload` | Required | Anything left in the frame is considered the payload and can include any sequence of bytes | |
+
+## Broker entry metadata
+
+Broker entry metadata is stored alongside the message metadata as a serialized protobuf message.
+It is created by the broker when the message arrived at the broker and passed without changes to the consumer if configured.
+
+| Field | Required or optional | Description |
+|:-------------------|:----------------|:------------------------------------------------------------------------------------------------------------------------------|
+| `broker_timestamp` | Optional | The timestamp when a message arrived at the broker (`id est` as the number of milliseconds since January 1st, 1970 in UTC) |
+| `index` | Optional | The index of the message. It is assigned by the broker.
+
+If you want to use broker entry metadata for **brokers**, configure the [`brokerEntryMetadataInterceptors`](reference-configuration.md#broker) parameter in the `broker.conf` file.
+
+If you want to use broker entry metadata for **consumers**:
+
+1. Use the client protocol version [18 or later](https://github.com/apache/pulsar/blob/ca37e67211feda4f7e0984e6414e707f1c1dfd07/pulsar-common/src/main/proto/PulsarApi.proto#L259).
+
+2. Configure the [`brokerEntryMetadataInterceptors`](reference-configuration.md#broker) parameter and set the [`enableExposingBrokerEntryMetadataToClient`](reference-configuration.md#broker) parameter to `true` in the `broker.conf` file.
## Message metadata
-Message metadata is stored alongside the application-specified payload as a serialized protobuf message. Metadata is created by the producer and passed on unchanged to the consumer.
+Message metadata is stored alongside the application-specified payload as a serialized protobuf message. Metadata is created by the producer and passed without changes to the consumer.
-| Field | Description |
-|:-------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `producer_name` | The name of the producer that published the message |
-| `sequence_id` | The sequence ID of the message, assigned by producer |
-| `publish_time` | The publish timestamp in Unix time (i.e. as the number of milliseconds since January 1st, 1970 in UTC) |
-| `properties` | A sequence of key/value pairs (using the [`KeyValue`](https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/proto/PulsarApi.proto#L32) message). These are application-defined keys and values with no special meaning to Pulsar. |
-| `replicated_from` *(optional)* | Indicates that the message has been replicated and specifies the name of the [cluster](reference-terminology.md#cluster) where the message was originally published |
-| `partition_key` *(optional)* | While publishing on a partition topic, if the key is present, the hash of the key is used to determine which partition to choose |
-| `compression` *(optional)* | Signals that payload has been compressed and with which compression library |
-| `uncompressed_size` *(optional)* | If compression is used, the producer must fill the uncompressed size field with the original payload size |
-| `num_messages_in_batch` *(optional)* | If this message is really a [batch](#batch-messages) of multiple entries, this field must be set to the number of messages in the batch |
+| Field | Required or optional | Description |
+|:-------------------------|:----------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `producer_name` | Required | The name of the producer that published the message |
+| `sequence_id` | Required | The sequence ID of the message, assigned by producer |
+| `publish_time` | Required | The publish timestamp in Unix time (i.e. as the number of milliseconds since January 1st, 1970 in UTC) |
+| `properties` | Required | A sequence of key/value pairs (using the [`KeyValue`](https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/proto/PulsarApi.proto#L32) message). These are application-defined keys and values with no special meaning to Pulsar. |
+| `replicated_from` | Optional | Indicates that the message has been replicated and specifies the name of the [cluster](reference-terminology.md#cluster) where the message was originally published |
+| `partition_key` | Optional | While publishing on a partition topic, if the key is present, the hash of the key is used to determine which partition to choose. Partition key is used as the message key. |
+| `compression` | Optional | Signals that payload has been compressed and with which compression library |
+| `uncompressed_size` | Optional | If compression is used, the producer must fill the uncompressed size field with the original payload size |
+| `num_messages_in_batch` | Optional | If this message is really a [batch](#batch-messages) of multiple entries, this field must be set to the number of messages in the batch |
### Batch messages
@@ -76,19 +96,19 @@ object.
For a single batch, the payload format will look like this:
-| Field | Description |
-|:--------------|:------------------------------------------------------------|
-| metadataSizeN | The size of the single message metadata serialized Protobuf |
-| metadataN | Single message metadata |
-| payloadN | Message payload passed by application |
+| Field | Required or optional | Description |
+|:----------------|:---------------------|:-----------------------------------------------------------|
+| `metadataSizeN` | Required |The size of the single message metadata serialized Protobuf |
+| `metadataN` | Required |Single message metadata |
+| `payloadN` | Required |Message payload passed by application |
Each metadata field looks like this;
-| Field | Description |
-|:---------------------------|:--------------------------------------------------------|
-| properties | Application-defined properties |
-| partition key *(optional)* | Key to indicate the hashing to a particular partition |
-| payload_size | Size of the payload for the single message in the batch |
+| Field | Required or optional | Description |
+|:----------------|:----------------------|:--------------------------------------------------------|
+| `properties` | Required | Application-defined properties |
+| `partition key` | Optional | Key to indicate the hashing to a particular partition |
+| `payload_size` | Required | Size of the payload for the single message in the batch |
When compression is enabled, the whole batch will be compressed at once.
@@ -170,6 +190,10 @@ messages to the broker, referring to the producer id negotiated before.
+If the client does not receive a response indicating producer creation success or failure,
+the client should first send a command to close the original producer before sending a
+command to re-attempt producer creation.
+
##### Command Producer
```protobuf
@@ -273,6 +297,11 @@ When receiving a `CloseProducer` command, the broker will stop accepting any
more messages for the producer, wait until all pending messages are persisted
and then reply `Success` to the client.
+If the client does not receive a response to a `Producer` command within a timeout,
+the client must first send a `CloseProducer` command before sending another
+`Producer` command. The client does not need to await a response to the `CloseProducer`
+command before sending the next `Producer` command.
+
The broker can send a `CloseProducer` command to client when it's performing
a graceful failover (eg: broker is being restarted, or the topic is being unloaded
by load balancer to be transferred to a different broker).
diff --git a/site2/website-next/versioned_docs/version-2.4.1/develop-load-manager.md b/site2/website-next/versioned_docs/version-2.4.1/develop-load-manager.md
index 509209b..9687f30 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/develop-load-manager.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/develop-load-manager.md
@@ -2,7 +2,6 @@
id: develop-load-manager
title: Modular load manager
sidebar_label: "Modular load manager"
-original_id: develop-load-manager
---
The *modular load manager*, implemented in [`ModularLoadManagerImpl`](https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImpl.java), is a flexible alternative to the previously implemented load manager, [`SimpleLoadManagerImpl`](https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/SimpleLoadManagerImpl.java), which attempts to simplify how load [...]
diff --git a/site2/website-next/versioned_docs/version-2.4.1/develop-tools.md b/site2/website-next/versioned_docs/version-2.4.1/develop-tools.md
index b545779..d034926 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/develop-tools.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/develop-tools.md
@@ -2,7 +2,6 @@
id: develop-tools
title: Simulation tools
sidebar_label: "Simulation tools"
-original_id: develop-tools
---
It is sometimes necessary create an test environment and incur artificial load to observe how well load managers
diff --git a/site2/website-next/versioned_docs/version-2.4.1/io-cdc.md b/site2/website-next/versioned_docs/version-2.4.1/io-cdc.md
index 20f16ae..df37397 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/io-cdc.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/io-cdc.md
@@ -2,7 +2,6 @@
id: io-cdc
title: CDC connector
sidebar_label: "CDC connector"
-original_id: io-cdc
---
CDC source connectors capture log changes of databases (such as MySQL, MongoDB, and PostgreSQL) into Pulsar.
diff --git a/site2/website-next/versioned_docs/version-2.4.1/io-overview.md b/site2/website-next/versioned_docs/version-2.4.1/io-overview.md
index 68960a8..810de78 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/io-overview.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/io-overview.md
@@ -2,7 +2,6 @@
id: io-overview
title: Pulsar connector overview
sidebar_label: "Overview"
-original_id: io-overview
---
import Tabs from '@theme/Tabs';
diff --git a/site2/website-next/versioned_docs/version-2.4.1/io-quickstart.md b/site2/website-next/versioned_docs/version-2.4.1/io-quickstart.md
index acab1eb..10a0380 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/io-quickstart.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/io-quickstart.md
@@ -522,7 +522,7 @@ Now that we have a MySQL running locally. In this section, we will configure a J
To run a JDBC sink connector, you need to prepare a yaml config file including the information that Pulsar IO runtime needs to know. For example, how Pulsar IO can find the MySQL cluster, what is the JDBCURL and the table that Pulsar IO will use for writing messages to.
- Create a _pulsar-mysql-jdbc-sink.yaml_ file, copy the following contents to this file, and place the file in the `pulsar/connectors` folder.
+ Create a _pulsar-mysql-jdbc-sink.yaml_ file , copy the following contents to this file, and place the file in the `pulsar/connectors` folder.
```text
diff --git a/site2/website-next/versioned_docs/version-2.4.1/pulsar-2.0.md b/site2/website-next/versioned_docs/version-2.4.1/pulsar-2.0.md
index 11c5e66..560c8c1 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/pulsar-2.0.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/pulsar-2.0.md
@@ -2,7 +2,6 @@
id: pulsar-2.0
title: Pulsar 2.0
sidebar_label: "Pulsar 2.0"
-original_id: pulsar-2.0
---
Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more.
diff --git a/site2/website-next/versioned_docs/version-2.4.1/reference-pulsar-admin.md b/site2/website-next/versioned_docs/version-2.4.1/pulsar-admin.md
similarity index 100%
rename from site2/website-next/versioned_docs/version-2.4.1/reference-pulsar-admin.md
rename to site2/website-next/versioned_docs/version-2.4.1/pulsar-admin.md
diff --git a/site2/website-next/versioned_docs/version-2.4.1/reference-cli-tools.md b/site2/website-next/versioned_docs/version-2.4.1/reference-cli-tools.md
index 8c2c64f..0c8aea1 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/reference-cli-tools.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/reference-cli-tools.md
@@ -2,7 +2,6 @@
id: reference-cli-tools
title: Pulsar command-line tools
sidebar_label: "Pulsar CLI tools"
-original_id: reference-cli-tools
---
Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more.
@@ -16,8 +15,12 @@ All Pulsar command-line tools can be run from the `bin` directory of your [insta
* [`bookkeeper`](#bookkeeper)
* [`broker-tool`](#broker-tool)
-> ### Getting help
-> You can get help for any CLI tool, command, or subcommand using the `--help` flag, or `-h` for short. Here's an example:
+> **Important**
+>
+> - This page only shows **some frequently used commands**. For the latest information about `pulsar`, `pulsar-client`, and `pulsar-perf`, including commands, flags, descriptions, and more information, see [Pulsar tools](https://pulsar.apache.org/tools/).
+>
+> - You can get help for any CLI tool, command, or subcommand using the `--help` flag, or `-h` for short. Here's an example:
+>
> ```shell
>
@@ -45,7 +48,6 @@ Commands:
* `bookie`
* `broker`
* `compact-topic`
-* `discovery`
* `configuration-store`
* `initialize-cluster-metadata`
* `proxy`
@@ -53,6 +55,7 @@ Commands:
* `websocket`
* `zookeeper`
* `zookeeper-shell`
+* `autorecovery`
Example:
@@ -71,14 +74,13 @@ The table below lists the environment variables that you can use to configure th
|`PULSAR_BOOKKEEPER_CONF`|description: Configuration file for bookie|`conf/bookkeeper.conf`|
|`PULSAR_ZK_CONF`|Configuration file for zookeeper|`conf/zookeeper.conf`|
|`PULSAR_CONFIGURATION_STORE_CONF`|Configuration file for the configuration store|`conf/global_zookeeper.conf`|
-|`PULSAR_DISCOVERY_CONF`|Configuration file for discovery service|`conf/discovery.conf`|
|`PULSAR_WEBSOCKET_CONF`|Configuration file for websocket proxy|`conf/websocket.conf`|
|`PULSAR_STANDALONE_CONF`|Configuration file for standalone|`conf/standalone.conf`|
|`PULSAR_EXTRA_OPTS`|Extra options to be passed to the jvm||
|`PULSAR_EXTRA_CLASSPATH`|Extra paths for Pulsar's classpath||
|`PULSAR_PID_DIR`|Folder where the pulsar server PID file should be stored||
|`PULSAR_STOP_TIMEOUT`|Wait time before forcefully killing the Bookie server instance if attempts to stop it are not successful||
-
+|`PULSAR_GC_LOG`|Gc options to be passed to the jvm||
### `bookie`
@@ -165,26 +167,6 @@ $ pulsar compact-topic --topic topic-to-compact
```
-### `discovery`
-
-Run a discovery server
-
-Usage
-
-```bash
-
-$ pulsar discovery
-
-```
-
-Example
-
-```bash
-
-$ PULSAR_DISCOVERY_CONF=/path/to/discovery.conf pulsar discovery
-
-```
-
### `configuration-store`
Starts up the Pulsar configuration store
@@ -224,14 +206,14 @@ Options
|`-ub` , `--broker-service-url`|The broker service URL for the new cluster||
|`-tb` , `--broker-service-url-tls`|The broker service URL for the new cluster with TLS encryption||
|`-c` , `--cluster`|Cluster name||
-|`-cs` , `--configuration-store`|The configuration store quorum connection string||
+|`-cms` , `--configuration-metadata-store`|The configuration metadata store quorum connection string||
|`--existing-bk-metadata-service-uri`|The metadata service URI of the existing BookKeeper cluster that you want to use||
|`-h` , `--help`|Cluster name|false|
|`--initial-num-stream-storage-containers`|The number of storage containers of BookKeeper stream storage|16|
|`--initial-num-transaction-coordinators`|The number of transaction coordinators assigned in a cluster|16|
|`-uw` , `--web-service-url`|The web service URL for the new cluster||
|`-tw` , `--web-service-url-tls`|The web service URL for the new cluster with TLS encryption||
-|`-zk` , `--zookeeper`|The local ZooKeeper quorum connection string||
+|`-md` , `--metadata-store`|The metadata store service url||
|`--zookeeper-session-timeout-ms`|The local ZooKeeper session timeout. The time unit is in millisecond(ms)|30000|
@@ -355,6 +337,23 @@ Options
|`-c`, `--conf`|Configuration file for ZooKeeper||
|`-server`|Configuration zk address, eg: `127.0.0.1:2181`||
+### `autorecovery`
+
+Runs an auto-recovery service.
+
+Usage
+
+```bash
+
+$ pulsar autorecovery options
+
+```
+
+Options
+
+|Flag|Description|Default|
+|---|---|---|
+|`-c`, `--conf`|Configuration for the autorecovery|N/A|
## `pulsar-client`
@@ -407,6 +406,7 @@ Options
|`-m`, `--messages`|Comma-separated string of messages to send; either -m or -f must be specified|[]|
|`-n`, `--num-produce`|The number of times to send the message(s); the count of messages/files * num-produce should be below 1000|1|
|`-r`, `--rate`|Rate (in messages per second) at which to produce; a value 0 means to produce messages as fast as possible|0.0|
+|`-db`, `--disable-batching`|Disable batch sending of messages|false|
|`-c`, `--chunking`|Split the message and publish in chunks if the message size is larger than the allowed max size|false|
|`-s`, `--separator`|Character to split messages string with.|","|
|`-k`, `--key`|Message key to add|key=value string, like k1=v1,k2=v2.|
@@ -462,6 +462,7 @@ $ pulsar-daemon command
Commands
* `start`
* `stop`
+* `restart`
### `start`
@@ -492,7 +493,14 @@ Options
|---|---|---|
|-force|Stop the service forcefully if not stopped by normal shutdown.|false|
+### `restart`
+Restart a service that has already been started.
+```bash
+
+$ pulsar-daemon restart service
+
+```
## `pulsar-perf`
A tool for performance testing a Pulsar broker.
@@ -514,6 +522,7 @@ Commands
* `monitor-brokers`
* `simulation-client`
* `simulation-controller`
+* `transaction`
* `help`
Environment variables
@@ -526,6 +535,7 @@ The table below lists the environment variables that you can use to configure th
|`PULSAR_CLIENT_CONF`|Configuration file for the client|conf/client.conf|
|`PULSAR_EXTRA_OPTS`|Extra options to be passed to the JVM||
|`PULSAR_EXTRA_CLASSPATH`|Extra paths for Pulsar's classpath||
+|`PULSAR_GC_LOG`|Gc options to be passed to the jvm||
### `consume`
@@ -544,7 +554,7 @@ Options
|Flag|Description|Default|
|---|---|---|
|`--auth-params`|Authentication parameters, whose format is determined by the implementation of method `configure` in authentication plugin class. For example, `key1:val1,key2:val2` or `{"key1":"val1","key2":"val2"}`.||
-|`--auth_plugin`|Authentication plugin class name||
+|`--auth-plugin`|Authentication plugin class name||
|`-ac`, `--auto_ack_chunk_q_full`|Auto ack for the oldest message in consumer's receiver queue if the queue full|false|
|`--listener-name`|Listener name for the broker||
|`--acks-delay-millis`|Acknowledgements grouping delay in millis|100|
@@ -553,11 +563,13 @@ Options
|`-v`, `--encryption-key-value-file`|The file which contains the private key to decrypt payload||
|`-h`, `--help`|Help message|false|
|`--conf-file`|Configuration file||
+|`-m`, `--num-messages`|Number of messages to consume in total. If the value is equal to or smaller than 0, it keeps consuming messages.|0|
|`-e`, `--expire_time_incomplete_chunked_messages`|The expiration time for incomplete chunk messages (in milliseconds)|0|
|`-c`, `--max-connections`|Max number of TCP connections to a single broker|100|
|`-mc`, `--max_chunked_msg`|Max pending chunk messages|0|
|`-n`, `--num-consumers`|Number of consumers (per topic)|1|
|`-ioThreads`, `--num-io-threads`|Set the number of threads to be used for handling connections to brokers|1|
+|`-lt`, `--num-listener-threads`|Set the number of threads to be used for message listeners|1|
|`-ns`, `--num-subscriptions`|Number of subscriptions (per topic)|1|
|`-t`, `--num-topics`|The number of topics|1|
|`-pm`, `--pool-messages`|Use the pooled message|true|
@@ -571,10 +583,21 @@ Options
|`-ss`, `--subscriptions`|A list of subscriptions to consume on (e.g. sub1,sub2)|sub|
|`-st`, `--subscription-type`|Subscriber type. Possible values are Exclusive, Shared, Failover, Key_Shared.|Exclusive|
|`-sp`, `--subscription-position`|Subscriber position. Possible values are Latest, Earliest.|Latest|
-|`-time`, `--test-duration`|Test duration (in seconds). If the value is 0 or smaller than 0, it keeps consuming messages|0|
+|`-time`, `--test-duration`|Test duration (in seconds). If this value is less than or equal to 0, it keeps consuming messages.|0|
|`--trust-cert-file`|Path for the trusted TLS certificate file||
|`--tls-allow-insecure`|Allow insecure TLS connection||
+Below are **transaction** related options.
+
+If you want `--txn-timeout`, `--numMessage-perTransaction`, `-nmt`, `-ntxn`, or `-abort` take effect, set `--txn-enable` to true.
+
+|Flag|Description|Default|
+|---|---|---|
+`-tto`, `--txn-timeout`|Set the time of transaction timeout (in second). |10
+`-nmt`, `--numMessage-perTransaction`|The number of messages acknowledged by a transaction. |50
+`-txn`, `--txn-enable`|Enable or disable a transaction.|false
+`-ntxn`|The number of opened transactions. 0 means the number of transactions is unlimited. |0
+`-abort`|Abort a transaction. |true
### `produce`
Run a producer
@@ -594,7 +617,7 @@ Options
|`-am`, `--access-mode`|Producer access mode. Valid values are `Shared`, `Exclusive` and `WaitForExclusive`|Shared|
|`-au`, `--admin-url`|Pulsar admin URL||
|`--auth-params`|Authentication parameters, whose format is determined by the implementation of method `configure` in authentication plugin class. For example, `key1:val1,key2:val2` or `{"key1":"val1","key2":"val2"}`.||
-|`--auth_plugin`|Authentication plugin class name||
+|`--auth-plugin`|Authentication plugin class name||
|`--listener-name`|Listener name for the broker||
|`-b`, `--batch-time-window`|Batch messages in a window of the specified number of milliseconds|1|
|`-bb`, `--batch-max-bytes`|Maximum number of bytes per batch|4194304|
@@ -613,9 +636,9 @@ Options
|`-c`, `--max-connections`|Max number of TCP connections to a single broker|100|
|`-o`, `--max-outstanding`|Max number of outstanding messages|1000|
|`-p`, `--max-outstanding-across-partitions`|Max number of outstanding messages across partitions|50000|
+|`-m`, `--num-messages`|Number of messages to publish in total. If this value is less than or equal to 0, it keeps publishing messages.|0|
|`-mk`, `--message-key-generation-mode`|The generation mode of message key. Valid options are `autoIncrement`, `random`||
|`-ioThreads`, `--num-io-threads`|Set the number of threads to be used for handling connections to brokers|1|
-|`-m`, `--num-messages`|Number of messages to publish in total. If the value is 0 or smaller than 0, it keeps publishing messages.|0|
|`-n`, `--num-producers`|The number of producers (per topic)|1|
|`-threads`, `--num-test-threads`|Number of test threads|1|
|`-t`, `--num-topic`|The number of topics|1|
@@ -629,11 +652,21 @@ Options
|`-u`, `--service-url`|Pulsar service URL||
|`-s`, `--size`|Message size (in bytes)|1024|
|`-i`, `--stats-interval-seconds`|Statistics interval seconds. If 0, statistics will be disabled.|0|
-|`-time`, `--test-duration`|Test duration (in seconds). If the value is 0 or smaller than 0, it keeps publishing messages.|0|
+|`-time`, `--test-duration`|Test duration (in seconds). If this value is less than or equal to 0, it keeps publishing messages.|0|
|`--trust-cert-file`|Path for the trusted TLS certificate file||
|`--warmup-time`|Warm-up time in seconds|1|
|`--tls-allow-insecure`|Allow insecure TLS connection||
+Below are **transaction** related options.
+
+If you want `--txn-timeout`, `--numMessage-perTransaction`, or `-abort` take effect, set `--txn-enable` to true.
+
+|Flag|Description|Default|
+|---|---|---|
+`-tto`, `--txn-timeout`|Set the time of transaction timeout (in second). |5
+`-nmt`, `--numMessage-perTransaction`|The number of messages acknowledged by a transaction. |50
+`-txn`, `--txn-enable`|Enable or disable a transaction.|true
+`-abort`|Abort a transaction. |true
### `read`
Run a topic reader
@@ -651,19 +684,21 @@ Options
|Flag|Description|Default|
|---|---|---|
|`--auth-params`|Authentication parameters, whose format is determined by the implementation of method `configure` in authentication plugin class. For example, `key1:val1,key2:val2` or `{"key1":"val1","key2":"val2"}`.||
-|`--auth_plugin`|Authentication plugin class name||
+|`--auth-plugin`|Authentication plugin class name||
|`--listener-name`|Listener name for the broker||
|`--conf-file`|Configuration file||
|`-h`, `--help`|Help message|false|
+|`-n`, `--num-messages`|Number of messages to consume in total. If the value is equal to or smaller than 0, it keeps consuming messages.|0|
|`-c`, `--max-connections`|Max number of TCP connections to a single broker|100|
|`-ioThreads`, `--num-io-threads`|Set the number of threads to be used for handling connections to brokers|1|
+|`-lt`, `--num-listener-threads`|Set the number of threads to be used for message listeners|1|
|`-t`, `--num-topics`|The number of topics|1|
|`-r`, `--rate`|Simulate a slow message reader (rate in msg/s)|0|
|`-q`, `--receiver-queue-size`|Size of the receiver queue|1000|
|`-u`, `--service-url`|Pulsar service URL||
|`-m`, `--start-message-id`|Start message id. This can be either 'earliest', 'latest' or a specific message id by using 'lid:eid'|earliest|
|`-i`, `--stats-interval-seconds`|Statistics interval seconds. If 0, statistics will be disabled.|0|
-|`-time`, `--test-duration`|Test duration (in seconds). If the value is 0 or smaller than 0, it keeps consuming messages.|0|
+|`-time`, `--test-duration`|Test duration (in seconds). If this value is less than or equal to 0, it keeps consuming messages.|0|
|`--trust-cert-file`|Path for the trusted TLS certificate file||
|`--use-tls`|Use TLS encryption on the connection|false|
|`--tls-allow-insecure`|Allow insecure TLS connection||
@@ -684,16 +719,19 @@ Options
|Flag|Description|Default|
|---|---|---|
|`--auth-params`|Authentication parameters, whose format is determined by the implementation of method `configure` in authentication plugin class. For example, `key1:val1,key2:val2` or `{"key1":"val1","key2":"val2"}`.||
-|`--auth_plugin`|Authentication plugin class name||
+|`--auth-plugin`|Authentication plugin class name||
|`--conf-file`|Configuration file||
|`-h`, `--help`|Help message|false|
-|`-m`, `--num-messages`|Number of messages to publish in total. If the value is 0 or smaller than 0, it keeps publishing messages|0|
+|`-m`, `--num-messages`|Number of messages to publish in total. If this value is less than or equal to 0, it keeps publishing messages.|0|
|`-t`, `--num-topic`|The number of topics|1|
|`-f`, `--payload-file`|Use payload from a file instead of empty buffer||
+|`-e`, `--payload-delimiter`|The delimiter used to split lines when using payload from a file|\n|
+|`-fp`, `--format-payload`|Format %i as a message index in the stream from producer and/or %t as the timestamp nanoseconds|false|
+|`-fc`, `--format-class`|Custom formatter class name|`org.apache.pulsar.testclient.DefaultMessageFormatter`|
|`-u`, `--proxy-url`|Pulsar Proxy URL, e.g., "ws://localhost:8080/"||
|`-r`, `--rate`|Publish rate msg/s across topics|100|
|`-s`, `--size`|Message size in byte|1024|
-|`-time`, `--test-duration`|Test duration (in seconds). If the value is 0 or smaller than 0, it keeps publishing messages|0|
+|`-time`, `--test-duration`|Test duration (in seconds). If this value is less than or equal to 0, it keeps publishing messages.|0|
### `managed-ledger`
@@ -717,11 +755,11 @@ Options
|`-h`, `--help`|Help message|false|
|`-c`, `--max-connections`|Max number of TCP connections to a single bookie|1|
|`-o`, `--max-outstanding`|Max number of outstanding requests|1000|
-|`-m`, `--num-messages`|Number of messages to publish in total. If the value is 0 or smaller than 0, it keeps publishing messages|0|
+|`-m`, `--num-messages`|Number of messages to publish in total. If this value is less than or equal to 0, it keeps publishing messages.|0|
|`-t`, `--num-topic`|Number of managed ledgers|1|
|`-r`, `--rate`|Write rate msg/s across managed ledgers|100|
|`-s`, `--size`|Message size in byte|1024|
-|`-time`, `--test-duration`|Test duration (in seconds). If the value is 0 or smaller than 0, it keeps publishing messages|0|
+|`-time`, `--test-duration`|Test duration (in seconds). If this value is less than or equal to 0, it keeps publishing messages.|0|
|`--threads`|Number of threads writing|1|
|`-w`, `--write-quorum`|Ledger write quorum|1|
|`-zk`, `--zookeeperServers`|ZooKeeper connection string||
@@ -785,6 +823,45 @@ Options
|`--cluster`|The cluster to test on||
|`-h`, `--help`|Help message|false|
+### `transaction`
+
+Run a transaction. For more information, see [Pulsar transactions](txn-why).
+
+**Usage**
+
+```bash
+
+$ pulsar-perf transaction options
+
+```
+
+**Options**
+
+|Flag|Description|Default|
+|---|---|---|
+`-au`, `--admin-url`|Pulsar admin URL.|N/A
+`--conf-file`|Configuration file.|N/A
+`-h`, `--help`|Help messages.|N/A
+`-c`, `--max-connections`|Maximum number of TCP connections to a single broker.|100
+`-ioThreads`, `--num-io-threads`|Set the number of threads to be used for handling connections to brokers. |1
+`-ns`, `--num-subscriptions`|Number of subscriptions per topic.|1
+`-threads`, `--num-test-threads`|Number of test threads. <br /><br />This thread is for a new transaction to ack messages from consumer topics, produce messages to producer topics, and commit or abort this transaction. <br /><br /> Increasing the number of threads increases the parallelism of the performance test, consequently, it increases the intensity of the stress test.|1
+`-nmc`, `--numMessage-perTransaction-consume`|Set the number of messages consumed in a transaction. <br /><br /> If transaction is disabled, it means the number of messages consumed in a task instead of in a transaction.|1
+`-nmp`, `--numMessage-perTransaction-produce`|Set the number of messages produced in a transaction. <br /><br />If transaction is disabled, it means the number of messages produced in a task instead of in a transaction.|1
+`-ntxn`, `--number-txn`|Set the number of transactions. <br /><br /> 0 means the number of transactions is unlimited. <br /><br /> If transaction is disabled, it means the number of tasks instead of transactions. |0
+`-np`, `--partitions`|Create partitioned topics with a given number of partitions. <br /><br /> 0 means not trying to create a topic.
+`-q`, `--receiver-queue-size`|Size of the receiver queue.|1000
+`-u`, `--service-url`|Pulsar service URL.|N/A
+`-sp`, `--subscription-position`|Subscription position.|Earliest
+`-st`, `--subscription-type`|Subscription type.|Shared
+`-ss`, `--subscriptions`|A list of subscriptions to consume. <br /><br /> For example, sub1,sub2.|[sub]
+`-time`, `--test-duration`|Test duration (in second). <br /><br /> 0 means keeping publishing messages.|0
+`--topics-c`|All topics assigned to consumers.|[test-consume]
+`--topics-p`|All topics assigned to producers . |[test-produce]
+`--txn-disEnable`|Disable transaction.|true
+`-tto`, `--txn-timeout`|Set the time of transaction timeout (in second). <br /><br /> If you want `--txn-timeout` takes effect, set `--txn-enable` to true.|5
+`-abort`|Abort the transaction. <br /><br /> If you want `-abort` takes effect, set `--txn-disEnable` to false.|true
+`-txnRate`|Set the rate of opened transactions or tasks. <br /><br /> 0 means no limit.|0
### `help`
This help message
@@ -809,7 +886,7 @@ $ bookkeeper command
```
Commands
-* `auto-recovery`
+* `autorecovery`
* `bookie`
* `localbookie`
* `upgrade`
@@ -829,16 +906,17 @@ The table below lists the environment variables that you can use to configure th
|ENTRY_FORMATTER_CLASS|The Java class used to format entries||
|BOOKIE_PID_DIR|Folder where the BookKeeper server PID file should be stored||
|BOOKIE_STOP_TIMEOUT|Wait time before forcefully killing the Bookie server instance if attempts to stop it are not successful||
+|BOOKIE_GC_LOG|Gc options to be passed to the jvm||
-### `auto-recovery`
-Runs an auto-recovery service daemon
+### `autorecovery`
+Runs an auto-recovery service
Usage
```bash
-$ bookkeeper auto-recovery options
+$ bookkeeper autorecovery options
```
@@ -846,7 +924,7 @@ Options
|Flag|Description|Default|
|---|---|---|
-|`-c`, `--conf`|Configuration for the auto-recovery daemon||
+|`-c`, `--conf`|Configuration for the auto-recovery||
### `bookie`
@@ -864,7 +942,7 @@ Options
|Flag|Description|Default|
|---|---|---|
-|`-c`, `--conf`|Configuration for the auto-recovery daemon||
+|`-c`, `--conf`|Configuration for the auto-recovery||
|-readOnly|Force start a read-only bookie server|false|
|-withAutoRecovery|Start auto-recovery service bookie server|false|
@@ -895,7 +973,7 @@ Options
|Flag|Description|Default|
|---|---|---|
-|`-c`, `--conf`|Configuration for the auto-recovery daemon||
+|`-c`, `--conf`|Configuration for the auto-recovery||
|`-u`, `--upgrade`|Upgrade the bookie’s directories||
diff --git a/site2/website-next/versioned_docs/version-2.4.1/reference-terminology.md b/site2/website-next/versioned_docs/version-2.4.1/reference-terminology.md
index d0e7368..ebc114d 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/reference-terminology.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/reference-terminology.md
@@ -2,7 +2,6 @@
id: reference-terminology
title: Pulsar Terminology
sidebar_label: "Terminology"
-original_id: reference-terminology
---
Here is a glossary of terms related to Apache Pulsar:
diff --git a/site2/website-next/versioned_docs/version-2.4.1/security-encryption.md b/site2/website-next/versioned_docs/version-2.4.1/security-encryption.md
index cc43082..90d0dbe 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/security-encryption.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/security-encryption.md
@@ -2,7 +2,6 @@
id: security-encryption
title: Pulsar Encryption
sidebar_label: "End-to-End Encryption"
-original_id: security-encryption
---
import Tabs from '@theme/Tabs';
diff --git a/site2/website-next/versioned_docs/version-2.4.1/security-extending.md b/site2/website-next/versioned_docs/version-2.4.1/security-extending.md
index c088e3a..dd0030e 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/security-extending.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/security-extending.md
@@ -2,7 +2,6 @@
id: security-extending
title: Extending Authentication and Authorization in Pulsar
sidebar_label: "Extending"
-original_id: security-extending
---
Pulsar provides a way to use custom authentication and authorization mechanisms.
diff --git a/site2/website-next/versioned_docs/version-2.4.1/security-jwt.md b/site2/website-next/versioned_docs/version-2.4.1/security-jwt.md
index f12c5b6..8dd8d80 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/security-jwt.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/security-jwt.md
@@ -2,7 +2,6 @@
id: security-jwt
title: Client authentication using tokens based on JSON Web Tokens
sidebar_label: "Authentication using JWT"
-original_id: security-jwt
---
import Tabs from '@theme/Tabs';
diff --git a/site2/website-next/versioned_docs/version-2.4.1/security-token-admin.md b/site2/website-next/versioned_docs/version-2.4.1/security-token-admin.md
index 6736097..9424e83 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/security-token-admin.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/security-token-admin.md
@@ -2,7 +2,6 @@
id: security-token-admin
title: Token authentication admin
sidebar_label: "Token authentication admin"
-original_id: security-token-admin
---
## Token Authentication Overview
diff --git a/site2/website-next/versioned_docs/version-2.4.1/sql-deployment-configurations.md b/site2/website-next/versioned_docs/version-2.4.1/sql-deployment-configurations.md
index 6c6fd87..9e7ff5a 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/sql-deployment-configurations.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/sql-deployment-configurations.md
@@ -2,7 +2,6 @@
id: sql-deployment-configurations
title: Pulsar SQL configuration and deployment
sidebar_label: "Configuration and deployment"
-original_id: sql-deployment-configurations
---
You can configure Presto Pulsar connector and deploy a cluster with the following instruction.
@@ -27,6 +26,84 @@ pulsar.entry-read-batch-size=100
# default number of splits to use per query
pulsar.target-num-splits=4
+# max size of one batch message (default value is 5MB)
+pulsar.max-message-size=5242880
+
+# number of split used when querying data from pulsar
+pulsar.target-num-splits=2
+
+# size of queue to buffer entry read from pulsar
+pulsar.max-split-entry-queue-size=1000
+
+# size of queue to buffer message extract from entries
+pulsar.max-split-message-queue-size=10000
+
+# status provider to record connector metrics
+pulsar.stats-provider=org.apache.bookkeeper.stats.NullStatsProvider
+
+# config in map format for stats provider e.g. {"key1":"val1","key2":"val2"}
+pulsar.stats-provider-configs={}
+
+# whether to rewrite Pulsar's default topic delimiter '/'
+pulsar.namespace-delimiter-rewrite-enable=false
+
+# delimiter used to rewrite Pulsar's default delimiter '/', use if default is causing incompatibility with other system like Superset
+pulsar.rewrite-namespace-delimiter=“/”
+
+# maximum number of thread pool size for ledger offloader.
+pulsar.managed-ledger-offload-max-threads=2
+
+# driver used to offload or read cold data to or from long-term storage
+pulsar.managed-ledger-offload-driver=null
+
+# directory to load offloaders nar file.
+pulsar.offloaders-directory="./offloaders"
+
+# properties and configurations related to specific offloader implementation as map e.g. {"key1":"val1","key2":"val2"}
+pulsar.offloader-properties={}
+
+# authentication plugin used to authenticate to Pulsar cluster
+pulsar.auth-plugin=null
+
+# authentication parameter used to authenticate to the Pulsar cluster as a string e.g. "key1:val1,key2:val2".
+pulsar.auth-params=null
+
+# whether the Pulsar client accept an untrusted TLS certificate from broker
+pulsar.tls-allow-insecure-connection=null
+
+# whether to allow hostname verification when a client connects to broker over TLS.
+pulsar.tls-hostname-verification-enable=null
+
+# path for the trusted TLS certificate file of Pulsar broker
+pulsar.tls-trust-cert-file-path=null
+
+# set the threshold for BookKeeper request throttle, default is disabled
+pulsar.bookkeeper-throttle-value=0
+
+# set the number of IO thread
+pulsar.bookkeeper-num-io-threads=2 * Runtime.getRuntime().availableProcessors()
+
+# set the number of worker thread
+pulsar.bookkeeper-num-worker-threads=Runtime.getRuntime().availableProcessors()
+
+# whether to use BookKeeper V2 wire protocol
+pulsar.bookkeeper-use-v2-protocol=true
+
+# interval to check the need for sending an explicit LAC, default is disabled
+pulsar.bookkeeper-explicit-interval=0
+
+# size for managed ledger entry cache (in MB).
+pulsar.managed-ledger-cache-size-MB=0
+
+# number of threads to be used for managed ledger tasks dispatching
+pulsar.managed-ledger-num-worker-threads=Runtime.getRuntime().availableProcessors()
+
+# number of threads to be used for managed ledger scheduled tasks
+pulsar.managed-ledger-num-scheduler-threads=Runtime.getRuntime().availableProcessors()
+
+# directory used to store extraction NAR file
+pulsar.nar-extraction-directory=System.getProperty("java.io.tmpdir")
+
```
You can connect Presto to a Pulsar cluster with multiple hosts. To configure multiple hosts for brokers, add multiple URLs to `pulsar.web-service-url`. To configure multiple hosts for ZooKeeper, add multiple URIs to `pulsar.zookeeper-uri`. The following is an example.
@@ -38,6 +115,21 @@ pulsar.zookeeper-uri=localhost1,localhost2:2181
```
+A frequently asked question is why my latest message not showing up when querying with Pulsar SQL.
+It's not a bug but controlled by a setting, by default BookKeeper LAC only advanced when subsequent entries are added.
+If there is no subsequent entries added, the last entry written will not be visible to readers until the ledger is closed.
+This is not a problem for Pulsar which uses managed ledger, but Pulsar SQL directly read from BookKeeper ledger.
+We can add following setting to change the behavior:
+In Broker config, set
+bookkeeperExplicitLacIntervalInMills > 0
+bookkeeperUseV2WireProtocol=false
+
+And in Presto config, set
+pulsar.bookkeeper-explicit-interval > 0
+pulsar.bookkeeper-use-v2-protocol=false
+
+However,keep in mind that using bk V3 protocol will introduce additional GC overhead to BK as it uses Protobuf.
+
## Query data from existing Presto clusters
If you already have a Presto cluster, you can copy the Presto Pulsar connector plugin to your existing cluster. Download the archived plugin package with the following command.
diff --git a/site2/website-next/versioned_docs/version-2.4.1/sql-getting-started.md b/site2/website-next/versioned_docs/version-2.4.1/sql-getting-started.md
index 8a5cd71..6a2d873 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/sql-getting-started.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/sql-getting-started.md
@@ -2,7 +2,6 @@
id: sql-getting-started
title: Query data with Pulsar SQL
sidebar_label: "Query data"
-original_id: sql-getting-started
---
Before querying data in Pulsar, you need to install Pulsar and built-in connectors.
diff --git a/site2/website-next/versioned_docs/version-2.4.1/sql-overview.md b/site2/website-next/versioned_docs/version-2.4.1/sql-overview.md
index 4a4d5f0..2f827f4 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/sql-overview.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/sql-overview.md
@@ -2,7 +2,6 @@
id: sql-overview
title: Pulsar SQL Overview
sidebar_label: "Overview"
-original_id: sql-overview
---
Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL)](https://trino.io/).
diff --git a/site2/website-next/versioned_docs/version-2.4.1/standalone-docker.md b/site2/website-next/versioned_docs/version-2.4.1/standalone-docker.md
index 05ac2a1..7ee20c2 100644
--- a/site2/website-next/versioned_docs/version-2.4.1/standalone-docker.md
+++ b/site2/website-next/versioned_docs/version-2.4.1/standalone-docker.md
@@ -2,14 +2,11 @@
id: standalone-docker
title: Set up a standalone Pulsar in Docker
sidebar_label: "Run Pulsar in Docker"
-original_id: standalone-docker
---
-For local development and testing, you can run Pulsar in standalone
-mode on your own machine within a Docker container.
+For local development and testing, you can run Pulsar in standalone mode on your own machine within a Docker container.
-If you have not installed Docker, download the [Community edition](https://www.docker.com/community-edition)
-and follow the instructions for your OS.
+If you have not installed Docker, download the [Community edition](https://www.docker.com/community-edition) and follow the instructions for your OS.
## Start Pulsar in Docker
@@ -17,13 +14,7 @@ and follow the instructions for your OS.
```shell
- $ docker run -it \
- -p 6650:6650 \
- -p 8080:8080 \
- --mount source=pulsardata,target=/pulsar/data \
- --mount source=pulsarconf,target=/pulsar/conf \
- apachepulsar/pulsar:@pulsar:version@ \
- bin/pulsar standalone
+ $ docker run -it -p 6650:6650 -p 8080:8080 --mount source=pulsardata,target=/pulsar/data --mount source=pulsarconf,target=/pulsar/conf apachepulsar/pulsar:@pulsar:version@ bin/pulsar standalone
```
@@ -36,8 +27,9 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th
```
-2017-08-09 22:34:04,030 - INFO - [main:WebService@213] - Web Service started at http://127.0.0.1:8080
-2017-08-09 22:34:04,038 - INFO - [main:PulsarService@335] - messaging service is ready, bootstrap service on port=8080, broker url=pulsar://127.0.0.1:6650, cluster=standalone, configs=org.apache.pulsar.broker.ServiceConfiguration@4db60246
+08:18:30.970 [main] INFO org.apache.pulsar.broker.web.WebService - HTTP Service started at http://0.0.0.0:8080
+...
+07:53:37.322 [main] INFO org.apache.pulsar.broker.PulsarService - messaging service is ready, bootstrap service port = 8080, broker url= pulsar://localhost:6650, cluster=standalone, configs=org.apache.pulsar.broker.ServiceConfiguration@98b63c1
...
```
@@ -60,7 +52,7 @@ use one of these root URLs to interact with your cluster:
* `pulsar://localhost:6650`
* `http://localhost:8080`
-The following example will guide you get started with Pulsar quickly by using the [Python](client-libraries-python)
+The following example will guide you get started with Pulsar quickly by using the [Python client API](client-libraries-python)
client API.
Install the Pulsar Python client library directly from [PyPI](https://pypi.org/project/pulsar-client/):
@@ -128,51 +120,93 @@ The output is something like this:
```json
{
- "averageMsgSize": 0.0,
- "msgRateIn": 0.0,
- "msgRateOut": 0.0,
- "msgThroughputIn": 0.0,
- "msgThroughputOut": 0.0,
- "publishers": [
- {
- "address": "/172.17.0.1:35048",
- "averageMsgSize": 0.0,
- "clientVersion": "1.19.0-incubating",
- "connectedSince": "2017-08-09 20:59:34.621+0000",
- "msgRateIn": 0.0,
- "msgThroughputIn": 0.0,
- "producerId": 0,
- "producerName": "standalone-0-1"
- }
- ],
- "replication": {},
- "storageSize": 16,
- "subscriptions": {
- "my-sub": {
- "blockedSubscriptionOnUnackedMsgs": false,
- "consumers": [
+ "msgRateIn": 0.0,
+ "msgThroughputIn": 0.0,
+ "msgRateOut": 1.8332950480217471,
+ "msgThroughputOut": 91.33142602871978,
+ "bytesInCounter": 7097,
+ "msgInCounter": 143,
+ "bytesOutCounter": 6607,
+ "msgOutCounter": 133,
+ "averageMsgSize": 0.0,
+ "msgChunkPublished": false,
+ "storageSize": 7097,
+ "backlogSize": 0,
+ "offloadedStorageSize": 0,
+ "publishers": [
{
- "address": "/172.17.0.1:35064",
- "availablePermits": 996,
- "blockedConsumerOnUnackedMsgs": false,
- "clientVersion": "1.19.0-incubating",
- "connectedSince": "2017-08-09 21:05:39.222+0000",
- "consumerName": "166111",
- "msgRateOut": 0.0,
- "msgRateRedeliver": 0.0,
- "msgThroughputOut": 0.0,
- "unackedMessages": 0
+ "accessMode": "Shared",
+ "msgRateIn": 0.0,
+ "msgThroughputIn": 0.0,
+ "averageMsgSize": 0.0,
+ "chunkedMessageRate": 0.0,
+ "producerId": 0,
+ "metadata": {},
+ "address": "/127.0.0.1:35604",
+ "connectedSince": "2021-07-04T09:05:43.04788Z",
+ "clientVersion": "2.8.0",
+ "producerName": "standalone-2-5"
+ }
+ ],
+ "waitingPublishers": 0,
+ "subscriptions": {
+ "my-sub": {
+ "msgRateOut": 1.8332950480217471,
+ "msgThroughputOut": 91.33142602871978,
+ "bytesOutCounter": 6607,
+ "msgOutCounter": 133,
+ "msgRateRedeliver": 0.0,
+ "chunkedMessageRate": 0,
+ "msgBacklog": 0,
+ "backlogSize": 0,
+ "msgBacklogNoDelayed": 0,
+ "blockedSubscriptionOnUnackedMsgs": false,
+ "msgDelayed": 0,
+ "unackedMessages": 0,
+ "type": "Exclusive",
+ "activeConsumerName": "3c544f1daa",
+ "msgRateExpired": 0.0,
+ "totalMsgExpired": 0,
+ "lastExpireTimestamp": 0,
+ "lastConsumedFlowTimestamp": 1625389101290,
+ "lastConsumedTimestamp": 1625389546070,
+ "lastAckedTimestamp": 1625389546162,
+ "lastMarkDeleteAdvancedTimestamp": 1625389546163,
+ "consumers": [
+ {
+ "msgRateOut": 1.8332950480217471,
+ "msgThroughputOut": 91.33142602871978,
+ "bytesOutCounter": 6607,
+ "msgOutCounter": 133,
+ "msgRateRedeliver": 0.0,
+ "chunkedMessageRate": 0.0,
+ "consumerName": "3c544f1daa",
+ "availablePermits": 867,
+ "unackedMessages": 0,
+ "avgMessagesPerEntry": 6,
+ "blockedConsumerOnUnackedMsgs": false,
+ "lastAckedTimestamp": 1625389546162,
+ "lastConsumedTimestamp": 1625389546070,
+ "metadata": {},
+ "address": "/127.0.0.1:35472",
+ "connectedSince": "2021-07-04T08:58:21.287682Z",
+ "clientVersion": "2.8.0"
+ }
+ ],
+ "isDurable": true,
+ "isReplicated": false,
+ "allowOutOfOrderDelivery": false,
+ "consumersAfterMarkDeletePosition": {},
+ "nonContiguousDeletedMessagesRanges": 0,
+ "nonContiguousDeletedMessagesRangesSerializedSize": 0,
+ "durable": true,
+ "replicated": false
}
- ],
- "msgBacklog": 0,
- "msgRateExpired": 0.0,
- "msgRateOut": 0.0,
- "msgRateRedeliver": 0.0,
- "msgThroughputOut": 0.0,
- "type": "Exclusive",
- "unackedMessages": 0
- }
- }
+ },
+ "replication": {},
+ "deduplicationStatus": "Disabled",
+ "nonContiguousDeletedMessagesRanges": 0,
+ "nonContiguousDeletedMessagesRangesSerializedSize": 0
}
```
diff --git a/site2/website-next/versioned_docs/version-2.4.1/getting-started-standalone.md b/site2/website-next/versioned_docs/version-2.4.1/standalone.md
similarity index 100%
rename from site2/website-next/versioned_docs/version-2.4.1/getting-started-standalone.md
rename to site2/website-next/versioned_docs/version-2.4.1/standalone.md
diff --git a/site2/website-next/versioned_docs/version-2.4.2/adaptors-kafka.md b/site2/website-next/versioned_docs/version-2.4.2/adaptors-kafka.md
index ad0d886..27382e9 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/adaptors-kafka.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/adaptors-kafka.md
@@ -2,7 +2,6 @@
id: adaptors-kafka
title: Pulsar adaptor for Apache Kafka
sidebar_label: "Kafka client wrapper"
-original_id: adaptors-kafka
---
@@ -261,6 +260,7 @@ You can configure Pulsar authentication provider directly from the Kafka propert
| [`pulsar.producer.batching.enabled`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ProducerConfiguration.html#setBatchingEnabled-boolean-) | `true` | Control whether automatic batching of messages is enabled for the producer. |
| [`pulsar.producer.batching.max.messages`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ProducerConfiguration.html#setBatchingMaxMessages-int-) | `1000` | The maximum number of messages in a batch. |
| [`pulsar.block.if.producer.queue.full`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ProducerConfiguration.html#setBlockIfQueueFull-boolean-) | | Specify the block producer if queue is full. |
+| [`pulsar.crypto.reader.factory.class.name`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ProducerConfiguration.html#setCryptoKeyReader-org.apache.pulsar.client.api.CryptoKeyReader-) | | Specify the CryptoReader-Factory(`CryptoKeyReaderFactory`) classname which allows producer to create CryptoKeyReader. |
### Pulsar consumer Properties
@@ -272,3 +272,4 @@ You can configure Pulsar authentication provider directly from the Kafka propert
| [`pulsar.consumer.acknowledgments.group.time.millis`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html#acknowledgmentGroupTime-long-java.util.concurrent.TimeUnit-) | 100 | Set the maximum amount of group time for consumers to send the acknowledgments to the broker. |
| [`pulsar.consumer.total.receiver.queue.size.across.partitions`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerConfiguration.html#setMaxTotalReceiverQueueSizeAcrossPartitions-int-) | 50000 | Set the maximum size of the total receiver queue across partitions. |
| [`pulsar.consumer.subscription.topics.mode`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html#subscriptionTopicsMode-Mode-) | PersistentOnly | Set the subscription topic mode for consumers. |
+| [`pulsar.crypto.reader.factory.class.name`](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ProducerConfiguration.html#setCryptoKeyReader-org.apache.pulsar.client.api.CryptoKeyReader-) | | Specify the CryptoReader-Factory(`CryptoKeyReaderFactory`) classname which allows consumer to create CryptoKeyReader. |
diff --git a/site2/website-next/versioned_docs/version-2.4.2/adaptors-spark.md b/site2/website-next/versioned_docs/version-2.4.2/adaptors-spark.md
index e14f13b..afa5a7e 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/adaptors-spark.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/adaptors-spark.md
@@ -2,7 +2,6 @@
id: adaptors-spark
title: Pulsar adaptor for Apache Spark
sidebar_label: "Apache Spark"
-original_id: adaptors-spark
---
## Spark Streaming receiver
diff --git a/site2/website-next/versioned_docs/version-2.4.2/adaptors-storm.md b/site2/website-next/versioned_docs/version-2.4.2/adaptors-storm.md
index 76d5071..9df9076 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/adaptors-storm.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/adaptors-storm.md
@@ -2,7 +2,6 @@
id: adaptors-storm
title: Pulsar adaptor for Apache Storm
sidebar_label: "Apache Storm"
-original_id: adaptors-storm
---
Pulsar Storm is an adaptor for integrating with [Apache Storm](http://storm.apache.org/) topologies. It provides core Storm implementations for sending and receiving data.
diff --git a/site2/website-next/versioned_docs/version-2.4.2/admin-api-brokers.md b/site2/website-next/versioned_docs/version-2.4.2/admin-api-brokers.md
index dbac453..10a90ca 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/admin-api-brokers.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/admin-api-brokers.md
@@ -2,7 +2,6 @@
id: admin-api-brokers
title: Managing Brokers
sidebar_label: "Brokers"
-original_id: admin-api-brokers
---
import Tabs from '@theme/Tabs';
@@ -26,9 +25,9 @@ Pulsar brokers consist of two components:
[Brokers](reference-terminology.md#broker) can be managed via:
-* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin) tool
+* The `brokers` command of the [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) tool
* The `/admin/v2/brokers` endpoint of the admin {@inject: rest:REST:/} API
-* The `brokers` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin.html} object in the [Java API](client-libraries-java)
+* The `brokers` method of the `PulsarAdmin` object in the [Java API](client-libraries-java)
In addition to being configurable when you start them up, brokers can also be [dynamically configured](#dynamic-broker-configuration).
diff --git a/site2/website-next/versioned_docs/version-2.4.2/admin-api-clusters.md b/site2/website-next/versioned_docs/version-2.4.2/admin-api-clusters.md
index 972c7e1..8687ae6 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/admin-api-clusters.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/admin-api-clusters.md
@@ -2,7 +2,6 @@
id: admin-api-clusters
title: Managing Clusters
sidebar_label: "Clusters"
-original_id: admin-api-clusters
---
import Tabs from '@theme/Tabs';
@@ -13,7 +12,7 @@ import TabItem from '@theme/TabItem';
>
> This page only shows **some frequently used operations**.
>
-> - For the latest and complete information about `Pulsar admin`, including commands, flags, descriptions, and more information, see [Pulsar admin doc](https://pulsar.apache.org/tools/pulsar-admin/).
+> - For the latest and complete information about `Pulsar admin`, including commands, flags, descriptions, and more, see [Pulsar admin doc](https://pulsar.apache.org/tools/pulsar-admin/)
>
> - For the latest and complete information about `REST API`, including parameters, responses, samples, and more, see {@inject: rest:REST:/} API doc.
>
@@ -24,9 +23,9 @@ servers (aka [bookies](reference-terminology.md#bookie)), and a [ZooKeeper](http
Clusters can be managed via:
-* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin) tool
+* The `clusters` command of the [`pulsar-admin`]([reference-pulsar-admin.md](https://pulsar.apache.org/tools/pulsar-admin/)) tool
* The `/admin/v2/clusters` endpoint of the admin {@inject: rest:REST:/} API
-* The `clusters` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java)
+* The `clusters` method of the `PulsarAdmin` object in the [Java API](client-libraries-java)
## Clusters resources
diff --git a/site2/website-next/versioned_docs/version-2.4.2/admin-api-namespaces.md b/site2/website-next/versioned_docs/version-2.4.2/admin-api-namespaces.md
index 216cb6f..c53fa3c 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/admin-api-namespaces.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/admin-api-namespaces.md
@@ -2,7 +2,6 @@
id: admin-api-namespaces
title: Managing Namespaces
sidebar_label: "Namespaces"
-original_id: admin-api-namespaces
---
import Tabs from '@theme/Tabs';
@@ -23,9 +22,9 @@ Pulsar [namespaces](reference-terminology.md#namespace) are logical groupings of
Namespaces can be managed via:
-* The [`namespaces`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin) tool
+* The `namespaces` command of the [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) tool
* The `/admin/v2/namespaces` endpoint of the admin {@inject: rest:REST:/} API
-* The `namespaces` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java)
+* The `namespaces` method of the `PulsarAdmin` object in the [Java API](client-libraries-java)
## Namespaces resources
@@ -49,8 +48,12 @@ $ pulsar-admin namespaces create test-tenant/test-namespace
</TabItem>
<TabItem value="REST API">
+```
+
{@inject: endpoint|PUT|/admin/v2/namespaces/:tenant/:namespace|operation/createNamespace?version=@pulsar:version_number@}
+```
+
</TabItem>
<TabItem value="Java">
@@ -105,8 +108,12 @@ $ pulsar-admin namespaces policies test-tenant/test-namespace
</TabItem>
<TabItem value="REST API">
+```
+
{@inject: endpoint|GET|/admin/v2/namespaces/:tenant/:namespace|operation/getPolicies?version=@pulsar:version_number@}
+```
+
</TabItem>
<TabItem value="Java">
@@ -142,8 +149,12 @@ test-tenant/ns2
</TabItem>
<TabItem value="REST API">
+```
+
{@inject: endpoint|GET|/admin/v2/namespaces/:tenant|operation/getTenantNamespaces?version=@pulsar:version_number@}
+```
+
</TabItem>
<TabItem value="Java">
@@ -177,8 +188,12 @@ $ pulsar-admin namespaces delete test-tenant/ns1
</TabItem>
<TabItem value="REST API">
+```
+
{@inject: endpoint|DELETE|/admin/v2/namespaces/:tenant/:namespace|operation/deleteNamespace?version=@pulsar:version_number@}
+```
+
</TabItem>
<TabItem value="Java">
@@ -196,7 +211,7 @@ admin.namespaces().deleteNamespace(namespace);
#### Set replication cluster
-It sets replication clusters for a namespace, so Pulsar can internally replicate publish message from one colo to another colo.
+You can set replication clusters for a namespace to enable Pulsar to internally replicate the published messages from one colocation facility to another.
<Tabs
defaultValue="pulsar-admin"
@@ -234,7 +249,7 @@ admin.namespaces().setNamespaceReplicationClusters(namespace, clusters);
#### Get replication cluster
-It gives a list of replication clusters for a given namespace.
+You can get the list of replication clusters for a given namespace.
<Tabs
defaultValue="pulsar-admin"
@@ -281,13 +296,13 @@ admin.namespaces().getNamespaceReplicationClusters(namespace)
Backlog quota helps the broker to restrict bandwidth/storage of a namespace once it reaches a certain threshold limit. Admin can set the limit and take corresponding action after the limit is reached.
- 1. producer_request_hold: broker will hold and not persist produce request payload
+ 1. producer_request_hold: broker holds but not persists produce request payload
- 2. producer_exception: broker disconnects with the client by giving an exception.
+ 2. producer_exception: broker disconnects with the client by giving an exception
- 3. consumer_backlog_eviction: broker will start discarding backlog messages
+ 3. consumer_backlog_eviction: broker starts discarding backlog messages
- Backlog quota restriction can be taken care by defining restriction of backlog-quota-type: destination_storage
+Backlog quota restriction can be taken care by defining restriction of backlog-quota-type: destination_storage.
<Tabs
defaultValue="pulsar-admin"
@@ -300,12 +315,6 @@ $ pulsar-admin namespaces set-backlog-quota --limit 10G --limitTime 36000 --poli
```
-```
-
-N/A
-
-```
-
</TabItem>
<TabItem value="REST API">
@@ -330,7 +339,7 @@ admin.namespaces().setBacklogQuota(namespace, new BacklogQuota(limit, limitTime,
#### Get backlog quota policies
-It shows a configured backlog quota for a given namespace.
+You can get a configured backlog quota for a given namespace.
<Tabs
defaultValue="pulsar-admin"
@@ -378,7 +387,7 @@ admin.namespaces().getBacklogQuotaMap(namespace);
#### Remove backlog quota policies
-It removes backlog quota policies for a given namespace
+You can remove backlog quota policies for a given namespace.
<Tabs
defaultValue="pulsar-admin"
@@ -391,12 +400,6 @@ $ pulsar-admin namespaces remove-backlog-quota test-tenant/ns1
```
-```
-
-N/A
-
-```
-
</TabItem>
<TabItem value="REST API">
@@ -423,7 +426,7 @@ admin.namespaces().removeBacklogQuota(namespace, backlogQuotaType)
#### Set persistence policies
-Persistence policies allow to configure persistency-level for all topic messages under a given namespace.
+Persistence policies allow users to configure persistency-level for all topic messages under a given namespace.
- Bookkeeper-ack-quorum: Number of acks (guaranteed copies) to wait for each entry, default: 0
@@ -444,12 +447,6 @@ $ pulsar-admin namespaces set-persistence --bookkeeper-ack-quorum 2 --bookkeeper
```
-```
-
-N/A
-
-```
-
</TabItem>
<TabItem value="REST API">
@@ -474,7 +471,7 @@ admin.namespaces().setPersistence(namespace,new PersistencePolicies(bookkeeperEn
#### Get persistence policies
-It shows the configured persistence policies of a given namespace.
+You can get the configured persistence policies of a given namespace.
<Tabs
defaultValue="pulsar-admin"
@@ -537,12 +534,6 @@ $ pulsar-admin namespaces unload --bundle 0x00000000_0xffffffff test-tenant/ns1
```
-```
-
-N/A
-
-```
-
</TabItem>
<TabItem value="REST API">
@@ -567,8 +558,7 @@ admin.namespaces().unloadNamespaceBundle(namespace, bundle)
#### Split namespace bundles
-Each namespace bundle can contain multiple topics and each bundle can be served by only one broker.
-If a single bundle is creating an excessive load on a broker, an admin splits the bundle using this command permitting one or more of the new bundles to be unloaded thus spreading the load across the brokers.
+One namespace bundle can contain multiple topics but can be served by only one broker. If a single bundle is creating an excessive load on a broker, an admin can split the bundle using the command below, permitting one or more of the new bundles to be unloaded, thus balancing the load across the brokers.
<Tabs
defaultValue="pulsar-admin"
@@ -581,12 +571,6 @@ $ pulsar-admin namespaces split-bundle --bundle 0x00000000_0xffffffff test-tenan
```
-```
-
-N/A
-
-```
-
</TabItem>
<TabItem value="REST API">
@@ -613,7 +597,7 @@ admin.namespaces().splitNamespaceBundle(namespace, bundle)
#### Set message-ttl
-It configures message’s time to live (in seconds) duration.
+You can configure the time to live (in seconds) duration for messages. In the example below, the message-ttl is set as 100s.
<Tabs
defaultValue="pulsar-admin"
@@ -626,12 +610,6 @@ $ pulsar-admin namespaces set-message-ttl --messageTTL 100 test-tenant/ns1
```
-```
-
-N/A
-
-```
-
</TabItem>
<TabItem value="REST API">
@@ -656,7 +634,7 @@ admin.namespaces().setNamespaceMessageTTL(namespace, messageTTL)
#### Get message-ttl
-It gives a message ttl of configured namespace.
+When the message-ttl for a namespace is set, you can use the command below to get the configured value. This example comtinues the example of the command `set message-ttl`, so the returned value is 100(s).
<Tabs
defaultValue="pulsar-admin"
@@ -684,6 +662,12 @@ $ pulsar-admin namespaces get-message-ttl test-tenant/ns1
```
+```
+
+100
+
+```
+
</TabItem>
<TabItem value="Java">
@@ -693,6 +677,12 @@ admin.namespaces().getNamespaceMessageTTL(namespace)
```
+```
+
+100
+
+```
+
</TabItem>
</Tabs>
@@ -712,12 +702,6 @@ $ pulsar-admin namespaces remove-message-ttl test-tenant/ns1
```
-```
-
-100
-
-```
-
</TabItem>
<TabItem value="REST API">
@@ -758,12 +742,6 @@ $ pulsar-admin namespaces clear-backlog --sub my-subscription test-tenant/ns1
```
-```
-
-N/A
-
-```
-
</TabItem>
<TabItem value="REST API">
@@ -801,12 +779,6 @@ $ pulsar-admin namespaces clear-backlog --bundle 0x00000000_0xffffffff --sub m
```
-```
-
-N/A
-
-```
-
</TabItem>
<TabItem value="REST API">
@@ -842,13 +814,7 @@ Each namespace contains multiple topics and the retention size (storage size) of
```
-$ pulsar-admin set-retention --size 100 --time 10 test-tenant/ns1
-
-```
-
-```
-
-N/A
+$ pulsar-admin namespaces set-retention --size 100 --time 10 test-tenant/ns1
```
@@ -932,9 +898,7 @@ disables the throttling.
:::note
- If neither `clusterDispatchRate` nor `topicDispatchRate` is configured, dispatch throttling is disabled.
->
- If `topicDispatchRate` is not configured, `clusterDispatchRate` takes effect.
->
- If `topicDispatchRate` is configured, `topicDispatchRate` takes effect.
:::
diff --git a/site2/website-next/versioned_docs/version-2.4.2/admin-api-permissions.md b/site2/website-next/versioned_docs/version-2.4.2/admin-api-permissions.md
index e2ca469..faedbf1 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/admin-api-permissions.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/admin-api-permissions.md
@@ -2,7 +2,6 @@
id: admin-api-permissions
title: Managing permissions
sidebar_label: "Permissions"
-original_id: admin-api-permissions
---
import Tabs from '@theme/Tabs';
@@ -13,7 +12,7 @@ import TabItem from '@theme/TabItem';
>
> This page only shows **some frequently used operations**.
>
-> - For the latest and complete information about `Pulsar admin`, including commands, flags, descriptions, and more information, see [Pulsar admin doc](https://pulsar.apache.org/tools/pulsar-admin/).
+> - For the latest and complete information about `Pulsar admin`, including commands, flags, descriptions, and more, see [Pulsar admin doc](https://pulsar.apache.org/tools/pulsar-admin/)
>
> - For the latest and complete information about `REST API`, including parameters, responses, samples, and more, see {@inject: rest:REST:/} API doc.
>
diff --git a/site2/website-next/versioned_docs/version-2.4.2/admin-api-persistent-topics.md b/site2/website-next/versioned_docs/version-2.4.2/admin-api-persistent-topics.md
index b6d293b..8a7abae 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/admin-api-persistent-topics.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/admin-api-persistent-topics.md
@@ -2,7 +2,6 @@
id: admin-api-persistent-topics
title: Managing persistent topics
sidebar_label: "Persistent topics"
-original_id: admin-api-persistent-topics
---
For details of the content, refer to [manage topics](admin-api-topics).
\ No newline at end of file
diff --git a/site2/website-next/versioned_docs/version-2.4.2/admin-api-schemas.md b/site2/website-next/versioned_docs/version-2.4.2/admin-api-schemas.md
index 9ffe21f..8399a03 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/admin-api-schemas.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/admin-api-schemas.md
@@ -2,6 +2,5 @@
id: admin-api-schemas
title: Managing Schemas
sidebar_label: "Schemas"
-original_id: admin-api-schemas
---
diff --git a/site2/website-next/versioned_docs/version-2.4.2/admin-api-tenants.md b/site2/website-next/versioned_docs/version-2.4.2/admin-api-tenants.md
index fe68336..570ac31 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/admin-api-tenants.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/admin-api-tenants.md
@@ -2,7 +2,6 @@
id: admin-api-tenants
title: Managing Tenants
sidebar_label: "Tenants"
-original_id: admin-api-tenants
---
import Tabs from '@theme/Tabs';
@@ -13,7 +12,7 @@ import TabItem from '@theme/TabItem';
>
> This page only shows **some frequently used operations**.
>
-> - For the latest and complete information about `Pulsar admin`, including commands, flags, descriptions, and more information, see [Pulsar admin doc](https://pulsar.apache.org/tools/pulsar-admin/).
+> - For the latest and complete information about `Pulsar admin`, including commands, flags, descriptions, and more, see [Pulsar admin doc](https://pulsar.apache.org/tools/pulsar-admin/)
>
> - For the latest and complete information about `REST API`, including parameters, responses, samples, and more, see {@inject: rest:REST:/} API doc.
>
@@ -80,22 +79,26 @@ $ pulsar-admin tenants create my-tenant
```
-When creating a tenant, you can assign admin roles using the `-r`/`--admin-roles` flag. You can specify multiple roles as a comma-separated list. Here are some examples:
+When creating a tenant, you can optionally assign admin roles using the `-r`/`--admin-roles`
+flag, and clusters using the `-c`/`--allowed-clusters` flag. You can specify multiple values
+as a comma-separated list. Here are some examples:
```shell
$ pulsar-admin tenants create my-tenant \
- --admin-roles role1,role2,role3
+ --admin-roles role1,role2,role3 \
+ --allowed-clusters cluster1
$ pulsar-admin tenants create my-tenant \
-r role1
+ -c cluster1
```
</TabItem>
<TabItem value="REST API">
-{@inject: endpoint|POST|/admin/v2/tenants/:tenant|operation/createTenant?version=@pulsar:version_number@}
+{@inject: endpoint|PUT|/admin/v2/tenants/:tenant|operation/createTenant?version=@pulsar:version_number@}
</TabItem>
<TabItem value="JAVA">
@@ -140,7 +143,7 @@ $ pulsar-admin tenants get my-tenant
</TabItem>
<TabItem value="REST API">
-{@inject: endpoint|GET|/admin/v2/tenants/:cluster|operation/getTenant?version=@pulsar:version_number@}
+{@inject: endpoint|GET|/admin/v2/tenants/:tenant|operation/getTenant?version=@pulsar:version_number@}
</TabItem>
<TabItem value="JAVA">
@@ -175,7 +178,7 @@ $ pulsar-admin tenants delete my-tenant
</TabItem>
<TabItem value="REST API">
-{@inject: endpoint|DELETE|/admin/v2/tenants/:cluster|operation/deleteTenant?version=@pulsar:version_number@}
+{@inject: endpoint|DELETE|/admin/v2/tenants/:tenant|operation/deleteTenant?version=@pulsar:version_number@}
</TabItem>
<TabItem value="JAVA">
@@ -210,7 +213,7 @@ $ pulsar-admin tenants update my-tenant
</TabItem>
<TabItem value="REST API">
-{@inject: endpoint|DELETE|/admin/v2/tenants/:cluster|operation/updateTenant?version=@pulsar:version_number@}
+{@inject: endpoint|POST|/admin/v2/tenants/:tenant|operation/updateTenant?version=@pulsar:version_number@}
</TabItem>
<TabItem value="JAVA">
diff --git a/site2/website-next/versioned_docs/version-2.4.2/administration-geo.md b/site2/website-next/versioned_docs/version-2.4.2/administration-geo.md
index 037be9c..3a85608 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/administration-geo.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/administration-geo.md
@@ -166,4 +166,4 @@ Consumer<String> consumer = client.newConsumer(Schema.STRING)
### Limitations
* When you enable replicated subscription, you're creating a consistent distributed snapshot to establish an association between message ids from different clusters. The snapshots are taken periodically. The default value is `1 second`. It means that a consumer failing over to a different cluster can potentially receive 1 second of duplicates. You can also configure the frequency of the snapshot in the `broker.conf` file.
-* Only the base line cursor position is synced in replicated subscriptions while the individual acknowledgments are not synced. This means the messages acknowledged out-of-order could end up getting delivered again, in the case of a cluster failover.
+* Only the base line cursor position is synced in replicated subscriptions while the individual acknowledgments are not synced. This means the messages acknowledged out-of-order could end up getting delivered again, in the case of a cluster failover.
\ No newline at end of file
diff --git a/site2/website-next/versioned_docs/version-2.4.2/administration-load-balance.md b/site2/website-next/versioned_docs/version-2.4.2/administration-load-balance.md
index 3efba60..834b156 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/administration-load-balance.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/administration-load-balance.md
@@ -2,13 +2,11 @@
id: administration-load-balance
title: Pulsar load balance
sidebar_label: "Load balance"
-original_id: administration-load-balance
---
## Load balance across Pulsar brokers
-Pulsar is an horizontally scalable messaging system, so the traffic
-in a logical cluster must be spread across all the available Pulsar brokers as evenly as possible, which is a core requirement.
+Pulsar is an horizontally scalable messaging system, so the traffic in a logical cluster must be balanced across all the available Pulsar brokers as evenly as possible, which is a core requirement.
You can use multiple settings and tools to control the traffic distribution which require a bit of context to understand how the traffic is managed in Pulsar. Though, in most cases, the core requirement mentioned above is true out of the box and you should not worry about it.
@@ -36,11 +34,9 @@ Instead of individual topic or partition assignment, each broker takes ownership
The namespace is the "administrative" unit: many config knobs or operations are done at the namespace level.
-For assignment, a namespaces is sharded into a list of "bundles", with each bundle comprising
-a portion of overall hash range of the namespace.
+For assignment, a namespaces is sharded into a list of "bundles", with each bundle comprising a portion of overall hash range of the namespace.
-Topics are assigned to a particular bundle by taking the hash of the topic name and checking in which
-bundle the hash falls into.
+Topics are assigned to a particular bundle by taking the hash of the topic name and checking in which bundle the hash falls into.
Each bundle is independent of the others and thus is independently assigned to different brokers.
@@ -72,8 +68,7 @@ On the same note, it is beneficial to start with more bundles than the number of
### Unload topics and bundles
-You can "unload" a topic in Pulsar with admin operation. Unloading means to close the topics,
-release ownership and reassign the topics to a new broker, based on current load.
+You can "unload" a topic in Pulsar with admin operation. Unloading means to close the topics, release ownership and reassign the topics to a new broker, based on current load.
When unloading happens, the client experiences a small latency blip, typically in the order of tens of milliseconds, while the topic is reassigned.
@@ -97,9 +92,11 @@ pulsar-admin namespaces unload tenant/namespace
### Split namespace bundles
-Since the load for the topics in a bundle might change over time, or predicting upfront might just be hard, brokers can split bundles into two. The new smaller bundles can be reassigned to different brokers.
+Since the load for the topics in a bundle might change over time and predicting the load might be hard, bundle split is designed to deal with these issues. The broker splits a bundle into two and the new smaller bundles can be reassigned to different brokers.
-The splitting happens based on some tunable thresholds. Any existing bundle that exceeds any of the threshold is a candidate to be split. By default the newly split bundles are also immediately offloaded to other brokers, to facilitate the traffic distribution.
+The splitting is based on some tunable thresholds. Any existing bundle that exceeds any of the threshold is a candidate to be split. By default the newly split bundles are also immediately offloaded to other brokers, to facilitate the traffic distribution.
+
+You can split namespace bundles in two ways, by setting `supportedNamespaceBundleSplitAlgorithms` to `range_equally_divide` or `topic_count_equally_divide` in `broker.conf` file. The former splits the bundle into two parts with the same hash range size; the latter splits the bundle into two parts with the same number of topics. You can also configure other parameters for namespace bundles.
```properties
@@ -130,13 +127,11 @@ loadBalancerNamespaceMaximumBundles=128
The support for automatic load shedding is available in the load manager of Pulsar. This means that whenever the system recognizes a particular broker is overloaded, the system forces some traffic to be reassigned to less loaded brokers.
-When a broker is identified as overloaded, the broker forces to "unload" a subset of the bundles, the
-ones with higher traffic, that make up for the overload percentage.
+When a broker is identified as overloaded, the broker forces to "unload" a subset of the bundles, the ones with higher traffic, that make up for the overload percentage.
For example, the default threshold is 85% and if a broker is over quota at 95% CPU usage, then the broker unloads the percent difference plus a 5% margin: `(95% - 85%) + 5% = 15%`.
-Given the selection of bundles to offload is based on traffic (as a proxy measure for cpu, network
-and memory), broker unloads bundles for at least 15% of traffic.
+Given the selection of bundles to offload is based on traffic (as a proxy measure for cpu, network and memory), broker unloads bundles for at least 15% of traffic.
The automatic load shedding is enabled by default and you can disable the automatic load shedding with this setting:
@@ -160,6 +155,20 @@ loadBalancerSheddingGracePeriodMinutes=30
```
+Pulsar supports three types of shedding strategies:
+
+##### ThresholdShedder
+This strategy tends to shed the bundles if any broker's usage is above the configured threshold. It does this by first computing the average resource usage per broker for the whole cluster. The resource usage for each broker is calculated using the following method: LocalBrokerData#getMaxResourceUsageWithWeight). The weights for each resource are configurable. Historical observations are included in the running average based on the broker's setting for loadBalancerHistoryResourcePercenta [...]
+`loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder`
+
+##### OverloadShedder
+This strategy will attempt to shed exactly one bundle on brokers which are overloaded, that is, whose maximum system resource usage exceeds loadBalancerBrokerOverloadedThresholdPercentage. To see which resources are considered when determining the maximum system resource. A bundle is recommended for unloading off that broker if and only if the following conditions hold: The broker has at least two bundles assigned and the broker has at least one bundle that has not been unloaded recently [...]
+`loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.impl.OverloadShedder`
+
+##### UniformLoadShedder
+This strategy tends to distribute load uniformly across all brokers. This strategy checks laod difference between broker with highest load and broker with lowest load. If the difference is higher than configured thresholds `loadBalancerMsgRateDifferenceShedderThreshold` and `loadBalancerMsgThroughputMultiplierDifferenceShedderThreshold` then it finds out bundles which can be unloaded to distribute traffic evenly across all brokers. Configure broker with below value to use this strategy.
+`loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.impl.UniformLoadShedder`
+
#### Broker overload thresholds
The determinations of when a broker is overloaded is based on threshold of CPU, network and memory usage. Whenever either of those metrics reaches the threshold, the system triggers the shedding (if enabled).
@@ -175,9 +184,7 @@ loadBalancerBrokerOverloadedThresholdPercentage=85
Pulsar gathers the usage stats from the system metrics.
-In case of network utilization, in some cases the network interface speed that Linux reports is
-not correct and needs to be manually overridden. This is the case in AWS EC2 instances with 1Gbps
-NIC speed for which the OS reports 10Gbps speed.
+In case of network utilization, in some cases the network interface speed that Linux reports is not correct and needs to be manually overridden. This is the case in AWS EC2 instances with 1Gbps NIC speed for which the OS reports 10Gbps speed.
Because of the incorrect max speed, the Pulsar load manager might think the broker has not reached the NIC capacity, while in fact the broker already uses all the bandwidth and the traffic is slowed down.
diff --git a/site2/website-next/versioned_docs/version-2.4.2/administration-proxy.md b/site2/website-next/versioned_docs/version-2.4.2/administration-proxy.md
index c046ed3..3cef937 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/administration-proxy.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/administration-proxy.md
@@ -2,10 +2,9 @@
id: administration-proxy
title: Pulsar proxy
sidebar_label: "Pulsar proxy"
-original_id: administration-proxy
---
-Pulsar proxy is an optional gateway. Pulsar proxy is used when direction connections between clients and Pulsar brokers are either infeasible or undesirable. For example, when you run Pulsar in a cloud environment or on [Kubernetes](https://kubernetes.io) or an analogous platform, you can run Pulsar proxy.
+Pulsar proxy is an optional gateway. Pulsar proxy is used when direct connections between clients and Pulsar brokers are either infeasible or undesirable. For example, when you run Pulsar in a cloud environment or on [Kubernetes](https://kubernetes.io) or an analogous platform, you can run Pulsar proxy.
## Configure the proxy
diff --git a/site2/website-next/versioned_docs/version-2.4.2/administration-stats.md b/site2/website-next/versioned_docs/version-2.4.2/administration-stats.md
index ac0c036..2ccd73c 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/administration-stats.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/administration-stats.md
@@ -2,7 +2,6 @@
id: administration-stats
title: Pulsar stats
sidebar_label: "Pulsar statistics"
-original_id: administration-stats
---
## Partitioned topics
diff --git a/site2/website-next/versioned_docs/version-2.4.2/administration-zk-bk.md b/site2/website-next/versioned_docs/version-2.4.2/administration-zk-bk.md
index de10d50..e5f9688 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/administration-zk-bk.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/administration-zk-bk.md
@@ -2,7 +2,6 @@
id: administration-zk-bk
title: ZooKeeper and BookKeeper administration
sidebar_label: "ZooKeeper and BookKeeper"
-original_id: administration-zk-bk
---
Pulsar relies on two external systems for essential tasks:
diff --git a/site2/website-next/versioned_docs/version-2.4.2/client-libraries-cpp.md b/site2/website-next/versioned_docs/version-2.4.2/client-libraries-cpp.md
index 333ec67..958861a 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/client-libraries-cpp.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/client-libraries-cpp.md
@@ -2,7 +2,6 @@
id: client-libraries-cpp
title: Pulsar C++ client
sidebar_label: "C++"
-original_id: client-libraries-cpp
---
You can use Pulsar C++ client to create Pulsar producers and consumers in C++.
@@ -11,7 +10,7 @@ All the methods in producer, consumer, and reader of a C++ client are thread-saf
## Supported platforms
-Pulsar C++ client is supported on **Linux** and **MacOS** platforms.
+Pulsar C++ client is supported on **Linux** ,**MacOS** and **Windows** platforms.
[Doxygen](http://www.doxygen.nl/)-generated API docs for the C++ client are available [here](/api/cpp).
@@ -21,8 +20,8 @@ You need to install the following components before using the C++ client:
* [CMake](https://cmake.org/)
* [Boost](http://www.boost.org/)
-* [Protocol Buffers](https://developers.google.com/protocol-buffers/) 2.6
-* [libcurl](https://curl.haxx.se/libcurl/)
+* [Protocol Buffers](https://developers.google.com/protocol-buffers/) >= 3
+* [libcurl](https://curl.se/libcurl/)
* [Google Test](https://github.com/google/googletest)
## Linux
@@ -147,6 +146,12 @@ $ rpm -ivh apache-pulsar-client*.rpm
After you install RPM successfully, Pulsar libraries are in the `/usr/lib` directory.
+:::note
+
+If you get the error that `libpulsar.so: cannot open shared object file: No such file or directory` when starting Pulsar client, you may need to run `ldconfig` first.
+
+:::
+
### Install Debian
1. Download a Debian package from the links in the table.
@@ -236,10 +241,8 @@ $ export OPENSSL_INCLUDE_DIR=/usr/local/opt/openssl/include/
$ export OPENSSL_ROOT_DIR=/usr/local/opt/openssl/
# Protocol Buffers installation
-$ brew tap homebrew/versions
-$ brew install protobuf260
-$ brew install boost
-$ brew install log4cxx
+$ brew install protobuf boost boost-python log4cxx
+# If you are using python3, you need to install boost-python3
# Google Test installation
$ git clone https://github.com/google/googletest.git
@@ -269,6 +272,50 @@ brew install libpulsar
```
+## Windows (64-bit)
+
+### Compilation
+
+1. Clone the Pulsar repository.
+
+```shell
+
+$ git clone https://github.com/apache/pulsar
+
+```
+
+2. Install all necessary dependencies.
+
+```shell
+
+cd ${PULSAR_HOME}/pulsar-client-cpp
+vcpkg install --feature-flags=manifests --triplet x64-windows
+
+```
+
+3. Build C++ libraries.
+
+```shell
+
+cmake -B ./build -A x64 -DBUILD_PYTHON_WRAPPER=OFF -DBUILD_TESTS=OFF -DVCPKG_TRIPLET=x64-windows -DCMAKE_BUILD_TYPE=Release -S .
+cmake --build ./build --config Release
+
+```
+
+> **NOTE**
+>
+> 1. For Windows 32-bit, you need to use `-A Win32` and `-DVCPKG_TRIPLET=x86-windows`.
+> 2. For MSVC Debug mode, you need to replace `Release` with `Debug` for both `CMAKE_BUILD_TYPE` variable and `--config` option.
+
+4. Client libraries are available in the following places.
+
+```
+
+${PULSAR_HOME}/pulsar-client-cpp/build/lib/Release/pulsar.lib
+${PULSAR_HOME}/pulsar-client-cpp/build/lib/Release/pulsar.dll
+
+```
+
## Connection URLs
To connect Pulsar using client libraries, you need to specify a Pulsar protocol URL.
@@ -299,109 +346,361 @@ pulsar+ssl://pulsar.us-west.example.com:6651
## Create a consumer
-To use Pulsar as a consumer, you need to create a consumer on the C++ client. The following is an example.
+To use Pulsar as a consumer, you need to create a consumer on the C++ client. There are two main ways of using the consumer:
+- [Blocking style](#blocking-example): synchronously calling `receive(msg)`.
+- [Non-blocking](#consumer-with-a-message-listener) (event based) style: using a message listener.
+
+### Blocking example
+
+The benefit of this approach is that it is the simplest code. Simply keeps calling `receive(msg)` which blocks until a message is received.
+
+This example starts a subscription at the earliest offset and consumes 100 messages.
```c++
-Client client("pulsar://localhost:6650");
+#include <pulsar/Client.h>
+
+using namespace pulsar;
+
+int main() {
+ Client client("pulsar://localhost:6650");
+
+ Consumer consumer;
+ ConsumerConfiguration config;
+ config.setSubscriptionInitialPosition(InitialPositionEarliest);
+ Result result = client.subscribe("persistent://public/default/my-topic", "consumer-1", config, consumer);
+ if (result != ResultOk) {
+ std::cout << "Failed to subscribe: " << result << std::endl;
+ return -1;
+ }
-Consumer consumer;
-Result result = client.subscribe("my-topic", "my-subscription-name", consumer);
-if (result != ResultOk) {
- LOG_ERROR("Failed to subscribe: " << result);
- return -1;
+ Message msg;
+ int ctr = 0;
+ // consume 100 messages
+ while (ctr < 100) {
+ consumer.receive(msg);
+ std::cout << "Received: " << msg
+ << " with payload '" << msg.getDataAsString() << "'" << std::endl;
+
+ consumer.acknowledge(msg);
+ ctr++;
+ }
+
+ std::cout << "Finished consuming synchronously!" << std::endl;
+
+ client.close();
+ return 0;
}
-Message msg;
+```
+
+### Consumer with a message listener
+
+You can avoid running a loop with blocking calls with an event based style by using a message listener which is invoked for each message that is received.
+
+This example starts a subscription at the earliest offset and consumes 100 messages.
-while (true) {
- consumer.receive(msg);
- LOG_INFO("Received: " << msg
- << " with payload '" << msg.getDataAsString() << "'");
+```c++
+
+#include <pulsar/Client.h>
+#include <atomic>
+#include <thread>
+
+using namespace pulsar;
+
+std::atomic<uint32_t> messagesReceived;
+
+void handleAckComplete(Result res) {
+ std::cout << "Ack res: " << res << std::endl;
+}
- consumer.acknowledge(msg);
+void listener(Consumer consumer, const Message& msg) {
+ std::cout << "Got message " << msg << " with content '" << msg.getDataAsString() << "'" << std::endl;
+ messagesReceived++;
+ consumer.acknowledgeAsync(msg.getMessageId(), handleAckComplete);
}
-client.close();
+int main() {
+ Client client("pulsar://localhost:6650");
+
+ Consumer consumer;
+ ConsumerConfiguration config;
+ config.setMessageListener(listener);
+ config.setSubscriptionInitialPosition(InitialPositionEarliest);
+ Result result = client.subscribe("persistent://public/default/my-topic", "consumer-1", config, consumer);
+ if (result != ResultOk) {
+ std::cout << "Failed to subscribe: " << result << std::endl;
+ return -1;
+ }
+
+ // wait for 100 messages to be consumed
+ while (messagesReceived < 100) {
+ std::this_thread::sleep_for(std::chrono::milliseconds(100));
+ }
+
+ std::cout << "Finished consuming asynchronously!" << std::endl;
+
+ client.close();
+ return 0;
+}
```
## Create a producer
-To use Pulsar as a producer, you need to create a producer on the C++ client. The following is an example.
+To use Pulsar as a producer, you need to create a producer on the C++ client. There are two main ways of using a producer:
+- [Blocking style](#simple-blocking-example) : each call to `send` waits for an ack from the broker.
+- [Non-blocking asynchronous style](#non-blocking-example) : `sendAsync` is called instead of `send` and a callback is supplied for when the ack is received from the broker.
+
+### Simple blocking example
+
+This example sends 100 messages using the blocking style. While simple, it does not produce high throughput as it waits for each ack to come back before sending the next message.
```c++
-Client client("pulsar://localhost:6650");
+#include <pulsar/Client.h>
+#include <thread>
-Producer producer;
-Result result = client.createProducer("my-topic", producer);
-if (result != ResultOk) {
- LOG_ERROR("Error creating producer: " << result);
- return -1;
-}
+using namespace pulsar;
+
+int main() {
+ Client client("pulsar://localhost:6650");
+
+ Result result = client.createProducer("persistent://public/default/my-topic", producer);
+ if (result != ResultOk) {
+ std::cout << "Error creating producer: " << result << std::endl;
+ return -1;
+ }
+
+ // Send 100 messages synchronously
+ int ctr = 0;
+ while (ctr < 100) {
+ std::string content = "msg" + std::to_string(ctr);
+ Message msg = MessageBuilder().setContent(content).setProperty("x", "1").build();
+ Result result = producer.send(msg);
+ if (result != ResultOk) {
+ std::cout << "The message " << content << " could not be sent, received code: " << result << std::endl;
+ } else {
+ std::cout << "The message " << content << " sent successfully" << std::endl;
+ }
+
+ std::this_thread::sleep_for(std::chrono::milliseconds(100));
+ ctr++;
+ }
+
+ std::cout << "Finished producing synchronously!" << std::endl;
-// Publish 10 messages to the topic
-for (int i = 0; i < 10; i++){
- Message msg = MessageBuilder().setContent("my-message").build();
- Result res = producer.send(msg);
- LOG_INFO("Message sent: " << res);
+ client.close();
+ return 0;
}
-client.close();
```
-## Enable authentication in connection URLs
-If you use TLS authentication when connecting to Pulsar, you need to add `ssl` in the connection URLs, and the default port is `6651`. The following is an example.
+### Non-blocking example
-```cpp
+This example sends 100 messages using the non-blocking style calling `sendAsync` instead of `send`. This allows the producer to have multiple messages inflight at a time which increases throughput.
-ClientConfiguration config = ClientConfiguration();
-config.setUseTls(true);
-config.setTlsTrustCertsFilePath("/path/to/cacert.pem");
-config.setTlsAllowInsecureConnection(false);
-config.setAuth(pulsar::AuthTls::create(
- "/path/to/client-cert.pem", "/path/to/client-key.pem"););
+The producer configuration `blockIfQueueFull` is useful here to avoid `ResultProducerQueueIsFull` errors when the internal queue for outgoing send requests becomes full. Once the internal queue is full, `sendAsync` becomes blocking which can make your code simpler.
-Client client("pulsar+ssl://my-broker.com:6651", config);
+Without this configuration, the result code `ResultProducerQueueIsFull` is passed to the callback. You must decide how to deal with that (retry, discard etc).
+
+```c++
+
+#include <pulsar/Client.h>
+#include <thread>
+
+using namespace pulsar;
+
+std::atomic<uint32_t> acksReceived;
+
+void callback(Result code, const MessageId& msgId, std::string msgContent) {
+ // message processing logic here
+ std::cout << "Received ack for msg: " << msgContent << " with code: "
+ << code << " -- MsgID: " << msgId << std::endl;
+ acksReceived++;
+}
+
+int main() {
+ Client client("pulsar://localhost:6650");
+
+ ProducerConfiguration producerConf;
+ producerConf.setBlockIfQueueFull(true);
+ Producer producer;
+ Result result = client.createProducer("persistent://public/default/my-topic",
+ producerConf, producer);
+ if (result != ResultOk) {
+ std::cout << "Error creating producer: " << result << std::endl;
+ return -1;
+ }
+
+ // Send 100 messages asynchronously
+ int ctr = 0;
+ while (ctr < 100) {
+ std::string content = "msg" + std::to_string(ctr);
+ Message msg = MessageBuilder().setContent(content).setProperty("x", "1").build();
+ producer.sendAsync(msg, std::bind(callback,
+ std::placeholders::_1, std::placeholders::_2, content));
+
+ std::this_thread::sleep_for(std::chrono::milliseconds(100));
+ ctr++;
+ }
+
+ // wait for 100 messages to be acked
+ while (acksReceived < 100) {
+ std::this_thread::sleep_for(std::chrono::milliseconds(100));
+ }
+
+ std::cout << "Finished producing asynchronously!" << std::endl;
+
+ client.close();
+ return 0;
+}
```
-For complete examples, refer to [C++ client examples](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/examples).
+### Partitioned topics and lazy producers
-## Schema
+When scaling out a Pulsar topic, you may configure a topic to have hundreds of partitions. Likewise, you may have also scaled out your producers so there are hundreds or even thousands of producers. This can put some strain on the Pulsar brokers as when you create a producer on a partitioned topic, internally it creates one internal producer per partition which involves communications to the brokers for each one. So for a topic with 1000 partitions and 1000 producers, it ends up creating [...]
-This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started).
+You can reduce the load caused by this combination of a large number of partitions and many producers by doing the following:
+- use SinglePartition partition routing mode (this ensures that all messages are only sent to a single, randomly selected partition)
+- use non-keyed messages (when messages are keyed, routing is based on the hash of the key and so messages will end up being sent to multiple partitions)
+- use lazy producers (this ensures that an internal producer is only created on demand when a message needs to be routed to a partition)
-### Create producer with Avro schema
+With our example above, that reduces the number of internal producers spread out over the 1000 producer apps from 1,000,000 to just 1000.
-The following example shows how to create a producer with an Avro schema.
+Note that there can be extra latency for the first message sent. If you set a low send timeout, this timeout could be reached if the initial connection handshake is slow to complete.
-```cpp
+```c++
-static const std::string exampleSchema =
- "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
- "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
-Producer producer;
ProducerConfiguration producerConf;
-producerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
-client.createProducer("topic-avro", producerConf, producer);
+producerConf.setPartitionsRoutingMode(ProducerConfiguration::UseSinglePartition);
+producerConf.setLazyStartPartitionedProducers(true);
```
-### Create consumer with Avro schema
-
-The following example shows how to create a consumer with an Avro schema.
+## Enable authentication in connection URLs
+If you use TLS authentication when connecting to Pulsar, you need to add `ssl` in the connection URLs, and the default port is `6651`. The following is an example.
```cpp
-static const std::string exampleSchema =
- "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
- "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
-ConsumerConfiguration consumerConf;
-Consumer consumer;
-consumerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
-client.subscribe("topic-avro", "sub-2", consumerConf, consumer)
+ClientConfiguration config = ClientConfiguration();
+config.setUseTls(true);
+config.setTlsTrustCertsFilePath("/path/to/cacert.pem");
+config.setTlsAllowInsecureConnection(false);
+config.setAuth(pulsar::AuthTls::create(
+ "/path/to/client-cert.pem", "/path/to/client-key.pem"););
+
+Client client("pulsar+ssl://my-broker.com:6651", config);
```
+For complete examples, refer to [C++ client examples](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/examples).
+
+## Schema
+
+This section describes some examples about schema. For more information about
+schema, see [Pulsar schema](schema-get-started).
+
+### Avro schema
+
+- The following example shows how to create a producer with an Avro schema.
+
+ ```cpp
+
+ static const std::string exampleSchema =
+ "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
+ "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
+ Producer producer;
+ ProducerConfiguration producerConf;
+ producerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
+ client.createProducer("topic-avro", producerConf, producer);
+
+ ```
+
+- The following example shows how to create a consumer with an Avro schema.
+
+ ```cpp
+
+ static const std::string exampleSchema =
+ "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
+ "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
+ ConsumerConfiguration consumerConf;
+ Consumer consumer;
+ consumerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
+ client.subscribe("topic-avro", "sub-2", consumerConf, consumer)
+
+ ```
+
+### ProtobufNative schema
+
+The following example shows how to create a producer and a consumer with a ProtobufNative schema.
+
+1. Generate the `User` class using Protobuf3.
+
+ :::note
+
+ You need to use Protobuf3 or later versions.
+
+ :::
+
+
+
+ ```protobuf
+
+ syntax = "proto3";
+
+ message User {
+ string name = 1;
+ int32 age = 2;
+ }
+
+ ```
+
+
+2. Include the `ProtobufNativeSchema.h` in your source code. Ensure the Protobuf dependency has been added to your project.
+
+
+ ```c++
+
+ #include <pulsar/ProtobufNativeSchema.h>
+
+ ```
+
+
+3. Create a producer to send a `User` instance.
+
+
+ ```c++
+
+ ProducerConfiguration producerConf;
+ producerConf.setSchema(createProtobufNativeSchema(User::GetDescriptor()));
+ Producer producer;
+ client.createProducer("topic-protobuf", producerConf, producer);
+ User user;
+ user.set_name("my-name");
+ user.set_age(10);
+ std::string content;
+ user.SerializeToString(&content);
+ producer.send(MessageBuilder().setContent(content).build());
+
+ ```
+
+
+4. Create a consumer to receive a `User` instance.
+
+
+ ```c++
+
+ ConsumerConfiguration consumerConf;
+ consumerConf.setSchema(createProtobufNativeSchema(User::GetDescriptor()));
+ consumerConf.setSubscriptionInitialPosition(InitialPositionEarliest);
+ Consumer consumer;
+ client.subscribe("topic-protobuf", "my-sub", consumerConf, consumer);
+ Message msg;
+ consumer.receive(msg);
+ User user2;
+ user2.ParseFromArray(msg.getData(), msg.getLength());
+
+ ```
+
diff --git a/site2/website-next/versioned_docs/version-2.4.2/client-libraries-go.md b/site2/website-next/versioned_docs/version-2.4.2/client-libraries-go.md
index 7e797bb..22df463 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/client-libraries-go.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/client-libraries-go.md
@@ -2,7 +2,6 @@
id: client-libraries-go
title: Pulsar Go client
sidebar_label: "Go"
-original_id: client-libraries-go
---
> Tips: Currently, the CGo client will be deprecated, if you want to know more about the CGo client, please refer to [CGo client docs](client-libraries-cgo)
@@ -286,7 +285,8 @@ defer client.Close()
topicName := newTopicName()
producer, err := client.CreateProducer(pulsar.ProducerOptions{
- Topic: topicName,
+ Topic: topicName,
+ DisableBatching: true,
})
if err != nil {
log.Fatal(err)
@@ -330,6 +330,85 @@ canc()
```
+#### How to use Prometheus metrics in producer
+
+Pulsar Go client registers client metrics using Prometheus. This section demonstrates how to create a simple Pulsar producer application that exposes Prometheus metrics via HTTP.
+
+1. Write a simple producer application.
+
+```go
+
+// Create a Pulsar client
+client, err := pulsar.NewClient(pulsar.ClientOptions{
+ URL: "pulsar://localhost:6650",
+})
+if err != nil {
+ log.Fatal(err)
+}
+
+defer client.Close()
+
+// Start a separate goroutine for Prometheus metrics
+// In this case, Prometheus metrics can be accessed via http://localhost:2112/metrics
+go func() {
+ prometheusPort := 2112
+ log.Printf("Starting Prometheus metrics at http://localhost:%v/metrics\n", prometheusPort)
+ http.Handle("/metrics", promhttp.Handler())
+ err = http.ListenAndServe(":"+strconv.Itoa(prometheusPort), nil)
+ if err != nil {
+ log.Fatal(err)
+ }
+}()
+
+// Create a producer
+producer, err := client.CreateProducer(pulsar.ProducerOptions{
+ Topic: "topic-1",
+})
+if err != nil {
+ log.Fatal(err)
+}
+
+defer producer.Close()
+
+ctx := context.Background()
+
+// Write your business logic here
+// In this case, you build a simple Web server. You can produce messages by requesting http://localhost:8082/produce
+webPort := 8082
+http.HandleFunc("/produce", func(w http.ResponseWriter, r *http.Request) {
+ msgId, err := producer.Send(ctx, &pulsar.ProducerMessage{
+ Payload: []byte(fmt.Sprintf("hello world")),
+ })
+ if err != nil {
+ log.Fatal(err)
+ } else {
+ log.Printf("Published message: %v", msgId)
+ fmt.Fprintf(w, "Published message: %v", msgId)
+ }
+})
+
+err = http.ListenAndServe(":"+strconv.Itoa(webPort), nil)
+if err != nil {
+ log.Fatal(err)
+}
+
+```
+
+2. To scrape metrics from applications, configure a local running Prometheus instance using a configuration file (`prometheus.yml`).
+
+```yaml
+
+scrape_configs:
+- job_name: pulsar-client-go-metrics
+ scrape_interval: 10s
+ static_configs:
+ - targets:
+ - localhost:2112
+
+```
+
+Now you can query Pulsar client metrics on Prometheus.
+
### Producer configuration
Name | Description | Default
@@ -607,6 +686,85 @@ defer consumer.Close()
```
+#### How to use Prometheus metrics in consumer
+
+In this guide, This section demonstrates how to create a simple Pulsar consumer application that exposes Prometheus metrics via HTTP.
+1. Write a simple consumer application.
+
+```go
+
+// Create a Pulsar client
+client, err := pulsar.NewClient(pulsar.ClientOptions{
+ URL: "pulsar://localhost:6650",
+})
+if err != nil {
+ log.Fatal(err)
+}
+
+defer client.Close()
+
+// Start a separate goroutine for Prometheus metrics
+// In this case, Prometheus metrics can be accessed via http://localhost:2112/metrics
+go func() {
+ prometheusPort := 2112
+ log.Printf("Starting Prometheus metrics at http://localhost:%v/metrics\n", prometheusPort)
+ http.Handle("/metrics", promhttp.Handler())
+ err = http.ListenAndServe(":"+strconv.Itoa(prometheusPort), nil)
+ if err != nil {
+ log.Fatal(err)
+ }
+}()
+
+// Create a consumer
+consumer, err := client.Subscribe(pulsar.ConsumerOptions{
+ Topic: "topic-1",
+ SubscriptionName: "sub-1",
+ Type: pulsar.Shared,
+})
+if err != nil {
+ log.Fatal(err)
+}
+
+defer consumer.Close()
+
+ctx := context.Background()
+
+// Write your business logic here
+// In this case, you build a simple Web server. You can consume messages by requesting http://localhost:8083/consume
+webPort := 8083
+http.HandleFunc("/consume", func(w http.ResponseWriter, r *http.Request) {
+ msg, err := consumer.Receive(ctx)
+ if err != nil {
+ log.Fatal(err)
+ } else {
+ log.Printf("Received message msgId: %v -- content: '%s'\n", msg.ID(), string(msg.Payload()))
+ fmt.Fprintf(w, "Received message msgId: %v -- content: '%s'\n", msg.ID(), string(msg.Payload()))
+ consumer.Ack(msg)
+ }
+})
+
+err = http.ListenAndServe(":"+strconv.Itoa(webPort), nil)
+if err != nil {
+ log.Fatal(err)
+}
+
+```
+
+2. To scrape metrics from applications, configure a local running Prometheus instance using a configuration file (`prometheus.yml`).
+
+```yaml
+
+scrape_configs:
+- job_name: pulsar-client-go-metrics
+ scrape_interval: 10s
+ static_configs:
+ - targets:
+ - localhost:2112
+
+```
+
+Now you can query Pulsar client metrics on Prometheus.
+
### Consumer configuration
Name | Description | Default
diff --git a/site2/website-next/versioned_docs/version-2.4.2/client-libraries-websocket.md b/site2/website-next/versioned_docs/version-2.4.2/client-libraries-websocket.md
index bc13b43..c663f97 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/client-libraries-websocket.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/client-libraries-websocket.md
@@ -2,7 +2,6 @@
id: client-libraries-websocket
title: Pulsar WebSocket API
sidebar_label: "WebSocket"
-original_id: client-libraries-websocket
---
Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page.
@@ -190,7 +189,7 @@ Key | Type | Required? | Explanation
`maxRedeliverCount` | int | no | Define a [maxRedeliverCount](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html#deadLetterPolicy-org.apache.pulsar.client.api.DeadLetterPolicy-) for the consumer (default: 0). Activates [Dead Letter Topic](https://github.com/apache/pulsar/wiki/PIP-22%3A-Pulsar-Dead-Letter-Topic) feature.
`deadLetterTopic` | string | no | Define a [deadLetterTopic](http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html#deadLetterPolicy-org.apache.pulsar.client.api.DeadLetterPolicy-) for the consumer (default: {topic}-{subscription}-DLQ). Activates [Dead Letter Topic](https://github.com/apache/pulsar/wiki/PIP-22%3A-Pulsar-Dead-Letter-Topic) feature.
`pullMode` | boolean | no | Enable pull mode (default: false). See "Flow Control" below.
-`negativeAckRedeliveryDelay` | int | no | When a message is negatively acknowledged, it will be redelivered to the DLQ.
+`negativeAckRedeliveryDelay` | int | no | When a message is negatively acknowledged, the delay time before the message is redelivered (in milliseconds). The default value is 60000.
`token` | string | no | Authentication token, this is used for the browser javascript client
NB: these parameter (except `pullMode`) apply to the internal consumer of the WebSocket service.
@@ -204,23 +203,60 @@ Server will push messages on the WebSocket session:
```json
{
- "messageId": "CAAQAw==",
- "payload": "SGVsbG8gV29ybGQ=",
- "properties": {"key1": "value1", "key2": "value2"},
- "publishTime": "2016-08-30 16:45:57.785",
- "redeliveryCount": 4
+ "messageId": "CAMQADAA",
+ "payload": "hvXcJvHW7kOSrUn17P2q71RA5SdiXwZBqw==",
+ "properties": {},
+ "publishTime": "2021-10-29T16:01:38.967-07:00",
+ "redeliveryCount": 0,
+ "encryptionContext": {
+ "keys": {
+ "client-rsa.pem": {
+ "keyValue": "jEuwS+PeUzmCo7IfLNxqoj4h7txbLjCQjkwpaw5AWJfZ2xoIdMkOuWDkOsqgFmWwxiecakS6GOZHs94x3sxzKHQx9Oe1jpwBg2e7L4fd26pp+WmAiLm/ArZJo6JotTeFSvKO3u/yQtGTZojDDQxiqFOQ1ZbMdtMZA8DpSMuq+Zx7PqLo43UdW1+krjQfE5WD+y+qE3LJQfwyVDnXxoRtqWLpVsAROlN2LxaMbaftv5HckoejJoB4xpf/dPOUqhnRstwQHf6klKT5iNhjsY4usACt78uILT0pEPd14h8wEBidBz/vAlC/zVMEqiDVzgNS7dqEYS4iHbf7cnWVCn3Hxw==",
+ "metadata": {}
+ }
+ },
+ "param": "Tfu1PxVm6S9D3+Hk",
+ "compressionType": "NONE",
+ "uncompressedMessageSize": 0,
+ "batchSize": {
+ "empty": false,
+ "present": true
+ }
+ }
}
```
-Key | Type | Required? | Explanation
-:---|:-----|:----------|:-----------
-`messageId` | string | yes | Message ID
-`payload` | string | yes | Base-64 encoded payload
-`publishTime` | string | yes | Publish timestamp
-`redeliveryCount` | number | yes | Number of times this message was already delivered
-`properties` | key-value pairs | no | Application-defined properties
-`key` | string | no | Original routing key set by producer
+Below are the parameters in the WebSocket consumer response.
+
+- General parameters
+
+ Key | Type | Required? | Explanation
+ :---|:-----|:----------|:-----------
+ `messageId` | string | yes | Message ID
+ `payload` | string | yes | Base-64 encoded payload
+ `publishTime` | string | yes | Publish timestamp
+ `redeliveryCount` | number | yes | Number of times this message was already delivered
+ `properties` | key-value pairs | no | Application-defined properties
+ `key` | string | no | Original routing key set by producer
+ `encryptionContext` | EncryptionContext | no | Encryption context that consumers can use to decrypt received messages
+ `param` | string | no | Initialization vector for cipher (Base64 encoding)
+ `batchSize` | string | no | Number of entries in a message (if it is a batch message)
+ `uncompressedMessageSize` | string | no | Message size before compression
+ `compressionType` | string | no | Algorithm used to compress the message payload
+
+- `encryptionContext` related parameter
+
+ Key | Type | Required? | Explanation
+ :---|:-----|:----------|:-----------
+ `keys` |key-EncryptionKey pairs | yes | Key in `key-EncryptionKey` pairs is an encryption key name. Value in `key-EncryptionKey` pairs is an encryption key object.
+
+- `encryptionKey` related parameters
+
+ Key | Type | Required? | Explanation
+ :---|:-----|:----------|:-----------
+ `keyValue` | string | yes | Encryption key (Base64 encoding)
+ `metadata` | key-value pairs | no | Application-defined metadata
#### Acknowledging the message
@@ -454,9 +490,15 @@ TOPIC = scheme + '://localhost:8080/ws/v2/producer/persistent/public/default/my-
ws = websocket.create_connection(TOPIC)
+# encode message
+s = "Hello World"
+firstEncoded = s.encode("UTF-8")
+binaryEncoded = base64.b64encode(firstEncoded)
+payloadString = binaryEncoded.decode('UTF-8')
+
# Send one message as JSON
ws.send(json.dumps({
- 'payload' : base64.b64encode('Hello World'),
+ 'payload' : payloadString,
'properties': {
'key1' : 'value1',
'key2' : 'value2'
@@ -466,9 +508,9 @@ ws.send(json.dumps({
response = json.loads(ws.recv())
if response['result'] == 'ok':
- print 'Message published successfully'
+ print( 'Message published successfully')
else:
- print 'Failed to publish message:', response
+ print('Failed to publish message:', response)
ws.close()
```
@@ -495,7 +537,7 @@ while True:
msg = json.loads(ws.recv())
if not msg: break
- print "Received: {} - payload: {}".format(msg, base64.b64decode(msg['payload']))
+ print( "Received: {} - payload: {}".format(msg, base64.b64decode(msg['payload'])))
# Acknowledge successful processing
ws.send(json.dumps({'messageId' : msg['messageId']}))
@@ -525,7 +567,7 @@ while True:
msg = json.loads(ws.recv())
if not msg: break
- print "Received: {} - payload: {}".format(msg, base64.b64decode(msg['payload']))
+ print ( "Received: {} - payload: {}".format(msg, base64.b64decode(msg['payload'])))
# Acknowledge successful processing
ws.send(json.dumps({'messageId' : msg['messageId']}))
diff --git a/site2/website-next/versioned_docs/version-2.4.2/client-libraries.md b/site2/website-next/versioned_docs/version-2.4.2/client-libraries.md
index 23e5a06..ab5b7c4 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/client-libraries.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/client-libraries.md
@@ -2,7 +2,6 @@
id: client-libraries
title: Pulsar client libraries
sidebar_label: "Overview"
-original_id: client-libraries
---
Pulsar supports the following client libraries:
@@ -16,7 +15,7 @@ Pulsar supports the following client libraries:
- [C# client](client-libraries-dotnet)
## Feature matrix
-Pulsar client feature matrix for different languages is listed on [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page.
+Pulsar client feature matrix for different languages is listed on [Pulsar Feature Matrix (Client and Function)](https://github.com/apache/pulsar/wiki/PIP-108%3A-Pulsar-Feature-Matrix-%28Client-and-Function%29) page.
## Third-party clients
@@ -33,3 +32,4 @@ Besides the official released clients, multiple projects on developing Pulsar cl
| Scala | [pulsar4s](https://github.com/sksamuel/pulsar4s) | [sksamuel](https://github.com/sksamuel) | [](https://opensource.org/licenses/Apache-2.0) | Idomatic, typesafe, and reactive Scala client for Apache Pulsar |
| Rust | [pulsar-rs](https://github.com/wyyerd/pulsar-rs) | [Wyyerd Group](https://github.com/wyyerd) | [](https://opensource.org/licenses/Apache-2.0) | Future-based Rust bindings for Apache Pulsar |
| .NET | [pulsar-client-dotnet](https://github.com/fsharplang-ru/pulsar-client-dotnet) | [Lanayx](https://github.com/Lanayx) | [](https://opensource.org/licenses/MIT) | Native .NET client for C#/F#/VB |
+| Node.js | [pulsar-flex](https://github.com/ayeo-flex-org/pulsar-flex) | [Daniel Sinai](https://github.com/danielsinai), [Ron Farkash](https://github.com/ronfarkash), [Gal Rosenberg](https://github.com/galrose)| [](https://opensource.org/licenses/MIT) | Native Nodejs client |
diff --git a/site2/website-next/versioned_docs/version-2.4.2/concepts-architecture-overview.md b/site2/website-next/versioned_docs/version-2.4.2/concepts-architecture-overview.md
index 6a501d2..8fe0717 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/concepts-architecture-overview.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/concepts-architecture-overview.md
@@ -2,7 +2,6 @@
id: concepts-architecture-overview
title: Architecture Overview
sidebar_label: "Architecture"
-original_id: concepts-architecture-overview
---
At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves.
@@ -146,7 +145,7 @@ Some important things to know about the Pulsar proxy:
## Service discovery
-[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide.
+[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL.
You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means.
diff --git a/site2/website-next/versioned_docs/version-2.4.2/concepts-authentication.md b/site2/website-next/versioned_docs/version-2.4.2/concepts-authentication.md
index b375ecb..335da8d 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/concepts-authentication.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/concepts-authentication.md
@@ -2,7 +2,6 @@
id: concepts-authentication
title: Authentication and Authorization
sidebar_label: "Authentication and Authorization"
-original_id: concepts-authentication
---
Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants.
diff --git a/site2/website-next/versioned_docs/version-2.4.2/concepts-clients.md b/site2/website-next/versioned_docs/version-2.4.2/concepts-clients.md
index b68f76a..65201b5 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/concepts-clients.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/concepts-clients.md
@@ -2,7 +2,6 @@
id: concepts-clients
title: Pulsar Clients
sidebar_label: "Clients"
-original_id: concepts-clients
---
Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications.
diff --git a/site2/website-next/versioned_docs/version-2.4.2/concepts-multi-tenancy.md b/site2/website-next/versioned_docs/version-2.4.2/concepts-multi-tenancy.md
index be752cc..8a17e72 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/concepts-multi-tenancy.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/concepts-multi-tenancy.md
@@ -2,7 +2,6 @@
id: concepts-multi-tenancy
title: Multi Tenancy
sidebar_label: "Multi Tenancy"
-original_id: concepts-multi-tenancy
---
Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed.
diff --git a/site2/website-next/versioned_docs/version-2.4.2/concepts-overview.md b/site2/website-next/versioned_docs/version-2.4.2/concepts-overview.md
index b903fa4..c76032c 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/concepts-overview.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/concepts-overview.md
@@ -2,7 +2,6 @@
id: concepts-overview
title: Pulsar Overview
sidebar_label: "Overview"
-original_id: concepts-overview
---
Pulsar is a multi-tenant, high-performance solution for server-to-server messaging. Pulsar was originally developed by Yahoo, it is under the stewardship of the [Apache Software Foundation](https://www.apache.org/).
diff --git a/site2/website-next/versioned_docs/version-2.4.2/concepts-replication.md b/site2/website-next/versioned_docs/version-2.4.2/concepts-replication.md
index 6e23962..11677cc 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/concepts-replication.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/concepts-replication.md
@@ -2,7 +2,6 @@
id: concepts-replication
title: Geo Replication
sidebar_label: "Geo Replication"
-original_id: concepts-replication
---
Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo) in Pulsar enables you to do that.
diff --git a/site2/website-next/versioned_docs/version-2.4.2/concepts-topic-compaction.md b/site2/website-next/versioned_docs/version-2.4.2/concepts-topic-compaction.md
index c85e703..3356298 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/concepts-topic-compaction.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/concepts-topic-compaction.md
@@ -2,7 +2,6 @@
id: concepts-topic-compaction
title: Topic Compaction
sidebar_label: "Topic Compaction"
-original_id: concepts-topic-compaction
---
Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time int [...]
diff --git a/site2/website-next/versioned_docs/version-2.4.2/cookbooks-bookkeepermetadata.md b/site2/website-next/versioned_docs/version-2.4.2/cookbooks-bookkeepermetadata.md
index b0fa98d..187cb65 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/cookbooks-bookkeepermetadata.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/cookbooks-bookkeepermetadata.md
@@ -1,7 +1,6 @@
---
id: cookbooks-bookkeepermetadata
title: BookKeeper Ledger Metadata
-original_id: cookbooks-bookkeepermetadata
---
Pulsar stores data on BookKeeper ledgers, you can understand the contents of a ledger by inspecting the metadata attached to the ledger.
diff --git a/site2/website-next/versioned_docs/version-2.4.2/cookbooks-deduplication.md b/site2/website-next/versioned_docs/version-2.4.2/cookbooks-deduplication.md
index 1669afa..307fe03 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/cookbooks-deduplication.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/cookbooks-deduplication.md
@@ -1,8 +1,7 @@
---
id: cookbooks-deduplication
title: Message deduplication
-sidebar_label: "Message deduplication"
-original_id: cookbooks-deduplication
+sidebar_label: "Message deduplication "
---
import Tabs from '@theme/Tabs';
diff --git a/site2/website-next/versioned_docs/version-2.4.2/cookbooks-encryption.md b/site2/website-next/versioned_docs/version-2.4.2/cookbooks-encryption.md
index f0d8fb8..fbd1c97 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/cookbooks-encryption.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/cookbooks-encryption.md
@@ -1,8 +1,7 @@
---
id: cookbooks-encryption
title: Pulsar Encryption
-sidebar_label: "Encryption"
-original_id: cookbooks-encryption
+sidebar_label: "Encryption "
---
Pulsar encryption allows applications to encrypt messages at the producer and decrypt at the consumer. Encryption is performed using the public/private key pair configured by the application. Encrypted messages can only be decrypted by consumers with a valid key.
diff --git a/site2/website-next/versioned_docs/version-2.4.2/cookbooks-message-queue.md b/site2/website-next/versioned_docs/version-2.4.2/cookbooks-message-queue.md
index eb43cbd..9b93a94 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/cookbooks-message-queue.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/cookbooks-message-queue.md
@@ -2,7 +2,6 @@
id: cookbooks-message-queue
title: Using Pulsar as a message queue
sidebar_label: "Message queue"
-original_id: cookbooks-message-queue
---
Message queues are essential components of many large-scale data architectures. If every single work object that passes through your system absolutely *must* be processed in spite of the slowness or downright failure of this or that system component, there's a good chance that you'll need a message queue to step in and ensure that unprocessed data is retained---with correct ordering---until the required actions are taken.
diff --git a/site2/website-next/versioned_docs/version-2.4.2/cookbooks-non-persistent.md b/site2/website-next/versioned_docs/version-2.4.2/cookbooks-non-persistent.md
index 391569a..d40c4fb 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/cookbooks-non-persistent.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/cookbooks-non-persistent.md
@@ -2,7 +2,6 @@
id: cookbooks-non-persistent
title: Non-persistent messaging
sidebar_label: "Non-persistent messaging"
-original_id: cookbooks-non-persistent
---
**Non-persistent topics** are Pulsar topics in which message data is *never* [persistently stored](concepts-architecture-overview.md#persistent-storage) and kept only in memory. This cookbook provides:
diff --git a/site2/website-next/versioned_docs/version-2.4.2/cookbooks-partitioned.md b/site2/website-next/versioned_docs/version-2.4.2/cookbooks-partitioned.md
index 7882fb9..2589693 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/cookbooks-partitioned.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/cookbooks-partitioned.md
@@ -2,6 +2,5 @@
id: cookbooks-partitioned
title: Partitioned topics
sidebar_label: "Partitioned Topics"
-original_id: cookbooks-partitioned
---
For details of the content, refer to [manage topics](admin-api-topics).
\ No newline at end of file
diff --git a/site2/website-next/versioned_docs/version-2.4.2/cookbooks-retention-expiry.md b/site2/website-next/versioned_docs/version-2.4.2/cookbooks-retention-expiry.md
index b9353b5..738cf42 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/cookbooks-retention-expiry.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/cookbooks-retention-expiry.md
@@ -2,7 +2,6 @@
id: cookbooks-retention-expiry
title: Message retention and expiry
sidebar_label: "Message retention and expiry"
-original_id: cookbooks-retention-expiry
---
import Tabs from '@theme/Tabs';
@@ -36,7 +35,7 @@ By default, when a Pulsar message arrives at a broker, the message is stored unt
Retention policies are useful when you use the Reader interface. The Reader interface does not use acknowledgements, and messages do not exist within backlogs. It is required to configure retention for Reader-only use cases.
-When you set a retention policy on topics in a namespace, you must set **both** a *size limit* and a *time limit*. You can refer to the following table to set retention policies in `pulsar-admin` and Java.
+When you set a retention policy on topics in a namespace, you must set **both** a *size limit* (via `defaultRetentionSizeInMB`) and a *time limit* (via `defaultRetentionTimeInMinutes`) . You can refer to the following table to set retention policies in `pulsar-admin` and Java.
|Time limit|Size limit| Message retention |
|----------|----------|------------------------|
@@ -152,7 +151,10 @@ admin.namespaces().setRetention(namespace, policies);
You can fetch the retention policy for a namespace by specifying the namespace. The output will be a JSON object with two keys: `retentionTimeInMinutes` and `retentionSizeInMB`.
-#### pulsar-admin
+<Tabs
+ defaultValue="pulsar-admin"
+ values={[{"label":"pulsar-admin","value":"pulsar-admin"},{"label":"REST API","value":"REST API"},{"label":"Java","value":"Java"}]}>
+<TabItem value="pulsar-admin">
Use the [`get-retention`](reference-pulsar-admin.md#namespaces) subcommand and specify the namespace.
@@ -168,11 +170,13 @@ $ pulsar-admin namespaces get-retention my-tenant/my-ns
```
-#### REST API
+</TabItem>
+<TabItem value="REST API">
{@inject: endpoint|GET|/admin/v2/namespaces/:tenant/:namespace/retention|operation/getRetention?version=@pulsar:version_number@}
-#### Java
+</TabItem>
+<TabItem value="Java">
```java
@@ -180,15 +184,17 @@ admin.namespaces().getRetention(namespace);
```
+</TabItem>
+
+</Tabs>
+
## Backlog quotas
*Backlogs* are sets of unacknowledged messages for a topic that have been stored by bookies. Pulsar stores all unacknowledged messages in backlogs until they are processed and acknowledged.
-You can control the allowable size of backlogs, at the namespace level, using *backlog quotas*. Setting a backlog quota involves setting:
+You can control the allowable size and/or time of backlogs, at the namespace level, using *backlog quotas*. Setting a backlog quota involves setting:
-TODO: Expand on is this per backlog or per topic?
-
-* an allowable *size threshold* for each topic in the namespace
+* an allowable *size and/or time threshold* for each topic in the namespace
* a *retention policy* that determines which action the [broker](reference-terminology.md#broker) takes if the threshold is exceeded.
The following retention policies are available:
@@ -210,9 +216,12 @@ Backlog quotas are handled at the namespace level. They can be managed via:
You can set a size and/or time threshold and backlog retention policy for all of the topics in a [namespace](reference-terminology.md#namespace) by specifying the namespace, a size limit and/or a time limit in second, and a policy by name.
-#### pulsar-admin
+<Tabs
+ defaultValue="pulsar-admin"
+ values={[{"label":"pulsar-admin","value":"pulsar-admin"},{"label":"REST API","value":"REST API"},{"label":"Java","value":"Java"}]}>
+<TabItem value="pulsar-admin">
-Use the [`set-backlog-quota`](reference-pulsar-admin.md#namespaces) subcommand and specify a namespace, a size limit using the `-l`/`--limit` flag, and a retention policy using the `-p`/`--policy` flag.
+Use the [`set-backlog-quota`](reference-pulsar-admin.md#namespaces) subcommand and specify a namespace, a size limit using the `-l`/`--limit` , `-lt`/`--limitTime` flag to limit backlog, a retention policy using the `-p`/`--policy` flag and a policy type using `-t`/`--type` (default is destination_storage).
##### Example
@@ -220,16 +229,26 @@ Use the [`set-backlog-quota`](reference-pulsar-admin.md#namespaces) subcommand a
$ pulsar-admin namespaces set-backlog-quota my-tenant/my-ns \
--limit 2G \
- --limitTime 36000 \
--policy producer_request_hold
```
-#### REST API
+```shell
+
+$ pulsar-admin namespaces set-backlog-quota my-tenant/my-ns/my-topic \
+--limitTime 3600 \
+--policy producer_request_hold \
+--type message_age
+
+```
+
+</TabItem>
+<TabItem value="REST API">
{@inject: endpoint|POST|/admin/v2/namespaces/:tenant/:namespace/backlogQuota|operation/getBacklogQuotaMap?version=@pulsar:version_number@}
-#### Java
+</TabItem>
+<TabItem value="Java">
```java
@@ -240,11 +259,18 @@ admin.namespaces().setBacklogQuota(namespace, quota);
```
+</TabItem>
+
+</Tabs>
+
### Get backlog threshold and backlog retention policy
You can see which size threshold and backlog retention policy has been applied to a namespace.
-#### pulsar-admin
+<Tabs
+ defaultValue="pulsar-admin"
+ values={[{"label":"pulsar-admin","value":"pulsar-admin"},{"label":"REST API","value":"REST API"},{"label":"Java","value":"Java"}]}>
+<TabItem value="pulsar-admin">
Use the [`get-backlog-quotas`](reference-pulsar-admin.md#pulsar-admin-namespaces-get-backlog-quotas) subcommand and specify a namespace. Here's an example:
@@ -260,11 +286,13 @@ $ pulsar-admin namespaces get-backlog-quotas my-tenant/my-ns
```
-#### REST API
+</TabItem>
+<TabItem value="REST API">
{@inject: endpoint|GET|/admin/v2/namespaces/:tenant/:namespace/backlogQuotaMap|operation/getBacklogQuotaMap?version=@pulsar:version_number@}
-#### Java
+</TabItem>
+<TabItem value="Java">
```java
@@ -273,11 +301,18 @@ Map<BacklogQuota.BacklogQuotaType,BacklogQuota> quotas =
```
+</TabItem>
+
+</Tabs>
+
### Remove backlog quotas
-#### pulsar-admin
+<Tabs
+ defaultValue="pulsar-admin"
+ values={[{"label":"pulsar-admin","value":"pulsar-admin"},{"label":"REST API","value":"REST API"},{"label":"Java","value":"Java"}]}>
+<TabItem value="pulsar-admin">
-Use the [`remove-backlog-quota`](reference-pulsar-admin.md#pulsar-admin-namespaces-remove-backlog-quota) subcommand and specify a namespace. Here's an example:
+Use the [`remove-backlog-quota`](reference-pulsar-admin.md#pulsar-admin-namespaces-remove-backlog-quota) subcommand and specify a namespace, use `t`/`--type` to specify backlog type to remove(default is destination_storage). Here's an example:
```shell
@@ -285,11 +320,13 @@ $ pulsar-admin namespaces remove-backlog-quota my-tenant/my-ns
```
-#### REST API
+</TabItem>
+<TabItem value="REST API">
{@inject: endpoint|DELETE|/admin/v2/namespaces/:tenant/:namespace/backlogQuota|operation/removeBacklogQuota?version=@pulsar:version_number@}
-#### Java
+</TabItem>
+<TabItem value="Java">
```java
@@ -297,6 +334,10 @@ admin.namespaces().removeBacklogQuota(namespace);
```
+</TabItem>
+
+</Tabs>
+
### Clear backlog
#### pulsar-admin
@@ -319,7 +360,10 @@ By default, Pulsar stores all unacknowledged messages forever. This can lead to
### Set the TTL for a namespace
-#### pulsar-admin
+<Tabs
+ defaultValue="pulsar-admin"
+ values={[{"label":"pulsar-admin","value":"pulsar-admin"},{"label":"REST API","value":"REST API"},{"label":"Java","value":"Java"}]}>
+<TabItem value="pulsar-admin">
Use the [`set-message-ttl`](reference-pulsar-admin.md#pulsar-admin-namespaces-set-message-ttl) subcommand and specify a namespace and a TTL (in seconds) using the `-ttl`/`--messageTTL` flag.
@@ -332,11 +376,13 @@ $ pulsar-admin namespaces set-message-ttl my-tenant/my-ns \
```
-#### REST API
+</TabItem>
+<TabItem value="REST API">
{@inject: endpoint|POST|/admin/v2/namespaces/:tenant/:namespace/messageTTL|operation/setNamespaceMessageTTL?version=@pulsar:version_number@}
-#### Java
+</TabItem>
+<TabItem value="Java">
```java
@@ -344,9 +390,16 @@ admin.namespaces().setNamespaceMessageTTL(namespace, ttlInSeconds);
```
+</TabItem>
+
+</Tabs>
+
### Get the TTL configuration for a namespace
-#### pulsar-admin
+<Tabs
+ defaultValue="pulsar-admin"
+ values={[{"label":"pulsar-admin","value":"pulsar-admin"},{"label":"REST API","value":"REST API"},{"label":"Java","value":"Java"}]}>
+<TabItem value="pulsar-admin">
Use the [`get-message-ttl`](reference-pulsar-admin.md#pulsar-admin-namespaces-get-message-ttl) subcommand and specify a namespace.
@@ -359,11 +412,13 @@ $ pulsar-admin namespaces get-message-ttl my-tenant/my-ns
```
-#### REST API
+</TabItem>
+<TabItem value="REST API">
{@inject: endpoint|GET|/admin/v2/namespaces/:tenant/:namespace/messageTTL|operation/getNamespaceMessageTTL?version=@pulsar:version_number@}
-#### Java
+</TabItem>
+<TabItem value="Java">
```java
@@ -371,9 +426,16 @@ admin.namespaces().getNamespaceMessageTTL(namespace)
```
+</TabItem>
+
+</Tabs>
+
### Remove the TTL configuration for a namespace
-#### pulsar-admin
+<Tabs
+ defaultValue="pulsar-admin"
+ values={[{"label":"pulsar-admin","value":"pulsar-admin"},{"label":"REST API","value":"REST API"},{"label":"Java","value":"Java"}]}>
+<TabItem value="pulsar-admin">
Use the [`remove-message-ttl`](reference-pulsar-admin.md#pulsar-admin-namespaces-remove-message-ttl) subcommand and specify a namespace.
@@ -385,11 +447,13 @@ $ pulsar-admin namespaces remove-message-ttl my-tenant/my-ns
```
-#### REST API
+</TabItem>
+<TabItem value="REST API">
{@inject: endpoint|DELETE|/admin/v2/namespaces/:tenant/:namespace/messageTTL|operation/removeNamespaceMessageTTL?version=@pulsar:version_number@}
-#### Java
+</TabItem>
+<TabItem value="Java">
```java
@@ -397,6 +461,10 @@ admin.namespaces().removeNamespaceMessageTTL(namespace)
```
+</TabItem>
+
+</Tabs>
+
## Delete messages from namespaces
If you do not have any retention period and that you never have much of a backlog, the upper limit for retaining messages, which are acknowledged, equals to the Pulsar segment rollover period + entry log rollover period + (garbage collection interval * garbage collection ratios).
diff --git a/site2/website-next/versioned_docs/version-2.4.2/cookbooks-tiered-storage.md b/site2/website-next/versioned_docs/version-2.4.2/cookbooks-tiered-storage.md
index 8f6a7fb..f2ea50d 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/cookbooks-tiered-storage.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/cookbooks-tiered-storage.md
@@ -2,14 +2,15 @@
id: cookbooks-tiered-storage
title: Tiered Storage
sidebar_label: "Tiered Storage"
-original_id: cookbooks-tiered-storage
---
Pulsar's **Tiered Storage** feature allows older backlog data to be offloaded to long term storage, thereby freeing up space in BookKeeper and reducing storage costs. This cookbook walks you through using tiered storage in your Pulsar cluster.
-* Tiered storage uses [Apache jclouds](https://jclouds.apache.org) to support [Amazon S3](https://aws.amazon.com/s3/) and [Google Cloud Storage](https://cloud.google.com/storage/)(GCS for short) for long term storage. With Jclouds, it is easy to add support for more [cloud storage providers](https://jclouds.apache.org/reference/providers/#blobstore-providers) in the future.
+* Tiered storage uses [Apache jclouds](https://jclouds.apache.org) to support [Amazon S3](https://aws.amazon.com/s3/) and [Google Cloud Storage](https://cloud.google.com/storage/)(GCS for short)
+for long term storage. With Jclouds, it is easy to add support for more [cloud storage providers](https://jclouds.apache.org/reference/providers/#blobstore-providers) in the future.
-* Tiered storage uses [Apache Hadoop](http://hadoop.apache.org/) to support filesystem for long term storage. With Hadoop, it is easy to add support for more filesystem in the future.
+* Tiered storage uses [Apache Hadoop](http://hadoop.apache.org/) to support filesystem for long term storage.
+With Hadoop, it is easy to add support for more filesystem in the future.
## When should I use Tiered Storage?
diff --git a/site2/website-next/versioned_docs/version-2.4.2/deploy-aws.md b/site2/website-next/versioned_docs/version-2.4.2/deploy-aws.md
index 6323051..2034749 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/deploy-aws.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/deploy-aws.md
@@ -2,7 +2,6 @@
id: deploy-aws
title: Deploying a Pulsar cluster on AWS using Terraform and Ansible
sidebar_label: "Amazon Web Services"
-original_id: deploy-aws
---
> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster).
@@ -148,7 +147,7 @@ Variable name | Description | Default
When you run the Ansible playbook, the following AWS resources are used:
* 9 total [Elastic Compute Cloud](https://aws.amazon.com/ec2) (EC2) instances running the [ami-9fa343e7](https://access.redhat.com/articles/3135091) Amazon Machine Image (AMI), which runs [Red Hat Enterprise Linux (RHEL) 7.4](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/7.4_release_notes/index). By default, that includes:
- * 3 small VMs for ZooKeeper ([t2.small](https://www.ec2instances.info/?selected=t2.small) instances)
+ * 3 small VMs for ZooKeeper ([t3.small](https://www.ec2instances.info/?selected=t3.small) instances)
* 3 larger VMs for BookKeeper [bookies](reference-terminology.md#bookie) ([i3.xlarge](https://www.ec2instances.info/?selected=i3.xlarge) instances)
* 2 larger VMs for Pulsar [brokers](reference-terminology.md#broker) ([c5.2xlarge](https://www.ec2instances.info/?selected=c5.2xlarge) instances)
* 1 larger VMs for Pulsar [proxy](reference-terminology.md#proxy) ([c5.2xlarge](https://www.ec2instances.info/?selected=c5.2xlarge) instances)
diff --git a/site2/website-next/versioned_docs/version-2.4.2/deploy-bare-metal-multi-cluster.md b/site2/website-next/versioned_docs/version-2.4.2/deploy-bare-metal-multi-cluster.md
index 643c122..9dd2526 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/deploy-bare-metal-multi-cluster.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/deploy-bare-metal-multi-cluster.md
@@ -2,38 +2,30 @@
id: deploy-bare-metal-multi-cluster
title: Deploying a multi-cluster on bare metal
sidebar_label: "Bare metal multi-cluster"
-original_id: deploy-bare-metal-multi-cluster
---
:::tip
-1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with
-Pulsar or using it in a startup or on a single team, you had better opt for a single cluster. For instructions on deploying a single cluster,
-see the guide [here](deploy-bare-metal).
-2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors`
-package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you
-run a separate cluster of function workers for [Pulsar Functions](functions-overview).
-3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`
-package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure
-this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage).
+1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster).
+2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview).
+3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage).
:::
-A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo). Deploying a multi-cluster Pulsar instance involves the following basic steps:
+A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo).Deploying a multi-cluster Pulsar instance consists of the following steps:
-* Deploying two separate [ZooKeeper](#deploy-zookeeper) quorums: a [local](#deploy-local-zookeeper) quorum for each cluster in the instance and a [configuration store](#configuration-store) quorum for instance-wide tasks
-* Initializing [cluster metadata](#cluster-metadata-initialization) for each cluster
-* Deploying a [BookKeeper cluster](#deploy-bookkeeper) of bookies in each Pulsar cluster
-* Deploying [brokers](#deploy-brokers) in each Pulsar cluster
+1. Deploying two separate ZooKeeper quorums: a local quorum for each cluster in the instance and a configuration store quorum for instance-wide tasks
+2. Initializing cluster metadata for each cluster
+3. Deploying a BookKeeper cluster of bookies in each Pulsar cluster
+4. Deploying brokers in each Pulsar cluster
-If you want to deploy a single Pulsar cluster, see [Clusters and Brokers](getting-started-standalone.md#start-the-cluster).
> #### Run Pulsar locally or on Kubernetes?
-> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar [...]
+> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services.
## System requirement
-Currently, Pulsar is available for 64-bit **macOS**, **Linux**, and **Windows**. To use Pulsar, you need to install 64-bit JRE/JDK 8 or later versions.
+Currently, Pulsar is available for 64-bit **macOS**, **Linux**, and **Windows**. You need to install 64-bit JRE/JDK 8 or later versions.
:::note
@@ -68,8 +60,6 @@ $ cd apache-pulsar-@pulsar:version@
```
-## What your package contains
-
The Pulsar binary package initially contains the following directories:
Directory | Contains
@@ -93,17 +83,17 @@ Directory | Contains
Each Pulsar instance relies on two separate ZooKeeper quorums.
-* [Local ZooKeeper](#deploy-local-zookeeper) operates at the cluster level and provides cluster-specific configuration management and coordination. Each Pulsar cluster needs to have a dedicated ZooKeeper cluster.
-* [Configuration Store](#deploy-the-configuration-store) operates at the instance level and provides configuration management for the entire system (and thus across clusters). An independent cluster of machines or the same machines that local ZooKeeper uses can provide the configuration store quorum.
+* Local ZooKeeper operates at the cluster level and provides cluster-specific configuration management and coordination. Each Pulsar cluster needs a dedicated ZooKeeper cluster.
+* Configuration Store operates at the instance level and provides configuration management for the entire system (and thus across clusters). An independent cluster of machines or the same machines that local ZooKeeper uses can provide the configuration store quorum.
-The configuration store quorum can be provided by an independent cluster of machines or by the same machines used by local ZooKeeper.
+You can use an independent cluster of machines or the same machines used by local ZooKeeper to provide the configuration store quorum.
### Deploy local ZooKeeper
ZooKeeper manages a variety of essential coordination-related and configuration-related tasks for Pulsar.
-You need to stand up one local ZooKeeper cluster *per Pulsar cluster* for deploying a Pulsar instance.
+You need to stand up one local ZooKeeper cluster per Pulsar cluster for deploying a Pulsar instance.
To begin, add all ZooKeeper servers to the quorum configuration specified in the [`conf/zookeeper.conf`](reference-configuration.md#zookeeper) file. Add a `server.N` line for each node in the cluster to the configuration, where `N` is the number of the ZooKeeper node. The following is an example for a three-node cluster:
@@ -117,7 +107,11 @@ server.3=zk3.us-west.example.com:2888:3888
On each host, you need to specify the ID of the node in the `myid` file of each node, which is in `data/zookeeper` folder of each server by default (you can change the file location via the [`dataDir`](reference-configuration.md#zookeeper-dataDir) parameter).
-> See the [Multi-server setup guide](https://zookeeper.apache.org/doc/r3.4.10/zookeeperAdmin.html#sc_zkMulitServerSetup) in the ZooKeeper documentation for detailed information on `myid` and more.
+:::tip
+
+See the [Multi-server setup guide](https://zookeeper.apache.org/doc/r3.4.10/zookeeperAdmin.html#sc_zkMulitServerSetup) in the ZooKeeper documentation for detailed information on `myid` and more.
+
+:::
On a ZooKeeper server at `zk1.us-west.example.com`, for example, you could set the `myid` value like this:
@@ -140,15 +134,15 @@ $ bin/pulsar-daemon start zookeeper
### Deploy the configuration store
-The ZooKeeper cluster that is configured and started up in the section above is a *local* ZooKeeper cluster that you can use to manage a single Pulsar cluster. In addition to a local cluster, however, a full Pulsar instance also requires a configuration store for handling some instance-level configuration and coordination tasks.
+The ZooKeeper cluster configured and started up in the section above is a local ZooKeeper cluster that you can use to manage a single Pulsar cluster. In addition to a local cluster, however, a full Pulsar instance also requires a configuration store for handling some instance-level configuration and coordination tasks.
-If you deploy a [single-cluster](#single-cluster-pulsar-instance) instance, you do not need a separate cluster for the configuration store. If, however, you deploy a [multi-cluster](#multi-cluster-pulsar-instance) instance, you should stand up a separate ZooKeeper cluster for configuration tasks.
+If you deploy a single-cluster instance, you do not need a separate cluster for the configuration store. If, however, you deploy a multi-cluster instance, you should stand up a separate ZooKeeper cluster for configuration tasks.
#### Single-cluster Pulsar instance
If your Pulsar instance consists of just one cluster, then you can deploy a configuration store on the same machines as the local ZooKeeper quorum but run on different TCP ports.
-To deploy a ZooKeeper configuration store in a single-cluster instance, add the same ZooKeeper servers that the local quorum uses to the configuration file in [`conf/global_zookeeper.conf`](reference-configuration.md#configuration-store) using the same method for [local ZooKeeper](#local-zookeeper), but make sure to use a different port (2181 is the default for ZooKeeper). The following is an example that uses port 2184 for a three-node ZooKeeper cluster:
+To deploy a ZooKeeper configuration store in a single-cluster instance, add the same ZooKeeper servers that the local quorum. You need to use the configuration file in [`conf/global_zookeeper.conf`](reference-configuration.md#configuration-store) using the same method for [local ZooKeeper](#local-zookeeper), but make sure to use a different port (2181 is the default for ZooKeeper). The following is an example that uses port 2184 for a three-node ZooKeeper cluster:
```properties
@@ -165,13 +159,11 @@ As before, create the `myid` files for each server on `data/global-zookeeper/myi
When you deploy a global Pulsar instance, with clusters distributed across different geographical regions, the configuration store serves as a highly available and strongly consistent metadata store that can tolerate failures and partitions spanning whole regions.
-The key here is to make sure the ZK quorum members are spread across at least 3 regions and that other regions run as observers.
+The key here is to make sure the ZK quorum members are spread across at least 3 regions, and other regions run as observers.
-Again, given the very low expected load on the configuration store servers, you can
-share the same hosts used for the local ZooKeeper quorum.
+Again, given the very low expected load on the configuration store servers, you can share the same hosts used for the local ZooKeeper quorum.
-For example, assume a Pulsar instance with the following clusters `us-west`,
-`us-east`, `us-central`, `eu-central`, `ap-south`. Also assume, each cluster has its own local ZK servers named such as the following:
+For example, assume a Pulsar instance with the following clusters `us-west`, `us-east`, `us-central`, `eu-central`, `ap-south`. Also assume, each cluster has its own local ZK servers named such as the following:
```
@@ -179,8 +171,7 @@ zk[1-3].${CLUSTER}.example.com
```
-In this scenario if you want to pick the quorum participants from few clusters and
-let all the others be ZK observers. For example, to form a 7 servers quorum, you can pick 3 servers from `us-west`, 2 from `us-central` and 2 from `us-east`.
+In this scenario if you want to pick the quorum participants from few clusters and let all the others be ZK observers. For example, to form a 7 servers quorum, you can pick 3 servers from `us-west`, 2 from `us-central` and 2 from `us-east`.
This method guarantees that writes to configuration store is possible even if one of these regions is unreachable.
@@ -227,7 +218,7 @@ $ bin/pulsar-daemon start configuration-store
## Cluster metadata initialization
-Once you set up the cluster-specific ZooKeeper and configuration store quorums for your instance, you need to write some metadata to ZooKeeper for each cluster in your instance. **you only needs to write these metadata once**.
+Once you set up the cluster-specific ZooKeeper and configuration store quorums for your instance, you need to write some metadata to ZooKeeper for each cluster in your instance. **you only need to write these metadata once**.
You can initialize this metadata using the [`initialize-cluster-metadata`](reference-cli-tools.md#pulsar-initialize-cluster-metadata) command of the [`pulsar`](reference-cli-tools.md#pulsar) CLI tool. The following is an example:
@@ -260,7 +251,7 @@ Make sure to run `initialize-cluster-metadata` for each cluster in your instance
BookKeeper provides [persistent message storage](concepts-architecture-overview.md#persistent-storage) for Pulsar.
-Each Pulsar broker needs to have its own cluster of bookies. The BookKeeper cluster shares a local ZooKeeper quorum with the Pulsar cluster.
+Each Pulsar broker needs its own cluster of bookies. The BookKeeper cluster shares a local ZooKeeper quorum with the Pulsar cluster.
### Configure bookies
@@ -280,7 +271,7 @@ $ bin/pulsar-daemon start bookie
You can verify that the bookie works properly using the `bookiesanity` command for the [BookKeeper shell](reference-cli-tools.md#bookkeeper-shell):
-```shell
+```bash
$ bin/bookkeeper shell bookiesanity
@@ -304,7 +295,7 @@ Bookie hosts are responsible for storing message data on disk. In order for book
Message entries written to bookies are always synced to disk before returning an acknowledgement to the Pulsar broker. To ensure low write latency, BookKeeper is
designed to use multiple devices:
-* A **journal** to ensure durability. For sequential writes, having fast [fsync](https://linux.die.net/man/2/fsync) operations on bookie hosts is critical. Typically, small and fast [solid-state drives](https://en.wikipedia.org/wiki/Solid-state_drive) (SSDs) should suffice, or [hard disk drives](https://en.wikipedia.org/wiki/Hard_disk_drive) (HDDs) with a [RAID](https://en.wikipedia.org/wiki/RAID)s controller and a battery-backed write cache. Both solutions can reach fsync latency of ~0.4 ms.
+* A **journal** to ensure durability. For sequential writes, having fast [fsync](https://linux.die.net/man/2/fsync) operations on bookie hosts is critical. Typically, small and fast [solid-state drives](https://en.wikipedia.org/wiki/Solid-state_drive) (SSDs) should suffice, or [hard disk drives](https://en.wikipedia.org/wiki/Hard_disk_drive) (HDDs) with a [RAID](https://en.wikipedia.org/wiki/RAID) controller and a battery-backed write cache. Both solutions can reach fsync latency of ~0.4 ms.
* A **ledger storage device** is where data is stored until all consumers acknowledge the message. Writes happen in the background, so write I/O is not a big concern. Reads happen sequentially most of the time and the backlog is drained only in case of consumer drain. To store large amounts of data, a typical configuration involves multiple HDDs with a RAID controller.
@@ -371,39 +362,13 @@ $ bin/pulsar broker
## Service discovery
-[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup).
+[Clients](getting-started-clients) connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL.
-You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means.
+You can use your own service discovery system. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means.
-> #### Service discovery already provided by many scheduling systems
+> **Service discovery already provided by many scheduling systems**
> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism.
-
-### Service discovery setup
-
-The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol) of Pulsar.
-
-To get started setting up the built-in service of discovery of Pulsar, you need to change a few parameters in the [`conf/discovery.conf`](reference-configuration.md#service-discovery) configuration file. Set the [`zookeeperServers`](reference-configuration.md#service-discovery-zookeeperServers) parameter to the ZooKeeper quorum connection string of the cluster and the [`configurationStoreServers`](reference-configuration.md#service-discovery-configurationStoreServers) setting to the [con [...]
-store](reference-terminology.md#configuration-store) quorum connection string.
-
-```properties
-
-# Zookeeper quorum connection string
-zookeeperServers=zk1.us-west.example.com:2181,zk2.us-west.example.com:2181,zk3.us-west.example.com:2181
-
-# Global configuration store connection string
-configurationStoreServers=zk1.us-west.example.com:2184,zk2.us-west.example.com:2184,zk3.us-west.example.com:2184
-
-```
-
-To start the discovery service:
-
-```shell
-
-$ bin/pulsar-daemon start discovery
-
-```
-
## Admin client and verification
At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients.
diff --git a/site2/website-next/versioned_docs/version-2.4.1/develop-bare-metal.md b/site2/website-next/versioned_docs/version-2.4.2/deploy-bare-metal.md
similarity index 100%
rename from site2/website-next/versioned_docs/version-2.4.1/develop-bare-metal.md
rename to site2/website-next/versioned_docs/version-2.4.2/deploy-bare-metal.md
diff --git a/site2/website-next/versioned_docs/version-2.4.2/deploy-dcos.md b/site2/website-next/versioned_docs/version-2.4.2/deploy-dcos.md
index f5f8d1f..07f446e 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/deploy-dcos.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/deploy-dcos.md
@@ -7,18 +7,17 @@ original_id: deploy-dcos
:::tip
-If you want to enable all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of
-`apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors).
+To enable all built-in [Pulsar IO](io-overview) connectors in your Pulsar deploymente, we recommend you use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image; the former has already bundled [all built-in connectors](io-overview.md#working-with-connectors).
:::
-[DC/OS](https://dcos.io/) (the <strong>D</strong>ata<strong>C</strong>enter <strong>O</strong>perating <strong>S</strong>ystem) is a distributed operating system used for deploying and managing applications and systems on [Apache Mesos](http://mesos.apache.org/). DC/OS is an open-source tool that [Mesosphere](https://mesosphere.com/) creates and maintains .
+[DC/OS](https://dcos.io/) (the <strong>D</strong>ata<strong>C</strong>enter <strong>O</strong>perating <strong>S</strong>ystem) is a distributed operating system for deploying and managing applications and systems on [Apache Mesos](http://mesos.apache.org/). DC/OS is an open-source tool created and maintained by [Mesosphere](https://mesosphere.com/).
Apache Pulsar is available as a [Marathon Application Group](https://mesosphere.github.io/marathon/docs/application-groups.html), which runs multiple applications as manageable sets.
## Prerequisites
-In order to run Pulsar on DC/OS, you need the following:
+You need to prepare your environment before running Pulsar on DC/OS.
* DC/OS version [1.9](https://docs.mesosphere.com/1.9/) or higher
* A [DC/OS cluster](https://docs.mesosphere.com/1.9/installing/) with at least three agent nodes
@@ -37,7 +36,7 @@ Each node in the DC/OS-managed Mesos cluster must have at least:
* 4 GB of memory
* 60 GB of total persistent disk
-Alternatively, you can change the configuration in `PulsarGroups.json` according to match your resources of DC/OS cluster.
+Alternatively, you can change the configuration in `PulsarGroups.json` accordingly to match your resources of the DC/OS cluster.
## Deploy Pulsar using the DC/OS command interface
@@ -56,9 +55,9 @@ This command deploys Docker container instances in three groups, which together
* 1 [Prometheus](http://prometheus.io/) instance and 1 [Grafana](https://grafana.com/) instance
-> When you run DC/OS, a ZooKeeper cluster already runs at `master.mesos:2181`, thus you do not have to install or start up ZooKeeper separately.
+> When you run DC/OS, a ZooKeeper cluster will be running at `master.mesos:2181`, thus you do not have to install or start up ZooKeeper separately.
-After executing the `dcos` command above, click on the **Services** tab in the DC/OS [GUI interface](https://docs.mesosphere.com/latest/gui/), which you can access at [http://m1.dcos](http://m1.dcos) in this example. You should see several applications in the process of deploying.
+After executing the `dcos` command above, click the **Services** tab in the DC/OS [GUI interface](https://docs.mesosphere.com/latest/gui/), which you can access at [http://m1.dcos](http://m1.dcos) in this example. You should see several applications during the deployment.
@@ -66,15 +65,15 @@ After executing the `dcos` command above, click on the **Services** tab in the D
## The BookKeeper group
-To monitor the status of the BookKeeper cluster deployment, click on the **bookkeeper** group in the parent **pulsar** group.
+To monitor the status of the BookKeeper cluster deployment, click the **bookkeeper** group in the parent **pulsar** group.
-At this point, 3 [bookies](reference-terminology.md#bookie) should be shown as green, which means that the bookies have been deployed successfully and are now running.
+At this point, the status of the 3 [bookies](reference-terminology.md#bookie) are green, which means that the bookies have been deployed successfully and are running.
-You can also click into each bookie instance to get more detailed information, such as the bookie running log.
+You can also click each bookie instance to get more detailed information, such as the bookie running log.
@@ -82,23 +81,23 @@ To display information about the BookKeeper in ZooKeeper, you can visit [http://
-## The Pulsar broker Group
+## The Pulsar broker group
-Similar to the BookKeeper group above, click into the **brokers** to check the status of the Pulsar brokers.
+Similar to the BookKeeper group above, click **brokers** to check the status of the Pulsar brokers.
-You can also click into each broker instance to get more detailed information, such as the broker running log.
+You can also click each broker instance to get more detailed information, such as the broker running log.
-Broker cluster information in Zookeeper is also available through the web UI. In this example, you can see that the `loadbalance` and `managed-ledgers` directories have been created.
+Broker cluster information in ZooKeeper is also available through the web UI. In this example, you can see that the `loadbalance` and `managed-ledgers` directories have been created.
-## Monitor Group
+## Monitor group
The **monitory** group consists of Prometheus and Grafana.
@@ -106,17 +105,17 @@ The **monitory** group consists of Prometheus and Grafana.
### Prometheus
-Click into the instance of `prom` to get the endpoint of Prometheus, which is `192.168.65.121:9090` in this example.
+Click the instance of `prom` to get the endpoint of Prometheus, which is `192.168.65.121:9090` in this example.
-If you click that endpoint, you can see the Prometheus dashboard. The [http://192.168.65.121:9090/targets](http://192.168.65.121:9090/targets) URL display all the bookies and brokers.
+If you click that endpoint, you can see the Prometheus dashboard. All the bookies and brokers are listed on [http://192.168.65.121:9090/targets](http://192.168.65.121:9090/targets).
### Grafana
-Click into `grafana` to get the endpoint for Grafana, which is `192.168.65.121:3000` in this example.
+Click `grafana` to get the endpoint for Grafana, which is `192.168.65.121:3000` in this example.
@@ -130,7 +129,7 @@ Now that you have a fully deployed Pulsar cluster, you can run a simple consumer
### Download and prepare the Pulsar Java tutorial
-You can clone a [Pulsar Java tutorial](https://github.com/streamlio/pulsar-java-tutorial) repo. This repo contains a simple Pulsar consumer and producer (you can find more information in the `README` file of the repo).
+You can clone a [Pulsar Java tutorial](https://github.com/streamlio/pulsar-java-tutorial) repo. This repo contains a simple Pulsar consumer and producer (you can find more information in the `README` file in this repo).
```bash
@@ -138,12 +137,13 @@ $ git clone https://github.com/streamlio/pulsar-java-tutorial
```
-Change the `SERVICE_URL` from `pulsar://localhost:6650` to `pulsar://a1.dcos:6650` in both [`ConsumerTutorial.java`](https://github.com/streamlio/pulsar-java-tutorial/blob/master/src/main/java/tutorial/ConsumerTutorial.java) and [`ProducerTutorial.java`](https://github.com/streamlio/pulsar-java-tutorial/blob/master/src/main/java/tutorial/ProducerTutorial.java).
-The `pulsar://a1.dcos:6650` endpoint is for the broker service. You can fetch the endpoint details for each broker instance from the DC/OS GUI. `a1.dcos` is a DC/OS client agent, which runs a broker. The client agent IP address can also replace this.
+Change the `SERVICE_URL` from `pulsar://localhost:6650` to `pulsar://a1.dcos:6650` in both [`ConsumerTutorial.java`](https://github.com/streamlio/pulsar-java-tutorial/blob/master/src/main/java/tutorial/ConsumerTutorial.java) file and [`ProducerTutorial.java`](https://github.com/streamlio/pulsar-java-tutorial/blob/master/src/main/java/tutorial/ProducerTutorial.java) file.
-Now, change the message number from 10 to 10000000 in main method of [`ProducerTutorial.java`](https://github.com/streamlio/pulsar-java-tutorial/blob/master/src/main/java/tutorial/ProducerTutorial.java) so that it can produce more messages.
+The `pulsar://a1.dcos:6650` endpoint is for the broker service. You can fetch the endpoint details for each broker instance from the DC/OS GUI. `a1.dcos` is a DC/OS client agent that runs a broker, and you can replace it with the client agent IP address.
-Now compile the project code using the command below:
+Now, you can change the message number from 10 to 10000000 in the main method in [`ProducerTutorial.java`](https://github.com/streamlio/pulsar-java-tutorial/blob/master/src/main/java/tutorial/ProducerTutorial.java) file to produce more messages.
+
+Then, you can compile the project code using the command below:
```bash
@@ -169,7 +169,7 @@ $ mvn exec:java -Dexec.mainClass="tutorial.ProducerTutorial"
```
-You can see the producer producing messages and the consumer consuming messages through the DC/OS GUI.
+You see that the producer is producing messages and the consumer is consuming messages through the DC/OS GUI.
@@ -177,20 +177,20 @@ You can see the producer producing messages and the consumer consuming messages
### View Grafana metric output
-While the producer and consumer run, you can access running metrics information from Grafana.
+While the producer and consumer are running, you can access the running metrics from Grafana.
## Uninstall Pulsar
-You can shut down and uninstall the `pulsar` application from DC/OS at any time in the following two ways:
+You can shut down and uninstall the `pulsar` application from DC/OS at any time in one of the following two ways:
-1. Using the DC/OS GUI, you can choose **Delete** at the right end of Pulsar group.
+1. Click the three dots at the right end of Pulsar group and choose **Delete** on the DC/OS GUI.
-2. You can use the following command:
+2. Use the command below.
```bash
diff --git a/site2/website-next/versioned_docs/version-2.4.2/deploy-kubernetes.md b/site2/website-next/versioned_docs/version-2.4.2/deploy-kubernetes.md
index dc7123d..4e170dc 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/deploy-kubernetes.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/deploy-kubernetes.md
@@ -2,7 +2,6 @@
id: deploy-kubernetes
title: Deploy Pulsar on Kubernetes
sidebar_label: "Kubernetes"
-original_id: deploy-kubernetes
---
To get up and running with these charts as fast as possible, in a **non-production** use case, we provide
diff --git a/site2/website-next/versioned_docs/version-2.4.2/deploy-monitoring.md b/site2/website-next/versioned_docs/version-2.4.2/deploy-monitoring.md
index 074ce3f..95ccdd6 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/deploy-monitoring.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/deploy-monitoring.md
@@ -2,7 +2,6 @@
id: deploy-monitoring
title: Monitor
sidebar_label: "Monitor"
-original_id: deploy-monitoring
---
You can use different ways to monitor a Pulsar cluster, exposing both metrics related to the usage of topics and the overall health of the individual components of the cluster.
@@ -127,17 +126,7 @@ The per-topic dashboard instructions are available at [Pulsar manager](administr
You can use grafana to create dashboard driven by the data that is stored in Prometheus.
-When you deploy Pulsar on Kubernetes, a `pulsar-grafana` Docker image is enabled by default. You can use the docker image with the principal dashboards.
-
-Enter the command below to use the dashboard manually:
-
-```shell
-
-docker run -p3000:3000 \
- -e PROMETHEUS_URL=http://$PROMETHEUS_HOST:9090/ \
- apachepulsar/pulsar-grafana:latest
-
-```
+When you deploy Pulsar on Kubernetes with the Pulsar Helm Chart, a `pulsar-grafana` Docker image is enabled by default. You can use the docker image with the principal dashboards.
The following are some Grafana dashboards examples:
@@ -145,4 +134,4 @@ The following are some Grafana dashboards examples:
- [apache-pulsar-grafana-dashboard](https://github.com/streamnative/apache-pulsar-grafana-dashboard): a collection of Grafana dashboard templates for different Pulsar components running on both Kubernetes and on-premise machines.
## Alerting rules
-You can set alerting rules according to your Pulsar environment. To configure alerting rules for Apache Pulsar, refer to [alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/).
+You can set alerting rules according to your Pulsar environment. To configure alerting rules for Apache Pulsar, refer to [alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/).
\ No newline at end of file
diff --git a/site2/website-next/versioned_docs/version-2.4.2/develop-binary-protocol.md b/site2/website-next/versioned_docs/version-2.4.2/develop-binary-protocol.md
index b233f10..fa03383 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/develop-binary-protocol.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/develop-binary-protocol.md
@@ -2,7 +2,6 @@
id: develop-binary-protocol
title: Pulsar binary protocol specification
sidebar_label: "Binary protocol"
-original_id: develop-binary-protocol
---
Pulsar uses a custom binary protocol for communications between producers/consumers and brokers. This protocol is designed to support required features, such as acknowledgements and flow control, while ensuring maximum transport and implementation efficiency.
@@ -29,42 +28,63 @@ The Pulsar protocol allows for two types of commands:
Simple (payload-free) commands have this basic structure:
-| Component | Description | Size (in bytes) |
-|:------------|:----------------------------------------------------------------------------------------|:----------------|
-| totalSize | The size of the frame, counting everything that comes after it (in bytes) | 4 |
-| commandSize | The size of the protobuf-serialized command | 4 |
-| message | The protobuf message serialized in a raw binary format (rather than in protobuf format) | |
+| Component | Description | Size (in bytes) |
+|:--------------|:----------------------------------------------------------------------------------------|:----------------|
+| `totalSize` | The size of the frame, counting everything that comes after it (in bytes) | 4 |
+| `commandSize` | The size of the protobuf-serialized command | 4 |
+| `message` | The protobuf message serialized in a raw binary format (rather than in protobuf format) | |
### Payload commands
Payload commands have this basic structure:
-| Component | Description | Size (in bytes) |
-|:-------------|:--------------------------------------------------------------------------------------------|:----------------|
-| totalSize | The size of the frame, counting everything that comes after it (in bytes) | 4 |
-| commandSize | The size of the protobuf-serialized command | 4 |
-| message | The protobuf message serialized in a raw binary format (rather than in protobuf format) | |
-| magicNumber | A 2-byte byte array (`0x0e01`) identifying the current format | 2 |
-| checksum | A [CRC32-C checksum](http://www.evanjones.ca/crc32c.html) of everything that comes after it | 4 |
-| metadataSize | The size of the message [metadata](#message-metadata) | 4 |
-| metadata | The message [metadata](#message-metadata) stored as a binary protobuf message | |
-| payload | Anything left in the frame is considered the payload and can include any sequence of bytes | |
+| Component | Required or optional| Description | Size (in bytes) |
+|:-----------------------------------|:----------|:--------------------------------------------------------------------------------------------|:----------------|
+| `totalSize` | Required | The size of the frame, counting everything that comes after it (in bytes) | 4 |
+| `commandSize` | Required | The size of the protobuf-serialized command | 4 |
+| `message` | Required | The protobuf message serialized in a raw binary format (rather than in protobuf format) | |
+| `magicNumberOfBrokerEntryMetadata` | Optional | A 2-byte byte array (`0x0e02`) identifying the broker entry metadata <br /> **Note**: `magicNumberOfBrokerEntryMetadata` , `brokerEntryMetadataSize`, and `brokerEntryMetadata` should be used **together**. | 2 |
+| `brokerEntryMetadataSize` | Optional | The size of the broker entry metadata | 4 |
+| `brokerEntryMetadata` | Optional | The broker entry metadata stored as a binary protobuf message | |
+| `magicNumber` | Required | A 2-byte byte array (`0x0e01`) identifying the current format | 2 |
+| `checksum` | Required | A [CRC32-C checksum](http://www.evanjones.ca/crc32c.html) of everything that comes after it | 4 |
+| `metadataSize` | Required | The size of the message [metadata](#message-metadata) | 4 |
+| `metadata` | Required | The message [metadata](#message-metadata) stored as a binary protobuf message | |
+| `payload` | Required | Anything left in the frame is considered the payload and can include any sequence of bytes | |
+
+## Broker entry metadata
+
+Broker entry metadata is stored alongside the message metadata as a serialized protobuf message.
+It is created by the broker when the message arrived at the broker and passed without changes to the consumer if configured.
+
+| Field | Required or optional | Description |
+|:-------------------|:----------------|:------------------------------------------------------------------------------------------------------------------------------|
+| `broker_timestamp` | Optional | The timestamp when a message arrived at the broker (`id est` as the number of milliseconds since January 1st, 1970 in UTC) |
+| `index` | Optional | The index of the message. It is assigned by the broker.
+
+If you want to use broker entry metadata for **brokers**, configure the [`brokerEntryMetadataInterceptors`](reference-configuration.md#broker) parameter in the `broker.conf` file.
+
+If you want to use broker entry metadata for **consumers**:
+
+1. Use the client protocol version [18 or later](https://github.com/apache/pulsar/blob/ca37e67211feda4f7e0984e6414e707f1c1dfd07/pulsar-common/src/main/proto/PulsarApi.proto#L259).
+
+2. Configure the [`brokerEntryMetadataInterceptors`](reference-configuration.md#broker) parameter and set the [`enableExposingBrokerEntryMetadataToClient`](reference-configuration.md#broker) parameter to `true` in the `broker.conf` file.
## Message metadata
-Message metadata is stored alongside the application-specified payload as a serialized protobuf message. Metadata is created by the producer and passed on unchanged to the consumer.
+Message metadata is stored alongside the application-specified payload as a serialized protobuf message. Metadata is created by the producer and passed without changes to the consumer.
-| Field | Description |
-|:-------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `producer_name` | The name of the producer that published the message |
-| `sequence_id` | The sequence ID of the message, assigned by producer |
-| `publish_time` | The publish timestamp in Unix time (i.e. as the number of milliseconds since January 1st, 1970 in UTC) |
-| `properties` | A sequence of key/value pairs (using the [`KeyValue`](https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/proto/PulsarApi.proto#L32) message). These are application-defined keys and values with no special meaning to Pulsar. |
-| `replicated_from` *(optional)* | Indicates that the message has been replicated and specifies the name of the [cluster](reference-terminology.md#cluster) where the message was originally published |
-| `partition_key` *(optional)* | While publishing on a partition topic, if the key is present, the hash of the key is used to determine which partition to choose |
-| `compression` *(optional)* | Signals that payload has been compressed and with which compression library |
-| `uncompressed_size` *(optional)* | If compression is used, the producer must fill the uncompressed size field with the original payload size |
-| `num_messages_in_batch` *(optional)* | If this message is really a [batch](#batch-messages) of multiple entries, this field must be set to the number of messages in the batch |
+| Field | Required or optional | Description |
+|:-------------------------|:----------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `producer_name` | Required | The name of the producer that published the message |
+| `sequence_id` | Required | The sequence ID of the message, assigned by producer |
+| `publish_time` | Required | The publish timestamp in Unix time (i.e. as the number of milliseconds since January 1st, 1970 in UTC) |
+| `properties` | Required | A sequence of key/value pairs (using the [`KeyValue`](https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/proto/PulsarApi.proto#L32) message). These are application-defined keys and values with no special meaning to Pulsar. |
+| `replicated_from` | Optional | Indicates that the message has been replicated and specifies the name of the [cluster](reference-terminology.md#cluster) where the message was originally published |
+| `partition_key` | Optional | While publishing on a partition topic, if the key is present, the hash of the key is used to determine which partition to choose. Partition key is used as the message key. |
+| `compression` | Optional | Signals that payload has been compressed and with which compression library |
+| `uncompressed_size` | Optional | If compression is used, the producer must fill the uncompressed size field with the original payload size |
+| `num_messages_in_batch` | Optional | If this message is really a [batch](#batch-messages) of multiple entries, this field must be set to the number of messages in the batch |
### Batch messages
@@ -76,19 +96,19 @@ object.
For a single batch, the payload format will look like this:
-| Field | Description |
-|:--------------|:------------------------------------------------------------|
-| metadataSizeN | The size of the single message metadata serialized Protobuf |
-| metadataN | Single message metadata |
-| payloadN | Message payload passed by application |
+| Field | Required or optional | Description |
+|:----------------|:---------------------|:-----------------------------------------------------------|
+| `metadataSizeN` | Required |The size of the single message metadata serialized Protobuf |
+| `metadataN` | Required |Single message metadata |
+| `payloadN` | Required |Message payload passed by application |
Each metadata field looks like this;
-| Field | Description |
-|:---------------------------|:--------------------------------------------------------|
-| properties | Application-defined properties |
-| partition key *(optional)* | Key to indicate the hashing to a particular partition |
-| payload_size | Size of the payload for the single message in the batch |
+| Field | Required or optional | Description |
+|:----------------|:----------------------|:--------------------------------------------------------|
+| `properties` | Required | Application-defined properties |
+| `partition key` | Optional | Key to indicate the hashing to a particular partition |
+| `payload_size` | Required | Size of the payload for the single message in the batch |
When compression is enabled, the whole batch will be compressed at once.
@@ -170,6 +190,10 @@ messages to the broker, referring to the producer id negotiated before.
+If the client does not receive a response indicating producer creation success or failure,
+the client should first send a command to close the original producer before sending a
+command to re-attempt producer creation.
+
##### Command Producer
```protobuf
@@ -273,6 +297,11 @@ When receiving a `CloseProducer` command, the broker will stop accepting any
more messages for the producer, wait until all pending messages are persisted
and then reply `Success` to the client.
+If the client does not receive a response to a `Producer` command within a timeout,
+the client must first send a `CloseProducer` command before sending another
+`Producer` command. The client does not need to await a response to the `CloseProducer`
+command before sending the next `Producer` command.
+
The broker can send a `CloseProducer` command to client when it's performing
a graceful failover (eg: broker is being restarted, or the topic is being unloaded
by load balancer to be transferred to a different broker).
diff --git a/site2/website-next/versioned_docs/version-2.4.2/develop-load-manager.md b/site2/website-next/versioned_docs/version-2.4.2/develop-load-manager.md
index 509209b..9687f30 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/develop-load-manager.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/develop-load-manager.md
@@ -2,7 +2,6 @@
id: develop-load-manager
title: Modular load manager
sidebar_label: "Modular load manager"
-original_id: develop-load-manager
---
The *modular load manager*, implemented in [`ModularLoadManagerImpl`](https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImpl.java), is a flexible alternative to the previously implemented load manager, [`SimpleLoadManagerImpl`](https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/SimpleLoadManagerImpl.java), which attempts to simplify how load [...]
diff --git a/site2/website-next/versioned_docs/version-2.4.2/develop-tools.md b/site2/website-next/versioned_docs/version-2.4.2/develop-tools.md
index b545779..d034926 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/develop-tools.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/develop-tools.md
@@ -2,7 +2,6 @@
id: develop-tools
title: Simulation tools
sidebar_label: "Simulation tools"
-original_id: develop-tools
---
It is sometimes necessary create an test environment and incur artificial load to observe how well load managers
diff --git a/site2/website-next/versioned_docs/version-2.4.2/io-cdc.md b/site2/website-next/versioned_docs/version-2.4.2/io-cdc.md
index 20f16ae..df37397 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/io-cdc.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/io-cdc.md
@@ -2,7 +2,6 @@
id: io-cdc
title: CDC connector
sidebar_label: "CDC connector"
-original_id: io-cdc
---
CDC source connectors capture log changes of databases (such as MySQL, MongoDB, and PostgreSQL) into Pulsar.
diff --git a/site2/website-next/versioned_docs/version-2.4.2/io-overview.md b/site2/website-next/versioned_docs/version-2.4.2/io-overview.md
index 68960a8..810de78 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/io-overview.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/io-overview.md
@@ -2,7 +2,6 @@
id: io-overview
title: Pulsar connector overview
sidebar_label: "Overview"
-original_id: io-overview
---
import Tabs from '@theme/Tabs';
diff --git a/site2/website-next/versioned_docs/version-2.4.2/pulsar-2.0.md b/site2/website-next/versioned_docs/version-2.4.2/pulsar-2.0.md
index 11c5e66..560c8c1 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/pulsar-2.0.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/pulsar-2.0.md
@@ -2,7 +2,6 @@
id: pulsar-2.0
title: Pulsar 2.0
sidebar_label: "Pulsar 2.0"
-original_id: pulsar-2.0
---
Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more.
diff --git a/site2/website-next/versioned_docs/version-2.4.2/reference-pulsar-admin.md b/site2/website-next/versioned_docs/version-2.4.2/pulsar-admin.md
similarity index 100%
rename from site2/website-next/versioned_docs/version-2.4.2/reference-pulsar-admin.md
rename to site2/website-next/versioned_docs/version-2.4.2/pulsar-admin.md
diff --git a/site2/website-next/versioned_docs/version-2.4.2/reference-cli-tools.md b/site2/website-next/versioned_docs/version-2.4.2/reference-cli-tools.md
index 8c2c64f..0c8aea1 100644
--- a/site2/website-next/versioned_docs/version-2.4.2/reference-cli-tools.md
+++ b/site2/website-next/versioned_docs/version-2.4.2/reference-cli-tools.md
@@ -2,7 +2,6 @@
id: reference-cli-tools
title: Pulsar command-line tools
sidebar_label: "Pulsar CLI tools"
-original_id: reference-cli-tools
---
Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more.
@@ -16,8 +15,12 @@ All Pulsar command-line tools can be run from the `bin` directory of your [insta
* [`bookkeeper`](#bookkeeper)
* [`broker-tool`](#broker-tool)
-> ### Getting help
-> You can get help for any CLI tool, command, or subcommand using the `--help` flag, or `-h` for short. Here's an example:
+> **Important**
+>
+> - This page only shows **some frequently used commands**. For the latest information about `pulsar`, `pulsar-client`, and `pulsar-perf`, including commands, flags, descriptions, and more information, see [Pulsar tools](https://pulsar.apache.org/tools/).
+>
+> - You can get help for any CLI tool, command, or subcommand using the `--help` flag, or `-h` for short. Here's an example:
+>
> ```shell
... 810 lines suppressed ...