You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by ur...@apache.org on 2022/05/07 12:03:50 UTC
[pulsar-site] branch main updated: Docs sync done from apache/pulsar(#2a6b0ce)
This is an automated email from the ASF dual-hosted git repository.
urfree pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git
The following commit(s) were added to refs/heads/main by this push:
new c33f618f7dc Docs sync done from apache/pulsar(#2a6b0ce)
c33f618f7dc is described below
commit c33f618f7dc7658bf32304e0c24d1ef062f42e88
Author: Pulsar Site Updater <de...@pulsar.apache.org>
AuthorDate: Sat May 7 12:03:47 2022 +0000
Docs sync done from apache/pulsar(#2a6b0ce)
---
site2/docs/tiered-storage-filesystem.md | 172 +++++++++---------
.../website-next/docs/tiered-storage-filesystem.md | 194 ++++++++++-----------
2 files changed, 177 insertions(+), 189 deletions(-)
diff --git a/site2/docs/tiered-storage-filesystem.md b/site2/docs/tiered-storage-filesystem.md
index b9dca2d1dad..ad0cd659170 100644
--- a/site2/docs/tiered-storage-filesystem.md
+++ b/site2/docs/tiered-storage-filesystem.md
@@ -24,7 +24,7 @@ This example uses Pulsar 2.5.1.
* Download the Pulsar tarball from the Pulsar [download page](https://pulsar.apache.org/download)
- * Use the [wget](https://www.gnu.org/software/wget) command to dowload the Pulsar tarball.
+ * Use the [wget](https://www.gnu.org/software/wget) command to download the Pulsar tarball.
```shell
wget https://archive.apache.org/dist/pulsar/pulsar-2.5.1/apache-pulsar-2.5.1-bin.tar.gz
@@ -40,7 +40,7 @@ This example uses Pulsar 2.5.1.
> **Note**
>
- > * If you run Pulsar in a bare metal cluster, ensure that the `offloaders` tarball is unzipped in every broker's Pulsar directory.
+ > * If you run Pulsar in a bare-metal cluster, ensure that the `offloaders` tarball is unzipped in every broker's Pulsar directory.
>
> * If you run Pulsar in Docker or deploying Pulsar using a Docker image (such as K8S and DCOS), you can use the `apachepulsar/pulsar-all` image. The `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders.
@@ -61,7 +61,7 @@ This example uses Pulsar 2.5.1.
> **Note**
>
- > * If you run Pulsar in a bare metal cluster, ensure that `offloaders` tarball is unzipped in every broker's Pulsar directory.
+ > * If you run Pulsar in a bare-metal cluster, ensure that `offloaders` tarball is unzipped in every broker's Pulsar directory.
>
> * If you run Pulsar in Docker or deploying Pulsar using a Docker image (such as K8s and DCOS), you can use the `apachepulsar/pulsar-all` image. The `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders.
@@ -139,7 +139,7 @@ pulsar-admin namespaces set-offload-threshold --size 10M my-tenant/my-namespace
> **Tip**
>
-> For more information about the `pulsar-admin namespaces set-offload-threshold options` command, including flags, descriptions, default values, and shorthands, see [here](reference-pulsar-admin.md#set-offload-threshold).
+> For more information about the `pulsar-admin namespaces set-offload-threshold options` command, including flags, descriptions, default values, and shorthands, see [here](https://pulsar.apache.org/tools/pulsar-admin/).
### Run filesystem offloader manually
@@ -153,7 +153,7 @@ To manually trigger the filesystem offloader via CLI tools, you need to specify
#### Example
-- This example manually run the filesystem offloader using pulsar-admin.
+- This example manually runs the filesystem offloader using pulsar-admin.
```bash
pulsar-admin topics offload --size-threshold 10M persistent://my-tenant/my-namespace/topic1
@@ -210,24 +210,20 @@ To manually trigger the filesystem offloader via CLI tools, you need to specify
> **Tip**
>
- > For more information about the `pulsar-admin topics offload-status options` command, including flags, descriptions, default values, and shorthands, see [here](reference-pulsar-admin.md#offload-status).
+ > For more information about the `pulsar-admin topics offload-status options` command, including flags, descriptions, default values, and shorthands, see [here](https://pulsar.apache.org/tools/pulsar-admin/).
## Tutorial
This section provides step-by-step instructions on how to use the filesystem offloader to move data from Pulsar to Hadoop Distributed File System (HDFS) or Network File system (NFS).
-<!--DOCUSAURUS_CODE_TABS-->
-<!--HDFS-->
-
-To move data from Pulsar to HDFS, follow these steps.
-
-### Step 1: Prepare the HDFS environment
-
-This tutorial sets up a Hadoop single node cluster and uses Hadoop 3.2.1.
+### Offload data to HDFS
> **Tip**
>
-> For details about how to set up a Hadoop single node cluster, see [here](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html).
+> This tutorial sets up a Hadoop single node cluster and uses Hadoop 3.2.1. For details about how to set up a Hadoop single node cluster, see [here](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html).
+
+
+#### Step 1: Prepare the HDFS environment
1. Download and uncompress Hadoop 3.2.1.
@@ -292,11 +288,11 @@ This tutorial sets up a Hadoop single node cluster and uses Hadoop 3.2.1.
-### Step 2: Install the filesystem offloader
+#### Step 2: Install the filesystem offloader
For details, see [installation](#installation).
-### Step 3: Configure the filesystem offloader
+#### Step 3: Configure the filesystem offloader
As indicated in the [configuration](#configuration) section, you need to configure some properties for the filesystem offloader driver before using it. This tutorial assumes that you have configured the filesystem offloader driver as below and run Pulsar in **standalone** mode.
@@ -317,74 +313,7 @@ managedLedgerMinLedgerRolloverTimeMinutes=1
managedLedgerMaxEntriesPerLedger=100
```
-<!--NFS-->
-
-> **Note**
->
-> In this section, it is assumed that you have enabled NFS service and set the shared path of your NFS service. In this section, `/Users/test` is used as the shared path of NFS service.
-
-To offload data to NFS, follow these steps.
-
-### Step 1: Install the filesystem offloader
-
-For details, see [installation](#installation).
-
-### Step 2: Mont your NFS to your local filesystem
-
-This example mounts mounts */Users/pulsar_nfs* to */Users/test*.
-
-```
-mount -e 192.168.0.103:/Users/test/Users/pulsar_nfs
-```
-
-### Step 3: Configure the filesystem offloader driver
-
-As indicated in the [configuration](#configuration) section, you need to configure some properties for the filesystem offloader driver before using it. This tutorial assumes that you have configured the filesystem offloader driver as below and run Pulsar in **standalone** mode.
-
-1. Set the following configurations in the `conf/standalone.conf` file.
-
- ```conf
- managedLedgerOffloadDriver=filesystem
- fileSystemProfilePath=../conf/filesystem_offload_core_site.xml
- ```
-
-2. Modify the *filesystem_offload_core_site.xml* as follows.
-
- ```
- <property>
- <name>fs.defaultFS</name>
- <value>file:///</value>
- </property>
-
- <property>
- <name>hadoop.tmp.dir</name>
- <value>file:///Users/pulsar_nfs</value>
- </property>
-
- <property>
- <name>io.file.buffer.size</name>
- <value>4096</value>
- </property>
-
- <property>
- <name>io.seqfile.compress.blocksize</name>
- <value>1000000</value>
- </property>
- <property>
-
- <name>io.seqfile.compression.type</name>
- <value>BLOCK</value>
- </property>
-
- <property>
- <name>io.map.index.interval</name>
- <value>128</value>
- </property>
- ```
-
-<!--END_DOCUSAURUS_CODE_TABS-->
-
-### Step 4: Offload data from BookKeeper to filesystem
+#### Step 4: Offload data from BookKeeper to filesystem
Execute the following commands in the repository where you download Pulsar tarball. For example, `~/path/to/apache-pulsar-2.5.1`.
@@ -394,7 +323,7 @@ Execute the following commands in the repository where you download Pulsar tarba
bin/pulsar standalone -a 127.0.0.1
```
-2. To ensure the data generated is not deleted immediately, it is recommended to set the [retention policy](https://pulsar.apache.org/docs/en/next/cookbooks-retention-expiry/#retention-policies), which can be either a **size** limit or a **time** limit. The larger value you set for the retention policy, the longer the data can be retained.
+2. To ensure the data generated is not deleted immediately, it is recommended to set the [retention policy](cookbooks-retention-expiry.md/#retention-policies), which can be either a **size** limit or a **time** limit. The larger value you set for the retention policy, the longer the data can be retained.
```
bin/pulsar-admin namespaces set-retention public/default --size 100M --time 2d
@@ -435,7 +364,7 @@ Execute the following commands in the repository where you download Pulsar tarba
}
```
-5. Wait a second and send more messages to the topic.
+5. Wait for a second and send more messages to the topic.
```
bin/pulsar-client produce -m "Hello FileSystem Offloader" -n 1000 public/default/fs-test
@@ -522,6 +451,75 @@ Execute the following commands in the repository where you download Pulsar tarba
+
+### Offload data to NFS
+
+> **Note**
+>
+> In this section, it is assumed that you have enabled NFS service and set the shared path of your NFS service. In this section, `/Users/test` is used as the shared path of NFS service.
+
+#### Step 1: Install the filesystem offloader
+
+For details, see [installation](#installation).
+
+#### Step 2: Mount your NFS to your local filesystem
+
+This example mounts */Users/pulsar_nfs* to */Users/test*.
+
+```
+mount -e 192.168.0.103:/Users/test/Users/pulsar_nfs
+```
+
+#### Step 3: Configure the filesystem offloader driver
+
+As indicated in the [configuration](#configuration) section, you need to configure some properties for the filesystem offloader driver before using it. This tutorial assumes that you have configured the filesystem offloader driver as below and run Pulsar in **standalone** mode.
+
+1. Set the following configurations in the `conf/standalone.conf` file.
+
+ ```conf
+ managedLedgerOffloadDriver=filesystem
+ fileSystemProfilePath=../conf/filesystem_offload_core_site.xml
+ ```
+
+2. Modify the *filesystem_offload_core_site.xml* as follows.
+
+ ```
+ <property>
+ <name>fs.defaultFS</name>
+ <value>file:///</value>
+ </property>
+
+ <property>
+ <name>hadoop.tmp.dir</name>
+ <value>file:///Users/pulsar_nfs</value>
+ </property>
+
+ <property>
+ <name>io.file.buffer.size</name>
+ <value>4096</value>
+ </property>
+
+ <property>
+ <name>io.seqfile.compress.blocksize</name>
+ <value>1000000</value>
+ </property>
+ <property>
+
+ <name>io.seqfile.compression.type</name>
+ <value>BLOCK</value>
+ </property>
+
+ <property>
+ <name>io.map.index.interval</name>
+ <value>128</value>
+ </property>
+ ```
+
+#### Step 4: Offload data from BookKeeper to filesystem
+
+Refer to the step 4 of [Offload data to HDFS](#step-4-offload-data-from-bookkeeper-to-filesystem).
+
+
## Read offloaded data from filesystem
* The offloaded data is stored as `MapFile` in the following new path of the filesystem:
diff --git a/site2/website-next/docs/tiered-storage-filesystem.md b/site2/website-next/docs/tiered-storage-filesystem.md
index 31a6249c0e5..cbb6c4a99f3 100644
--- a/site2/website-next/docs/tiered-storage-filesystem.md
+++ b/site2/website-next/docs/tiered-storage-filesystem.md
@@ -30,7 +30,7 @@ This example uses Pulsar 2.5.1.
* Download the Pulsar tarball from the Pulsar [download page](https://pulsar.apache.org/download)
- * Use the [wget](https://www.gnu.org/software/wget) command to dowload the Pulsar tarball.
+ * Use the [wget](https://www.gnu.org/software/wget) command to download the Pulsar tarball.
```shell
@@ -50,7 +50,7 @@ This example uses Pulsar 2.5.1.
:::note
- * If you run Pulsar in a bare metal cluster, ensure that the `offloaders` tarball is unzipped in every broker's Pulsar directory.
+ * If you run Pulsar in a bare-metal cluster, ensure that the `offloaders` tarball is unzipped in every broker's Pulsar directory.
* If you run Pulsar in Docker or deploying Pulsar using a Docker image (such as K8S and DCOS), you can use the `apachepulsar/pulsar-all` image. The `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders.
:::
@@ -76,7 +76,7 @@ This example uses Pulsar 2.5.1.
:::note
- * If you run Pulsar in a bare metal cluster, ensure that `offloaders` tarball is unzipped in every broker's Pulsar directory.
+ * If you run Pulsar in a bare-metal cluster, ensure that `offloaders` tarball is unzipped in every broker's Pulsar directory.
* If you run Pulsar in Docker or deploying Pulsar using a Docker image (such as K8s and DCOS), you can use the `apachepulsar/pulsar-all` image. The `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders.
:::
@@ -166,7 +166,7 @@ pulsar-admin namespaces set-offload-threshold --size 10M my-tenant/my-namespace
:::tip
-For more information about the `pulsar-admin namespaces set-offload-threshold options` command, including flags, descriptions, default values, and shorthands, see [here](reference-pulsar-admin.md#set-offload-threshold).
+For more information about the `pulsar-admin namespaces set-offload-threshold options` command, including flags, descriptions, default values, and shorthands, see [here](https://pulsar.apache.org/tools/pulsar-admin/).
:::
@@ -182,7 +182,7 @@ To manually trigger the filesystem offloader via CLI tools, you need to specify
#### Example
-- This example manually run the filesystem offloader using pulsar-admin.
+- This example manually runs the filesystem offloader using pulsar-admin.
```bash
@@ -257,7 +257,7 @@ To manually trigger the filesystem offloader via CLI tools, you need to specify
:::tip
- For more information about the `pulsar-admin topics offload-status options` command, including flags, descriptions, default values, and shorthands, see [here](reference-pulsar-admin.md#offload-status).
+ For more information about the `pulsar-admin topics offload-status options` command, including flags, descriptions, default values, and shorthands, see [here](https://pulsar.apache.org/tools/pulsar-admin/).
:::
@@ -265,24 +265,16 @@ To manually trigger the filesystem offloader via CLI tools, you need to specify
This section provides step-by-step instructions on how to use the filesystem offloader to move data from Pulsar to Hadoop Distributed File System (HDFS) or Network File system (NFS).
-````mdx-code-block
-<Tabs
- defaultValue="HDFS"
- values={[{"label":"HDFS","value":"HDFS"},{"label":"NFS","value":"NFS"}]}>
-<TabItem value="HDFS">
-
-To move data from Pulsar to HDFS, follow these steps.
-
-### Step 1: Prepare the HDFS environment
-
-This tutorial sets up a Hadoop single node cluster and uses Hadoop 3.2.1.
+### Offload data to HDFS
:::tip
-For details about how to set up a Hadoop single node cluster, see [here](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html).
+This tutorial sets up a Hadoop single node cluster and uses Hadoop 3.2.1. For details about how to set up a Hadoop single node cluster, see [here](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html).
:::
+#### Step 1: Prepare the HDFS environment
+
1. Download and uncompress Hadoop 3.2.1.
```
@@ -354,11 +346,11 @@ For details about how to set up a Hadoop single node cluster, see [here](https:/
-### Step 2: Install the filesystem offloader
+#### Step 2: Install the filesystem offloader
For details, see [installation](#installation).
-### Step 3: Configure the filesystem offloader
+#### Step 3: Configure the filesystem offloader
As indicated in the [configuration](#configuration) section, you need to configure some properties for the filesystem offloader driver before using it. This tutorial assumes that you have configured the filesystem offloader driver as below and run Pulsar in **standalone** mode.
@@ -385,86 +377,7 @@ managedLedgerMaxEntriesPerLedger=100
```
-</TabItem>
-<TabItem value="NFS">
-
-:::note
-
-In this section, it is assumed that you have enabled NFS service and set the shared path of your NFS service. In this section, `/Users/test` is used as the shared path of NFS service.
-
-:::
-
-To offload data to NFS, follow these steps.
-
-### Step 1: Install the filesystem offloader
-
-For details, see [installation](#installation).
-
-### Step 2: Mont your NFS to your local filesystem
-
-This example mounts mounts */Users/pulsar_nfs* to */Users/test*.
-
-```
-
-mount -e 192.168.0.103:/Users/test/Users/pulsar_nfs
-
-```
-
-### Step 3: Configure the filesystem offloader driver
-
-As indicated in the [configuration](#configuration) section, you need to configure some properties for the filesystem offloader driver before using it. This tutorial assumes that you have configured the filesystem offloader driver as below and run Pulsar in **standalone** mode.
-
-1. Set the following configurations in the `conf/standalone.conf` file.
-
- ```conf
-
- managedLedgerOffloadDriver=filesystem
- fileSystemProfilePath=../conf/filesystem_offload_core_site.xml
-
- ```
-
-2. Modify the *filesystem_offload_core_site.xml* as follows.
-
- ```
-
- <property>
- <name>fs.defaultFS</name>
- <value>file:///</value>
- </property>
-
- <property>
- <name>hadoop.tmp.dir</name>
- <value>file:///Users/pulsar_nfs</value>
- </property>
-
- <property>
- <name>io.file.buffer.size</name>
- <value>4096</value>
- </property>
-
- <property>
- <name>io.seqfile.compress.blocksize</name>
- <value>1000000</value>
- </property>
- <property>
-
- <name>io.seqfile.compression.type</name>
- <value>BLOCK</value>
- </property>
-
- <property>
- <name>io.map.index.interval</name>
- <value>128</value>
- </property>
-
- ```
-
-</TabItem>
-
-</Tabs>
-````
-
-### Step 4: Offload data from BookKeeper to filesystem
+#### Step 4: Offload data from BookKeeper to filesystem
Execute the following commands in the repository where you download Pulsar tarball. For example, `~/path/to/apache-pulsar-2.5.1`.
@@ -476,7 +389,7 @@ Execute the following commands in the repository where you download Pulsar tarba
```
-2. To ensure the data generated is not deleted immediately, it is recommended to set the [retention policy](https://pulsar.apache.org/docs/en/next/cookbooks-retention-expiry/#retention-policies), which can be either a **size** limit or a **time** limit. The larger value you set for the retention policy, the longer the data can be retained.
+2. To ensure the data generated is not deleted immediately, it is recommended to set the [retention policy](cookbooks-retention-expiry.md/#retention-policies), which can be either a **size** limit or a **time** limit. The larger value you set for the retention policy, the longer the data can be retained.
```
@@ -527,7 +440,7 @@ Execute the following commands in the repository where you download Pulsar tarba
```
-5. Wait a second and send more messages to the topic.
+5. Wait for a second and send more messages to the topic.
```
@@ -628,6 +541,83 @@ Execute the following commands in the repository where you download Pulsar tarba
+
+### Offload data to NFS
+
+:::note
+
+In this section, it is assumed that you have enabled NFS service and set the shared path of your NFS service. In this section, `/Users/test` is used as the shared path of NFS service.
+
+:::
+
+#### Step 1: Install the filesystem offloader
+
+For details, see [installation](#installation).
+
+#### Step 2: Mount your NFS to your local filesystem
+
+This example mounts */Users/pulsar_nfs* to */Users/test*.
+
+```
+
+mount -e 192.168.0.103:/Users/test/Users/pulsar_nfs
+
+```
+
+#### Step 3: Configure the filesystem offloader driver
+
+As indicated in the [configuration](#configuration) section, you need to configure some properties for the filesystem offloader driver before using it. This tutorial assumes that you have configured the filesystem offloader driver as below and run Pulsar in **standalone** mode.
+
+1. Set the following configurations in the `conf/standalone.conf` file.
+
+ ```conf
+
+ managedLedgerOffloadDriver=filesystem
+ fileSystemProfilePath=../conf/filesystem_offload_core_site.xml
+
+ ```
+
+2. Modify the *filesystem_offload_core_site.xml* as follows.
+
+ ```
+
+ <property>
+ <name>fs.defaultFS</name>
+ <value>file:///</value>
+ </property>
+
+ <property>
+ <name>hadoop.tmp.dir</name>
+ <value>file:///Users/pulsar_nfs</value>
+ </property>
+
+ <property>
+ <name>io.file.buffer.size</name>
+ <value>4096</value>
+ </property>
+
+ <property>
+ <name>io.seqfile.compress.blocksize</name>
+ <value>1000000</value>
+ </property>
+ <property>
+
+ <name>io.seqfile.compression.type</name>
+ <value>BLOCK</value>
+ </property>
+
+ <property>
+ <name>io.map.index.interval</name>
+ <value>128</value>
+ </property>
+
+ ```
+
+#### Step 4: Offload data from BookKeeper to filesystem
+
+Refer to the step 4 of [Offload data to HDFS](#step-4-offload-data-from-bookkeeper-to-filesystem).
+
+
## Read offloaded data from filesystem
* The offloaded data is stored as `MapFile` in the following new path of the filesystem: