You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by ti...@apache.org on 2023/10/15 17:30:05 UTC
[pulsar-site] branch main updated: Fix tiered-storage-filesystem NFS TabItem (#716)
This is an automated email from the ASF dual-hosted git repository.
tison pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git
The following commit(s) were added to refs/heads/main by this push:
new 78198564b9e6 Fix tiered-storage-filesystem NFS TabItem (#716)
78198564b9e6 is described below
commit 78198564b9e65373ce4cd6ddb34d39adf25ab918
Author: yangguodong <11...@qq.com>
AuthorDate: Sun Oct 15 12:29:59 2023 -0500
Fix tiered-storage-filesystem NFS TabItem (#716)
Signed-off-by: tison <wa...@gmail.com>
Co-authored-by: yangguodong.cn <ya...@bytedance.com>
Co-authored-by: tison <wa...@gmail.com>
---
docs/tiered-storage-filesystem.md | 52 ++++++++++-----------
docs/tiered-storage-s3.md | 38 ++++++++--------
.../version-2.11.x/tiered-storage-filesystem.md | 53 +++++++++++-----------
versioned_docs/version-2.11.x/tiered-storage-s3.md | 35 +++++++-------
.../version-3.0.x/tiered-storage-filesystem.md | 52 ++++++++++-----------
versioned_docs/version-3.0.x/tiered-storage-s3.md | 36 +++++++--------
.../version-3.1.x/tiered-storage-filesystem.md | 52 ++++++++++-----------
versioned_docs/version-3.1.x/tiered-storage-s3.md | 36 +++++++--------
8 files changed, 174 insertions(+), 180 deletions(-)
diff --git a/docs/tiered-storage-filesystem.md b/docs/tiered-storage-filesystem.md
index ec3179686800..d11a8a558688 100644
--- a/docs/tiered-storage-filesystem.md
+++ b/docs/tiered-storage-filesystem.md
@@ -9,7 +9,6 @@ import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
````
-
This chapter guides you through every step of installing and configuring the filesystem offloader and using it with Pulsar.
## Installation
@@ -47,40 +46,41 @@ You can configure the filesystem offloader driver in the `broker.conf` or `stand
- **Required** configurations are as below.
- Parameter | Description | Example value
- |---|---|---
- `managedLedgerOffloadDriver` | Offloader driver name, which is case-insensitive. | filesystem
- `fileSystemURI` | Connection address, which is the URI to access the default Hadoop distributed file system. | hdfs://127.0.0.1:9000
- `offloadersDirectory` | Offloader directory | offloaders
- `fileSystemProfilePath` | Hadoop profile path. The configuration file is stored in the Hadoop profile path. It contains various settings for Hadoop performance tuning. | conf/filesystem_offload_core_site.xml
+ | Parameter | Description | Example value |
+ |------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------|
+ | `managedLedgerOffloadDriver` | Offloader driver name, which is case-insensitive. | filesystem |
+ | `fileSystemURI` | Connection address, which is the URI to access the default Hadoop distributed file system. | hdfs://127.0.0.1:9000 |
+ | `offloadersDirectory` | Offloader directory | offloaders |
+ | `fileSystemProfilePath` | Hadoop profile path. The configuration file is stored in the Hadoop profile path. It contains various settings for Hadoop performance tuning. | conf/filesystem_offload_core_site.xml |
- **Optional** configurations are as below.
- Parameter| Description | Example value
- |---|---|---
- `managedLedgerMinLedgerRolloverTimeMinutes`|Minimum time between ledger rollover for a topic. <br /><br />**Note**: it is not recommended to set this parameter in the production environment.|10
- `managedLedgerMaxEntriesPerLedger`|Maximum number of entries to append to a ledger before triggering a rollover.<br /><br />**Note**: it is not recommended to set this parameter in the production environment.|50000
+ | Parameter | Description | Example value |
+ |---------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+ | `managedLedgerMinLedgerRolloverTimeMinutes` | Minimum time between ledger rollover for a topic. <br /><br />**Note**: it is not recommended to set this parameter in the production environment. | 10 |
+ | `managedLedgerMaxEntriesPerLedger` | Maximum number of entries to append to a ledger before triggering a rollover.<br /><br />**Note**: it is not recommended to set this parameter in the production environment. | 50000 |
+
</TabItem>
<TabItem value="NFS">
- **Required** configurations are as below.
- Parameter | Description | Example value
- |---|---|---
- `managedLedgerOffloadDriver` | Offloader driver name, which is case-insensitive. | filesystem
- `offloadersDirectory` | Offloader directory | offloaders
- `fileSystemProfilePath` | NFS profile path. The configuration file is stored in the NFS profile path. It contains various settings for performance tuning. | conf/filesystem_offload_core_site.xml
+
+ | Parameter | Description | Example value |
+ |------------------------------|----------------------------------------------------------------------------------------------------------------------------------|---------------------------------------|
+ | `managedLedgerOffloadDriver` | Offloader driver name, which is case-insensitive. | filesystem |
+ | `offloadersDirectory` | Offloader directory | offloaders |
+ | `fileSystemProfilePath` | NFS profile path. The configuration file is stored in the NFS profile path. It contains various settings for performance tuning. | conf/filesystem_offload_core_site.xml |
- **Optional** configurations are as below.
- Parameter| Description | Example value
- |---|---|---
- `managedLedgerMinLedgerRolloverTimeMinutes`|Minimum time between ledger rollover for a topic. <br /><br />**Note**: it is not recommended to set this parameter in the production environment.|10
- `managedLedgerMaxEntriesPerLedger`|Maximum number of entries to append to a ledger before triggering a rollover.<br /><br />**Note**: it is not recommended to set this parameter in the production environment.|50000
+ | Parameter | Description | Example value |
+ |---------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+ | `managedLedgerMinLedgerRolloverTimeMinutes` | Minimum time between ledger rollover for a topic. <br /><br />**Note**: it is not recommended to set this parameter in the production environment. | 10 |
+ | `managedLedgerMaxEntriesPerLedger` | Maximum number of entries to append to a ledger before triggering a rollover.<br /><br />**Note**: it is not recommended to set this parameter in the production environment. | 50000 |
</TabItem>
-
</Tabs>
````
@@ -88,11 +88,11 @@ You can configure the filesystem offloader driver in the `broker.conf` or `stand
You can configure the namespace policy to offload data automatically once a threshold is reached. The threshold is based on the size of data that a topic has stored on a Pulsar cluster. Once the topic storage reaches the threshold, an offload operation is triggered automatically.
-Threshold value|Action
-|---|---
-| > 0 | It triggers the offloading operation if the topic storage reaches its threshold.
-= 0|It causes a broker to offload data as soon as possible.
-< 0 |It disables automatic offloading operation.
+| Threshold value | Action |
+|-----------------|----------------------------------------------------------------------------------|
+| > 0 | It triggers the offloading operation if the topic storage reaches its threshold. |
+| = 0 | It causes a broker to offload data as soon as possible. |
+| < 0 | It disables automatic offloading operation. |
Automatic offload runs when a new segment is added to a topic log. If you set the threshold on a namespace, but few messages are being produced to the topic, the filesystem offloader does not work until the current segment is full.
diff --git a/docs/tiered-storage-s3.md b/docs/tiered-storage-s3.md
index cc47cf570114..e6b343bca174 100644
--- a/docs/tiered-storage-s3.md
+++ b/docs/tiered-storage-s3.md
@@ -19,7 +19,7 @@ This chapter guides you through every step of installing and configuring the S3
Follow the steps below to install the S3 offloader.
-1. [Download the Pulsar tarball](getting-started-standalone.md#download-pulsar-distribution).
+1. [Download the Pulsar tarball](getting-started-standalone.md#step-1-download-pulsar-distribution).
2. Download and untar the Pulsar offloaders package, then copy the Pulsar offloaders as `offloaders` in the Pulsar directory. See [Install tiered storage offloaders](tiered-storage-overview.md#how-to-install-tiered-storage-offloaders).
## Configuration
@@ -30,28 +30,27 @@ Before offloading data from BookKeeper to S3-compatible storage, you need to con
:::
-
### Configure S3 offloader driver
You can configure the S3 offloader driver in the configuration file `broker.conf` or `standalone.conf`.
- **Required** configurations are as below.
- | Required configuration | Description | Example value |
- | --- | --- |--- |
- | `managedLedgerOffloadDriver` | Offloader driver name, which is case-insensitive. | S3 |
- | `offloadersDirectory` | Offloader directory | offloaders |
- | `managedLedgerOffloadBucket` | [Bucket](#bucket-required) | pulsar-topic-offload |
- | `managedLedgerOffloadServiceEndpoint` | [Endpoint](#endpoint-required) | http://localhost:9000 |
+ | Required configuration | Description | Example value |
+ |---------------------------------------|---------------------------------------------------|-----------------------|
+ | `managedLedgerOffloadDriver` | Offloader driver name, which is case-insensitive. | S3 |
+ | `offloadersDirectory` | Offloader directory | offloaders |
+ | `managedLedgerOffloadBucket` | [Bucket](#bucket-required) | pulsar-topic-offload |
+ | `managedLedgerOffloadServiceEndpoint` | [Endpoint](#endpoint-required) | http://localhost:9000 |
- **Optional** configurations are as below.
- | Optional | Description | Default value |
- | --- | --- | --- |
- | `managedLedgerOffloadReadBufferSizeInBytes` | Block size for each individual read when reading back data from S3-compatible storage. | 1 MB |
- | `managedLedgerOffloadMaxBlockSizeInBytes` | Maximum block size sent during a multi-part upload to S3-compatible storage. It **cannot** be smaller than 5 MB. | 64 MB |
- | `managedLedgerMinLedgerRolloverTimeMinutes` | Minimum time between ledger rollover for a topic.<br /><br />**Note**: It's **not** recommended to change the default value in a production environment. | 10 |
- | `managedLedgerMaxEntriesPerLedger` | Maximum number of entries to append to a ledger before triggering a rollover.<br /><br />**Note**: It's **not** recommended to change the default value in a production environment. | 50000 |
+ | Optional | Description | Default value |
+ |---------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+ | `managedLedgerOffloadReadBufferSizeInBytes` | Block size for each individual read when reading back data from S3-compatible storage. | 1 MB |
+ | `managedLedgerOffloadMaxBlockSizeInBytes` | Maximum block size sent during a multi-part upload to S3-compatible storage. It **cannot** be smaller than 5 MB. | 64 MB |
+ | `managedLedgerMinLedgerRolloverTimeMinutes` | Minimum time between ledger rollover for a topic.<br /><br />**Note**: It's **not** recommended to change the default value in a production environment. | 10 |
+ | `managedLedgerMaxEntriesPerLedger` | Maximum number of entries to append to a ledger before triggering a rollover.<br /><br />**Note**: It's **not** recommended to change the default value in a production environment. | 50000 |
#### Bucket (required)
@@ -69,7 +68,6 @@ managedLedgerOffloadBucket=pulsar-topic-offload
The endpoint is the region where a bucket is located.
-
##### Example
This example sets the endpoint as `localhost`.
@@ -99,11 +97,11 @@ Exporting these environment variables makes them available in the environment of
Namespace policy can be configured to offload data automatically once a threshold is reached. The threshold is based on the size of data that a topic has stored in a Pulsar cluster. Once the topic reaches the threshold, an offloading operation is triggered automatically.
-| Threshold value | Action |
-| --- | --- |
-| > 0 | It triggers the offloading operation if the topic storage reaches its threshold. |
-| = 0 | It causes a broker to offload data as soon as possible. |
-| < 0 | It disables automatic offloading operation. |
+| Threshold value | Action |
+|-----------------|----------------------------------------------------------------------------------|
+| > 0 | It triggers the offloading operation if the topic storage reaches its threshold. |
+| = 0 | It causes a broker to offload data as soon as possible. |
+| < 0 | It disables automatic offloading operation. |
Automatic offloading runs when a new segment is added to a topic log. If you set the threshold for a namespace, but few messages are being produced to the topic, the offloader does not work until the current segment is full.
diff --git a/versioned_docs/version-2.11.x/tiered-storage-filesystem.md b/versioned_docs/version-2.11.x/tiered-storage-filesystem.md
index ec3179686800..ae9e43662415 100644
--- a/versioned_docs/version-2.11.x/tiered-storage-filesystem.md
+++ b/versioned_docs/version-2.11.x/tiered-storage-filesystem.md
@@ -9,7 +9,6 @@ import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
````
-
This chapter guides you through every step of installing and configuring the filesystem offloader and using it with Pulsar.
## Installation
@@ -47,52 +46,54 @@ You can configure the filesystem offloader driver in the `broker.conf` or `stand
- **Required** configurations are as below.
- Parameter | Description | Example value
- |---|---|---
- `managedLedgerOffloadDriver` | Offloader driver name, which is case-insensitive. | filesystem
- `fileSystemURI` | Connection address, which is the URI to access the default Hadoop distributed file system. | hdfs://127.0.0.1:9000
- `offloadersDirectory` | Offloader directory | offloaders
- `fileSystemProfilePath` | Hadoop profile path. The configuration file is stored in the Hadoop profile path. It contains various settings for Hadoop performance tuning. | conf/filesystem_offload_core_site.xml
+ | Parameter | Description | Example value |
+ |------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------|
+ | `managedLedgerOffloadDriver` | Offloader driver name, which is case-insensitive. | filesystem |
+ | `fileSystemURI` | Connection address, which is the URI to access the default Hadoop distributed file system. | hdfs://127.0.0.1:9000 |
+ | `offloadersDirectory` | Offloader directory | offloaders |
+ | `fileSystemProfilePath` | Hadoop profile path. The configuration file is stored in the Hadoop profile path. It contains various settings for Hadoop performance tuning. | conf/filesystem_offload_core_site.xml |
- **Optional** configurations are as below.
- Parameter| Description | Example value
- |---|---|---
- `managedLedgerMinLedgerRolloverTimeMinutes`|Minimum time between ledger rollover for a topic. <br /><br />**Note**: it is not recommended to set this parameter in the production environment.|10
- `managedLedgerMaxEntriesPerLedger`|Maximum number of entries to append to a ledger before triggering a rollover.<br /><br />**Note**: it is not recommended to set this parameter in the production environment.|50000
+ | Parameter | Description | Example value |
+ |---------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+ | `managedLedgerMinLedgerRolloverTimeMinutes` | Minimum time between ledger rollover for a topic. <br /><br />**Note**: it is not recommended to set this parameter in the production environment. | 10 |
+ | `managedLedgerMaxEntriesPerLedger` | Maximum number of entries to append to a ledger before triggering a rollover.<br /><br />**Note**: it is not recommended to set this parameter in the production environment. | 50000 |
+
</TabItem>
<TabItem value="NFS">
- **Required** configurations are as below.
- Parameter | Description | Example value
- |---|---|---
- `managedLedgerOffloadDriver` | Offloader driver name, which is case-insensitive. | filesystem
- `offloadersDirectory` | Offloader directory | offloaders
- `fileSystemProfilePath` | NFS profile path. The configuration file is stored in the NFS profile path. It contains various settings for performance tuning. | conf/filesystem_offload_core_site.xml
+
+ | Parameter | Description | Example value |
+ |------------------------------|----------------------------------------------------------------------------------------------------------------------------------|---------------------------------------|
+ | `managedLedgerOffloadDriver` | Offloader driver name, which is case-insensitive. | filesystem |
+ | `offloadersDirectory` | Offloader directory | offloaders |
+ | `fileSystemProfilePath` | NFS profile path. The configuration file is stored in the NFS profile path. It contains various settings for performance tuning. | conf/filesystem_offload_core_site.xml |
- **Optional** configurations are as below.
- Parameter| Description | Example value
- |---|---|---
- `managedLedgerMinLedgerRolloverTimeMinutes`|Minimum time between ledger rollover for a topic. <br /><br />**Note**: it is not recommended to set this parameter in the production environment.|10
- `managedLedgerMaxEntriesPerLedger`|Maximum number of entries to append to a ledger before triggering a rollover.<br /><br />**Note**: it is not recommended to set this parameter in the production environment.|50000
+ | Parameter | Description | Example value |
+ |---------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+ | `managedLedgerMinLedgerRolloverTimeMinutes` | Minimum time between ledger rollover for a topic. <br /><br />**Note**: it is not recommended to set this parameter in the production environment. | 10 |
+ | `managedLedgerMaxEntriesPerLedger` | Maximum number of entries to append to a ledger before triggering a rollover.<br /><br />**Note**: it is not recommended to set this parameter in the production environment. | 50000 |
</TabItem>
-
</Tabs>
````
+
### Run filesystem offloader automatically
You can configure the namespace policy to offload data automatically once a threshold is reached. The threshold is based on the size of data that a topic has stored on a Pulsar cluster. Once the topic storage reaches the threshold, an offload operation is triggered automatically.
-Threshold value|Action
-|---|---
-| > 0 | It triggers the offloading operation if the topic storage reaches its threshold.
-= 0|It causes a broker to offload data as soon as possible.
-< 0 |It disables automatic offloading operation.
+| Threshold value | Action |
+|-----------------|----------------------------------------------------------------------------------|
+| > 0 | It triggers the offloading operation if the topic storage reaches its threshold. |
+| = 0 | It causes a broker to offload data as soon as possible. |
+| < 0 | It disables automatic offloading operation. |
Automatic offload runs when a new segment is added to a topic log. If you set the threshold on a namespace, but few messages are being produced to the topic, the filesystem offloader does not work until the current segment is full.
diff --git a/versioned_docs/version-2.11.x/tiered-storage-s3.md b/versioned_docs/version-2.11.x/tiered-storage-s3.md
index 198813cc1c4a..6f7ce8a7e342 100644
--- a/versioned_docs/version-2.11.x/tiered-storage-s3.md
+++ b/versioned_docs/version-2.11.x/tiered-storage-s3.md
@@ -36,21 +36,21 @@ You can configure the S3 offloader driver in the configuration file `broker.conf
- **Required** configurations are as below.
- | Required configuration | Description | Example value |
- | --- | --- |--- |
- | `managedLedgerOffloadDriver` | Offloader driver name, which is case-insensitive. | S3 |
- | `offloadersDirectory` | Offloader directory | offloaders |
- | `managedLedgerOffloadBucket` | [Bucket](#bucket-required) | pulsar-topic-offload |
- | `managedLedgerOffloadServiceEndpoint` | [Endpoint](#endpoint-required) | http://localhost:9000 |
+ | Required configuration | Description | Example value |
+ |---------------------------------------|---------------------------------------------------|-----------------------|
+ | `managedLedgerOffloadDriver` | Offloader driver name, which is case-insensitive. | S3 |
+ | `offloadersDirectory` | Offloader directory | offloaders |
+ | `managedLedgerOffloadBucket` | [Bucket](#bucket-required) | pulsar-topic-offload |
+ | `managedLedgerOffloadServiceEndpoint` | [Endpoint](#endpoint-required) | http://localhost:9000 |
- **Optional** configurations are as below.
- | Optional | Description | Default value |
- | --- | --- | --- |
- | `managedLedgerOffloadReadBufferSizeInBytes` | Block size for each individual read when reading back data from S3-compatible storage. | 1 MB |
- | `managedLedgerOffloadMaxBlockSizeInBytes` | Maximum block size sent during a multi-part upload to S3-compatible storage. It **cannot** be smaller than 5 MB. | 64 MB |
- | `managedLedgerMinLedgerRolloverTimeMinutes` | Minimum time between ledger rollover for a topic.<br /><br />**Note**: It's **not** recommended to change the default value in a production environment. | 10 |
- | `managedLedgerMaxEntriesPerLedger` | Maximum number of entries to append to a ledger before triggering a rollover.<br /><br />**Note**: It's **not** recommended to change the default value in a production environment. | 50000 |
+ | Optional | Description | Default value |
+ |---------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+ | `managedLedgerOffloadReadBufferSizeInBytes` | Block size for each individual read when reading back data from S3-compatible storage. | 1 MB |
+ | `managedLedgerOffloadMaxBlockSizeInBytes` | Maximum block size sent during a multi-part upload to S3-compatible storage. It **cannot** be smaller than 5 MB. | 64 MB |
+ | `managedLedgerMinLedgerRolloverTimeMinutes` | Minimum time between ledger rollover for a topic.<br /><br />**Note**: It's **not** recommended to change the default value in a production environment. | 10 |
+ | `managedLedgerMaxEntriesPerLedger` | Maximum number of entries to append to a ledger before triggering a rollover.<br /><br />**Note**: It's **not** recommended to change the default value in a production environment. | 50000 |
#### Bucket (required)
@@ -68,7 +68,6 @@ managedLedgerOffloadBucket=pulsar-topic-offload
The endpoint is the region where a bucket is located.
-
##### Example
This example sets the endpoint as `localhost`.
@@ -98,11 +97,11 @@ Exporting these environment variables makes them available in the environment of
Namespace policy can be configured to offload data automatically once a threshold is reached. The threshold is based on the size of data that a topic has stored in a Pulsar cluster. Once the topic reaches the threshold, an offloading operation is triggered automatically.
-| Threshold value | Action |
-| --- | --- |
-| > 0 | It triggers the offloading operation if the topic storage reaches its threshold. |
-| = 0 | It causes a broker to offload data as soon as possible. |
-| < 0 | It disables automatic offloading operation. |
+| Threshold value | Action |
+|-----------------|----------------------------------------------------------------------------------|
+| > 0 | It triggers the offloading operation if the topic storage reaches its threshold. |
+| = 0 | It causes a broker to offload data as soon as possible. |
+| < 0 | It disables automatic offloading operation. |
Automatic offloading runs when a new segment is added to a topic log. If you set the threshold for a namespace, but few messages are being produced to the topic, the offloader does not work until the current segment is full.
diff --git a/versioned_docs/version-3.0.x/tiered-storage-filesystem.md b/versioned_docs/version-3.0.x/tiered-storage-filesystem.md
index ec3179686800..d11a8a558688 100644
--- a/versioned_docs/version-3.0.x/tiered-storage-filesystem.md
+++ b/versioned_docs/version-3.0.x/tiered-storage-filesystem.md
@@ -9,7 +9,6 @@ import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
````
-
This chapter guides you through every step of installing and configuring the filesystem offloader and using it with Pulsar.
## Installation
@@ -47,40 +46,41 @@ You can configure the filesystem offloader driver in the `broker.conf` or `stand
- **Required** configurations are as below.
- Parameter | Description | Example value
- |---|---|---
- `managedLedgerOffloadDriver` | Offloader driver name, which is case-insensitive. | filesystem
- `fileSystemURI` | Connection address, which is the URI to access the default Hadoop distributed file system. | hdfs://127.0.0.1:9000
- `offloadersDirectory` | Offloader directory | offloaders
- `fileSystemProfilePath` | Hadoop profile path. The configuration file is stored in the Hadoop profile path. It contains various settings for Hadoop performance tuning. | conf/filesystem_offload_core_site.xml
+ | Parameter | Description | Example value |
+ |------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------|
+ | `managedLedgerOffloadDriver` | Offloader driver name, which is case-insensitive. | filesystem |
+ | `fileSystemURI` | Connection address, which is the URI to access the default Hadoop distributed file system. | hdfs://127.0.0.1:9000 |
+ | `offloadersDirectory` | Offloader directory | offloaders |
+ | `fileSystemProfilePath` | Hadoop profile path. The configuration file is stored in the Hadoop profile path. It contains various settings for Hadoop performance tuning. | conf/filesystem_offload_core_site.xml |
- **Optional** configurations are as below.
- Parameter| Description | Example value
- |---|---|---
- `managedLedgerMinLedgerRolloverTimeMinutes`|Minimum time between ledger rollover for a topic. <br /><br />**Note**: it is not recommended to set this parameter in the production environment.|10
- `managedLedgerMaxEntriesPerLedger`|Maximum number of entries to append to a ledger before triggering a rollover.<br /><br />**Note**: it is not recommended to set this parameter in the production environment.|50000
+ | Parameter | Description | Example value |
+ |---------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+ | `managedLedgerMinLedgerRolloverTimeMinutes` | Minimum time between ledger rollover for a topic. <br /><br />**Note**: it is not recommended to set this parameter in the production environment. | 10 |
+ | `managedLedgerMaxEntriesPerLedger` | Maximum number of entries to append to a ledger before triggering a rollover.<br /><br />**Note**: it is not recommended to set this parameter in the production environment. | 50000 |
+
</TabItem>
<TabItem value="NFS">
- **Required** configurations are as below.
- Parameter | Description | Example value
- |---|---|---
- `managedLedgerOffloadDriver` | Offloader driver name, which is case-insensitive. | filesystem
- `offloadersDirectory` | Offloader directory | offloaders
- `fileSystemProfilePath` | NFS profile path. The configuration file is stored in the NFS profile path. It contains various settings for performance tuning. | conf/filesystem_offload_core_site.xml
+
+ | Parameter | Description | Example value |
+ |------------------------------|----------------------------------------------------------------------------------------------------------------------------------|---------------------------------------|
+ | `managedLedgerOffloadDriver` | Offloader driver name, which is case-insensitive. | filesystem |
+ | `offloadersDirectory` | Offloader directory | offloaders |
+ | `fileSystemProfilePath` | NFS profile path. The configuration file is stored in the NFS profile path. It contains various settings for performance tuning. | conf/filesystem_offload_core_site.xml |
- **Optional** configurations are as below.
- Parameter| Description | Example value
- |---|---|---
- `managedLedgerMinLedgerRolloverTimeMinutes`|Minimum time between ledger rollover for a topic. <br /><br />**Note**: it is not recommended to set this parameter in the production environment.|10
- `managedLedgerMaxEntriesPerLedger`|Maximum number of entries to append to a ledger before triggering a rollover.<br /><br />**Note**: it is not recommended to set this parameter in the production environment.|50000
+ | Parameter | Description | Example value |
+ |---------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+ | `managedLedgerMinLedgerRolloverTimeMinutes` | Minimum time between ledger rollover for a topic. <br /><br />**Note**: it is not recommended to set this parameter in the production environment. | 10 |
+ | `managedLedgerMaxEntriesPerLedger` | Maximum number of entries to append to a ledger before triggering a rollover.<br /><br />**Note**: it is not recommended to set this parameter in the production environment. | 50000 |
</TabItem>
-
</Tabs>
````
@@ -88,11 +88,11 @@ You can configure the filesystem offloader driver in the `broker.conf` or `stand
You can configure the namespace policy to offload data automatically once a threshold is reached. The threshold is based on the size of data that a topic has stored on a Pulsar cluster. Once the topic storage reaches the threshold, an offload operation is triggered automatically.
-Threshold value|Action
-|---|---
-| > 0 | It triggers the offloading operation if the topic storage reaches its threshold.
-= 0|It causes a broker to offload data as soon as possible.
-< 0 |It disables automatic offloading operation.
+| Threshold value | Action |
+|-----------------|----------------------------------------------------------------------------------|
+| > 0 | It triggers the offloading operation if the topic storage reaches its threshold. |
+| = 0 | It causes a broker to offload data as soon as possible. |
+| < 0 | It disables automatic offloading operation. |
Automatic offload runs when a new segment is added to a topic log. If you set the threshold on a namespace, but few messages are being produced to the topic, the filesystem offloader does not work until the current segment is full.
diff --git a/versioned_docs/version-3.0.x/tiered-storage-s3.md b/versioned_docs/version-3.0.x/tiered-storage-s3.md
index 198813cc1c4a..7126f2e4aa69 100644
--- a/versioned_docs/version-3.0.x/tiered-storage-s3.md
+++ b/versioned_docs/version-3.0.x/tiered-storage-s3.md
@@ -29,28 +29,27 @@ Before offloading data from BookKeeper to S3-compatible storage, you need to con
:::
-
### Configure S3 offloader driver
You can configure the S3 offloader driver in the configuration file `broker.conf` or `standalone.conf`.
- **Required** configurations are as below.
- | Required configuration | Description | Example value |
- | --- | --- |--- |
- | `managedLedgerOffloadDriver` | Offloader driver name, which is case-insensitive. | S3 |
- | `offloadersDirectory` | Offloader directory | offloaders |
- | `managedLedgerOffloadBucket` | [Bucket](#bucket-required) | pulsar-topic-offload |
- | `managedLedgerOffloadServiceEndpoint` | [Endpoint](#endpoint-required) | http://localhost:9000 |
+ | Required configuration | Description | Example value |
+ |---------------------------------------|---------------------------------------------------|-----------------------|
+ | `managedLedgerOffloadDriver` | Offloader driver name, which is case-insensitive. | S3 |
+ | `offloadersDirectory` | Offloader directory | offloaders |
+ | `managedLedgerOffloadBucket` | [Bucket](#bucket-required) | pulsar-topic-offload |
+ | `managedLedgerOffloadServiceEndpoint` | [Endpoint](#endpoint-required) | http://localhost:9000 |
- **Optional** configurations are as below.
- | Optional | Description | Default value |
- | --- | --- | --- |
- | `managedLedgerOffloadReadBufferSizeInBytes` | Block size for each individual read when reading back data from S3-compatible storage. | 1 MB |
- | `managedLedgerOffloadMaxBlockSizeInBytes` | Maximum block size sent during a multi-part upload to S3-compatible storage. It **cannot** be smaller than 5 MB. | 64 MB |
- | `managedLedgerMinLedgerRolloverTimeMinutes` | Minimum time between ledger rollover for a topic.<br /><br />**Note**: It's **not** recommended to change the default value in a production environment. | 10 |
- | `managedLedgerMaxEntriesPerLedger` | Maximum number of entries to append to a ledger before triggering a rollover.<br /><br />**Note**: It's **not** recommended to change the default value in a production environment. | 50000 |
+ | Optional | Description | Default value |
+ |---------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+ | `managedLedgerOffloadReadBufferSizeInBytes` | Block size for each individual read when reading back data from S3-compatible storage. | 1 MB |
+ | `managedLedgerOffloadMaxBlockSizeInBytes` | Maximum block size sent during a multi-part upload to S3-compatible storage. It **cannot** be smaller than 5 MB. | 64 MB |
+ | `managedLedgerMinLedgerRolloverTimeMinutes` | Minimum time between ledger rollover for a topic.<br /><br />**Note**: It's **not** recommended to change the default value in a production environment. | 10 |
+ | `managedLedgerMaxEntriesPerLedger` | Maximum number of entries to append to a ledger before triggering a rollover.<br /><br />**Note**: It's **not** recommended to change the default value in a production environment. | 50000 |
#### Bucket (required)
@@ -68,7 +67,6 @@ managedLedgerOffloadBucket=pulsar-topic-offload
The endpoint is the region where a bucket is located.
-
##### Example
This example sets the endpoint as `localhost`.
@@ -98,11 +96,11 @@ Exporting these environment variables makes them available in the environment of
Namespace policy can be configured to offload data automatically once a threshold is reached. The threshold is based on the size of data that a topic has stored in a Pulsar cluster. Once the topic reaches the threshold, an offloading operation is triggered automatically.
-| Threshold value | Action |
-| --- | --- |
-| > 0 | It triggers the offloading operation if the topic storage reaches its threshold. |
-| = 0 | It causes a broker to offload data as soon as possible. |
-| < 0 | It disables automatic offloading operation. |
+| Threshold value | Action |
+|-----------------|----------------------------------------------------------------------------------|
+| > 0 | It triggers the offloading operation if the topic storage reaches its threshold. |
+| = 0 | It causes a broker to offload data as soon as possible. |
+| < 0 | It disables automatic offloading operation. |
Automatic offloading runs when a new segment is added to a topic log. If you set the threshold for a namespace, but few messages are being produced to the topic, the offloader does not work until the current segment is full.
diff --git a/versioned_docs/version-3.1.x/tiered-storage-filesystem.md b/versioned_docs/version-3.1.x/tiered-storage-filesystem.md
index ec3179686800..d11a8a558688 100644
--- a/versioned_docs/version-3.1.x/tiered-storage-filesystem.md
+++ b/versioned_docs/version-3.1.x/tiered-storage-filesystem.md
@@ -9,7 +9,6 @@ import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
````
-
This chapter guides you through every step of installing and configuring the filesystem offloader and using it with Pulsar.
## Installation
@@ -47,40 +46,41 @@ You can configure the filesystem offloader driver in the `broker.conf` or `stand
- **Required** configurations are as below.
- Parameter | Description | Example value
- |---|---|---
- `managedLedgerOffloadDriver` | Offloader driver name, which is case-insensitive. | filesystem
- `fileSystemURI` | Connection address, which is the URI to access the default Hadoop distributed file system. | hdfs://127.0.0.1:9000
- `offloadersDirectory` | Offloader directory | offloaders
- `fileSystemProfilePath` | Hadoop profile path. The configuration file is stored in the Hadoop profile path. It contains various settings for Hadoop performance tuning. | conf/filesystem_offload_core_site.xml
+ | Parameter | Description | Example value |
+ |------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------|
+ | `managedLedgerOffloadDriver` | Offloader driver name, which is case-insensitive. | filesystem |
+ | `fileSystemURI` | Connection address, which is the URI to access the default Hadoop distributed file system. | hdfs://127.0.0.1:9000 |
+ | `offloadersDirectory` | Offloader directory | offloaders |
+ | `fileSystemProfilePath` | Hadoop profile path. The configuration file is stored in the Hadoop profile path. It contains various settings for Hadoop performance tuning. | conf/filesystem_offload_core_site.xml |
- **Optional** configurations are as below.
- Parameter| Description | Example value
- |---|---|---
- `managedLedgerMinLedgerRolloverTimeMinutes`|Minimum time between ledger rollover for a topic. <br /><br />**Note**: it is not recommended to set this parameter in the production environment.|10
- `managedLedgerMaxEntriesPerLedger`|Maximum number of entries to append to a ledger before triggering a rollover.<br /><br />**Note**: it is not recommended to set this parameter in the production environment.|50000
+ | Parameter | Description | Example value |
+ |---------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+ | `managedLedgerMinLedgerRolloverTimeMinutes` | Minimum time between ledger rollover for a topic. <br /><br />**Note**: it is not recommended to set this parameter in the production environment. | 10 |
+ | `managedLedgerMaxEntriesPerLedger` | Maximum number of entries to append to a ledger before triggering a rollover.<br /><br />**Note**: it is not recommended to set this parameter in the production environment. | 50000 |
+
</TabItem>
<TabItem value="NFS">
- **Required** configurations are as below.
- Parameter | Description | Example value
- |---|---|---
- `managedLedgerOffloadDriver` | Offloader driver name, which is case-insensitive. | filesystem
- `offloadersDirectory` | Offloader directory | offloaders
- `fileSystemProfilePath` | NFS profile path. The configuration file is stored in the NFS profile path. It contains various settings for performance tuning. | conf/filesystem_offload_core_site.xml
+
+ | Parameter | Description | Example value |
+ |------------------------------|----------------------------------------------------------------------------------------------------------------------------------|---------------------------------------|
+ | `managedLedgerOffloadDriver` | Offloader driver name, which is case-insensitive. | filesystem |
+ | `offloadersDirectory` | Offloader directory | offloaders |
+ | `fileSystemProfilePath` | NFS profile path. The configuration file is stored in the NFS profile path. It contains various settings for performance tuning. | conf/filesystem_offload_core_site.xml |
- **Optional** configurations are as below.
- Parameter| Description | Example value
- |---|---|---
- `managedLedgerMinLedgerRolloverTimeMinutes`|Minimum time between ledger rollover for a topic. <br /><br />**Note**: it is not recommended to set this parameter in the production environment.|10
- `managedLedgerMaxEntriesPerLedger`|Maximum number of entries to append to a ledger before triggering a rollover.<br /><br />**Note**: it is not recommended to set this parameter in the production environment.|50000
+ | Parameter | Description | Example value |
+ |---------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+ | `managedLedgerMinLedgerRolloverTimeMinutes` | Minimum time between ledger rollover for a topic. <br /><br />**Note**: it is not recommended to set this parameter in the production environment. | 10 |
+ | `managedLedgerMaxEntriesPerLedger` | Maximum number of entries to append to a ledger before triggering a rollover.<br /><br />**Note**: it is not recommended to set this parameter in the production environment. | 50000 |
</TabItem>
-
</Tabs>
````
@@ -88,11 +88,11 @@ You can configure the filesystem offloader driver in the `broker.conf` or `stand
You can configure the namespace policy to offload data automatically once a threshold is reached. The threshold is based on the size of data that a topic has stored on a Pulsar cluster. Once the topic storage reaches the threshold, an offload operation is triggered automatically.
-Threshold value|Action
-|---|---
-| > 0 | It triggers the offloading operation if the topic storage reaches its threshold.
-= 0|It causes a broker to offload data as soon as possible.
-< 0 |It disables automatic offloading operation.
+| Threshold value | Action |
+|-----------------|----------------------------------------------------------------------------------|
+| > 0 | It triggers the offloading operation if the topic storage reaches its threshold. |
+| = 0 | It causes a broker to offload data as soon as possible. |
+| < 0 | It disables automatic offloading operation. |
Automatic offload runs when a new segment is added to a topic log. If you set the threshold on a namespace, but few messages are being produced to the topic, the filesystem offloader does not work until the current segment is full.
diff --git a/versioned_docs/version-3.1.x/tiered-storage-s3.md b/versioned_docs/version-3.1.x/tiered-storage-s3.md
index 198813cc1c4a..7126f2e4aa69 100644
--- a/versioned_docs/version-3.1.x/tiered-storage-s3.md
+++ b/versioned_docs/version-3.1.x/tiered-storage-s3.md
@@ -29,28 +29,27 @@ Before offloading data from BookKeeper to S3-compatible storage, you need to con
:::
-
### Configure S3 offloader driver
You can configure the S3 offloader driver in the configuration file `broker.conf` or `standalone.conf`.
- **Required** configurations are as below.
- | Required configuration | Description | Example value |
- | --- | --- |--- |
- | `managedLedgerOffloadDriver` | Offloader driver name, which is case-insensitive. | S3 |
- | `offloadersDirectory` | Offloader directory | offloaders |
- | `managedLedgerOffloadBucket` | [Bucket](#bucket-required) | pulsar-topic-offload |
- | `managedLedgerOffloadServiceEndpoint` | [Endpoint](#endpoint-required) | http://localhost:9000 |
+ | Required configuration | Description | Example value |
+ |---------------------------------------|---------------------------------------------------|-----------------------|
+ | `managedLedgerOffloadDriver` | Offloader driver name, which is case-insensitive. | S3 |
+ | `offloadersDirectory` | Offloader directory | offloaders |
+ | `managedLedgerOffloadBucket` | [Bucket](#bucket-required) | pulsar-topic-offload |
+ | `managedLedgerOffloadServiceEndpoint` | [Endpoint](#endpoint-required) | http://localhost:9000 |
- **Optional** configurations are as below.
- | Optional | Description | Default value |
- | --- | --- | --- |
- | `managedLedgerOffloadReadBufferSizeInBytes` | Block size for each individual read when reading back data from S3-compatible storage. | 1 MB |
- | `managedLedgerOffloadMaxBlockSizeInBytes` | Maximum block size sent during a multi-part upload to S3-compatible storage. It **cannot** be smaller than 5 MB. | 64 MB |
- | `managedLedgerMinLedgerRolloverTimeMinutes` | Minimum time between ledger rollover for a topic.<br /><br />**Note**: It's **not** recommended to change the default value in a production environment. | 10 |
- | `managedLedgerMaxEntriesPerLedger` | Maximum number of entries to append to a ledger before triggering a rollover.<br /><br />**Note**: It's **not** recommended to change the default value in a production environment. | 50000 |
+ | Optional | Description | Default value |
+ |---------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+ | `managedLedgerOffloadReadBufferSizeInBytes` | Block size for each individual read when reading back data from S3-compatible storage. | 1 MB |
+ | `managedLedgerOffloadMaxBlockSizeInBytes` | Maximum block size sent during a multi-part upload to S3-compatible storage. It **cannot** be smaller than 5 MB. | 64 MB |
+ | `managedLedgerMinLedgerRolloverTimeMinutes` | Minimum time between ledger rollover for a topic.<br /><br />**Note**: It's **not** recommended to change the default value in a production environment. | 10 |
+ | `managedLedgerMaxEntriesPerLedger` | Maximum number of entries to append to a ledger before triggering a rollover.<br /><br />**Note**: It's **not** recommended to change the default value in a production environment. | 50000 |
#### Bucket (required)
@@ -68,7 +67,6 @@ managedLedgerOffloadBucket=pulsar-topic-offload
The endpoint is the region where a bucket is located.
-
##### Example
This example sets the endpoint as `localhost`.
@@ -98,11 +96,11 @@ Exporting these environment variables makes them available in the environment of
Namespace policy can be configured to offload data automatically once a threshold is reached. The threshold is based on the size of data that a topic has stored in a Pulsar cluster. Once the topic reaches the threshold, an offloading operation is triggered automatically.
-| Threshold value | Action |
-| --- | --- |
-| > 0 | It triggers the offloading operation if the topic storage reaches its threshold. |
-| = 0 | It causes a broker to offload data as soon as possible. |
-| < 0 | It disables automatic offloading operation. |
+| Threshold value | Action |
+|-----------------|----------------------------------------------------------------------------------|
+| > 0 | It triggers the offloading operation if the topic storage reaches its threshold. |
+| = 0 | It causes a broker to offload data as soon as possible. |
+| < 0 | It disables automatic offloading operation. |
Automatic offloading runs when a new segment is added to a topic log. If you set the threshold for a namespace, but few messages are being produced to the topic, the offloader does not work until the current segment is full.