You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by li...@apache.org on 2022/05/06 05:50:55 UTC
[pulsar] branch master updated: [feature][doc] Add offload metrics for broker and some minor fixes for tiered storage (#13919)
This is an automated email from the ASF dual-hosted git repository.
liuyu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 88cd81d99df [feature][doc] Add offload metrics for broker and some minor fixes for tiered storage (#13919)
88cd81d99df is described below
commit 88cd81d99dfde7674cbbcfbfccb2914af857114a
Author: momo-jun <60...@users.noreply.github.com>
AuthorDate: Fri May 6 13:50:47 2022 +0800
[feature][doc] Add offload metrics for broker and some minor fixes for tiered storage (#13919)
---
site2/docs/reference-metrics.md | 108 ++++++++++++++++++++--------------
site2/docs/tiered-storage-aliyun.md | 6 +-
site2/docs/tiered-storage-aws.md | 10 ++--
site2/docs/tiered-storage-azure.md | 6 +-
site2/docs/tiered-storage-gcs.md | 7 +--
site2/docs/tiered-storage-overview.md | 8 ++-
6 files changed, 82 insertions(+), 63 deletions(-)
diff --git a/site2/docs/reference-metrics.md b/site2/docs/reference-metrics.md
index 2589d5ca54c..dfe1898133c 100644
--- a/site2/docs/reference-metrics.md
+++ b/site2/docs/reference-metrics.md
@@ -42,7 +42,7 @@ in the `bookkeeper.conf` configuration file.
| Name | Type | Description |
|---|---|---|
-| bookie_SERVER_STATUS | Gauge | The server status for bookie server. <br><ul><li>1: the bookie is running in writable mode.</li><li>0: the bookie is running in readonly mode.</li></ul> |
+| bookie_SERVER_STATUS | Gauge | The server status for bookie server. <br /><ul><li>1: the bookie is running in writable mode.</li><li>0: the bookie is running in readonly mode.</li></ul> |
| bookkeeper_server_ADD_ENTRY_count | Counter | The total number of ADD_ENTRY requests received at the bookie. The `success` label is used to distinguish successes and failures. |
| bookkeeper_server_READ_ENTRY_count | Counter | The total number of READ_ENTRY requests received at the bookie. The `success` label is used to distinguish successes and failures. |
| bookie_WRITE_BYTES | Counter | The total number of bytes written to the bookie. |
@@ -112,6 +112,7 @@ The following metrics are available for broker:
- [Connection metrics](#connection-metrics)
- [Jetty metrics](#jetty-metrics)
- [Schema metrics](#schema-metrics)
+ - [Offload metrics](#offload-metrics)
- [Pulsar Functions](#pulsar-functions)
- [Proxy](#proxy)
- [Pulsar SQL Worker](#pulsar-sql-worker)
@@ -142,19 +143,19 @@ All the namespace metrics are labelled with the following labels:
| pulsar_subscriptions_count | Gauge | The number of Pulsar subscriptions of the namespace served by this broker. |
| pulsar_producers_count | Gauge | The number of active producers of the namespace connected to this broker. |
| pulsar_consumers_count | Gauge | The number of active consumers of the namespace connected to this broker. |
-| pulsar_rate_in | Gauge | The total message rate of the namespace coming into this broker (messages/second). |
-| pulsar_rate_out | Gauge | The total message rate of the namespace going out from this broker (messages/second). |
-| pulsar_throughput_in | Gauge | The total throughput of the namespace coming into this broker (bytes/second). |
-| pulsar_throughput_out | Gauge | The total throughput of the namespace going out from this broker (bytes/second). |
+| pulsar_rate_in | Gauge | The total message rate of the namespace coming into this broker (message per second). |
+| pulsar_rate_out | Gauge | The total message rate of the namespace going out from this broker (message per second). |
+| pulsar_throughput_in | Gauge | The total throughput of the namespace coming into this broker (byte per second). |
+| pulsar_throughput_out | Gauge | The total throughput of the namespace going out from this broker (byte per second). |
| pulsar_storage_size | Gauge | The total storage size of the topics in this namespace owned by this broker (bytes). |
| pulsar_storage_logical_size | Gauge | The storage size of topics in the namespace owned by the broker without replicas (in bytes). |
| pulsar_storage_backlog_size | Gauge | The total backlog size of the topics of this namespace owned by this broker (messages). |
| pulsar_storage_offloaded_size | Gauge | The total amount of the data in this namespace offloaded to the tiered storage (bytes). |
-| pulsar_storage_write_rate | Gauge | The total message batches (entries) written to the storage for this namespace (message batches / second). |
-| pulsar_storage_read_rate | Gauge | The total message batches (entries) read from the storage for this namespace (message batches / second). |
+| pulsar_storage_write_rate | Gauge | The total message batches (entries) written to the storage for this namespace (message batch per second). |
+| pulsar_storage_read_rate | Gauge | The total message batches (entries) read from the storage for this namespace (message batch per second). |
| pulsar_subscription_delayed | Gauge | The total message batches (entries) are delayed for dispatching. |
-| pulsar_storage_write_latency_le_* | Histogram | The entry rate of a namespace that the storage write latency is smaller with a given threshold.<br> Available thresholds: <br><ul><li>pulsar_storage_write_latency_le_0_5: <= 0.5ms </li><li>pulsar_storage_write_latency_le_1: <= 1ms</li><li>pulsar_storage_write_latency_le_5: <= 5ms</li><li>pulsar_storage_write_latency_le_10: <= 10ms</li><li>pulsar_storage_write_latency_le_20: <= 20ms</li><li>pulsar_storage_write_latency_le_50: <= 50ms</li>< [...]
-| pulsar_entry_size_le_* | Histogram | The entry rate of a namespace that the entry size is smaller with a given threshold.<br> Available thresholds: <br><ul><li>pulsar_entry_size_le_128: <= 128 bytes </li><li>pulsar_entry_size_le_512: <= 512 bytes</li><li>pulsar_entry_size_le_1_kb: <= 1 KB</li><li>pulsar_entry_size_le_2_kb: <= 2 KB</li><li>pulsar_entry_size_le_4_kb: <= 4 KB</li><li>pulsar_entry_size_le_16_kb: <= 16 KB</li><li>pulsar_entry_size_le_100_kb: <= 100 KB</li><li>pulsar_entry_s [...]
+| pulsar_storage_write_latency_le_* | Histogram | The entry rate of a namespace that the storage write latency is smaller with a given threshold.<br /> Available thresholds: <br /><ul><li>pulsar_storage_write_latency_le_0_5: <= 0.5ms </li><li>pulsar_storage_write_latency_le_1: <= 1ms</li><li>pulsar_storage_write_latency_le_5: <= 5ms</li><li>pulsar_storage_write_latency_le_10: <= 10ms</li><li>pulsar_storage_write_latency_le_20: <= 20ms</li><li>pulsar_storage_write_latency_le_50: <= 50ms</ [...]
+| pulsar_entry_size_le_* | Histogram | The entry rate of a namespace that the entry size is smaller with a given threshold.<br /> Available thresholds: <br /><ul><li>pulsar_entry_size_le_128: <= 128 bytes </li><li>pulsar_entry_size_le_512: <= 512 bytes</li><li>pulsar_entry_size_le_1_kb: <= 1 KB</li><li>pulsar_entry_size_le_2_kb: <= 2 KB</li><li>pulsar_entry_size_le_4_kb: <= 4 KB</li><li>pulsar_entry_size_le_16_kb: <= 16 KB</li><li>pulsar_entry_size_le_100_kb: <= 100 KB</li><li>pulsar_ent [...]
#### Replication metrics
@@ -164,12 +165,12 @@ All the replication metrics are also labelled with `remoteCluster=${pulsar_remot
| Name | Type | Description |
|---|---|---|
-| pulsar_replication_rate_in | Gauge | The total message rate of the namespace replicating from remote cluster (messages/second). |
-| pulsar_replication_rate_out | Gauge | The total message rate of the namespace replicating to remote cluster (messages/second). |
-| pulsar_replication_throughput_in | Gauge | The total throughput of the namespace replicating from remote cluster (bytes/second). |
-| pulsar_replication_throughput_out | Gauge | The total throughput of the namespace replicating to remote cluster (bytes/second). |
+| pulsar_replication_rate_in | Gauge | The total message rate of the namespace replicating from remote cluster (message per second). |
+| pulsar_replication_rate_out | Gauge | The total message rate of the namespace replicating to remote cluster (message per second). |
+| pulsar_replication_throughput_in | Gauge | The total throughput of the namespace replicating from remote cluster (byte per second). |
+| pulsar_replication_throughput_out | Gauge | The total throughput of the namespace replicating to remote cluster (byte per second). |
| pulsar_replication_backlog | Gauge | The total backlog of the namespace replicating to remote cluster (messages). |
-| pulsar_replication_rate_expired | Gauge | Total rate of messages expired (messages/second). |
+| pulsar_replication_rate_expired | Gauge | Total rate of messages expired (message per second). |
| pulsar_replication_connected_count | Gauge | The count of replication-subscriber up and running to replicate to remote cluster. |
| pulsar_replication_delay_in_seconds | Gauge | Time in seconds from the time a message was produced to the time when it is about to be replicated. |
@@ -189,21 +190,21 @@ All the topic metrics are labelled with the following labels:
| pulsar_subscriptions_count | Gauge | The number of Pulsar subscriptions of the topic served by this broker. |
| pulsar_producers_count | Gauge | The number of active producers of the topic connected to this broker. |
| pulsar_consumers_count | Gauge | The number of active consumers of the topic connected to this broker. |
-| pulsar_rate_in | Gauge | The total message rate of the topic coming into this broker (messages/second). |
-| pulsar_rate_out | Gauge | The total message rate of the topic going out from this broker (messages/second). |
+| pulsar_rate_in | Gauge | The total message rate of the topic coming into this broker (message per second). |
+| pulsar_rate_out | Gauge | The total message rate of the topic going out from this broker (message per second). |
| pulsar_publish_rate_limit_times | Gauge | The number of times the publish rate limit is triggered. |
-| pulsar_throughput_in | Gauge | The total throughput of the topic coming into this broker (bytes/second). |
-| pulsar_throughput_out | Gauge | The total throughput of the topic going out from this broker (bytes/second). |
+| pulsar_throughput_in | Gauge | The total throughput of the topic coming into this broker (byte per second). |
+| pulsar_throughput_out | Gauge | The total throughput of the topic going out from this broker (byte per second). |
| pulsar_storage_size | Gauge | The total storage size of the topics in this topic owned by this broker (bytes). |
| pulsar_storage_logical_size | Gauge | The storage size of topics in the namespace owned by the broker without replicas (in bytes). |
| pulsar_storage_backlog_size | Gauge | The total backlog size of the topics of this topic owned by this broker (messages). |
| pulsar_storage_offloaded_size | Gauge | The total amount of the data in this topic offloaded to the tiered storage (bytes). |
| pulsar_storage_backlog_quota_limit | Gauge | The total amount of the data in this topic that limit the backlog quota (bytes). |
-| pulsar_storage_write_rate | Gauge | The total message batches (entries) written to the storage for this topic (message batches / second). |
-| pulsar_storage_read_rate | Gauge | The total message batches (entries) read from the storage for this topic (message batches / second). |
+| pulsar_storage_write_rate | Gauge | The total message batches (entries) written to the storage for this topic (message batch per second). |
+| pulsar_storage_read_rate | Gauge | The total message batches (entries) read from the storage for this topic (message batch per second). |
| pulsar_subscription_delayed | Gauge | The total message batches (entries) are delayed for dispatching. |
-| pulsar_storage_write_latency_le_* | Histogram | The entry rate of a topic that the storage write latency is smaller with a given threshold.<br> Available thresholds: <br><ul><li>pulsar_storage_write_latency_le_0_5: <= 0.5ms </li><li>pulsar_storage_write_latency_le_1: <= 1ms</li><li>pulsar_storage_write_latency_le_5: <= 5ms</li><li>pulsar_storage_write_latency_le_10: <= 10ms</li><li>pulsar_storage_write_latency_le_20: <= 20ms</li><li>pulsar_storage_write_latency_le_50: <= 50ms</li><li>p [...]
-| pulsar_entry_size_le_* | Histogram | The entry rate of a topic that the entry size is smaller with a given threshold.<br> Available thresholds: <br><ul><li>pulsar_entry_size_le_128: <= 128 bytes </li><li>pulsar_entry_size_le_512: <= 512 bytes</li><li>pulsar_entry_size_le_1_kb: <= 1 KB</li><li>pulsar_entry_size_le_2_kb: <= 2 KB</li><li>pulsar_entry_size_le_4_kb: <= 4 KB</li><li>pulsar_entry_size_le_16_kb: <= 16 KB</li><li>pulsar_entry_size_le_100_kb: <= 100 KB</li><li>pulsar_entry_size_ [...]
+| pulsar_storage_write_latency_le_* | Histogram | The entry rate of a topic that the storage write latency is smaller with a given threshold.<br /> Available thresholds: <br /><ul><li>pulsar_storage_write_latency_le_0_5: <= 0.5ms </li><li>pulsar_storage_write_latency_le_1: <= 1ms</li><li>pulsar_storage_write_latency_le_5: <= 5ms</li><li>pulsar_storage_write_latency_le_10: <= 10ms</li><li>pulsar_storage_write_latency_le_20: <= 20ms</li><li>pulsar_storage_write_latency_le_50: <= 50ms</li>< [...]
+| pulsar_entry_size_le_* | Histogram | The entry rate of a topic that the entry size is smaller with a given threshold.<br /> Available thresholds: <br /><ul><li>pulsar_entry_size_le_128: <= 128 bytes </li><li>pulsar_entry_size_le_512: <= 512 bytes</li><li>pulsar_entry_size_le_1_kb: <= 1 KB</li><li>pulsar_entry_size_le_2_kb: <= 2 KB</li><li>pulsar_entry_size_le_4_kb: <= 4 KB</li><li>pulsar_entry_size_le_16_kb: <= 16 KB</li><li>pulsar_entry_size_le_100_kb: <= 100 KB</li><li>pulsar_entry_s [...]
| pulsar_in_bytes_total | Counter | The total number of messages in bytes received for this topic. |
| pulsar_in_messages_total | Counter | The total number of messages received for this topic. |
| pulsar_out_bytes_total | Counter | The total number of messages in bytes read from this topic. |
@@ -214,7 +215,7 @@ All the topic metrics are labelled with the following labels:
| pulsar_compaction_duration_time_in_mills | Gauge | The duration time of the compaction. |
| pulsar_compaction_read_throughput | Gauge | The read throughput of the compaction. |
| pulsar_compaction_write_throughput | Gauge | The write throughput of the compaction. |
-| pulsar_compaction_latency_le_* | Histogram | The compaction latency with given quantile. <br> Available thresholds: <br><ul><li>pulsar_compaction_latency_le_0_5: <= 0.5ms </li><li>pulsar_compaction_latency_le_1: <= 1ms</li><li>pulsar_compaction_latency_le_5: <= 5ms</li><li>pulsar_compaction_latency_le_10: <= 10ms</li><li>pulsar_compaction_latency_le_20: <= 20ms</li><li>pulsar_compaction_latency_le_50: <= 50ms</li><li>pulsar_compaction_latency_le_100: <= 100ms</li><li>pulsar_compaction_ [...]
+| pulsar_compaction_latency_le_* | Histogram | The compaction latency with given quantile. <br /> Available thresholds: <br /><ul><li>pulsar_compaction_latency_le_0_5: <= 0.5ms </li><li>pulsar_compaction_latency_le_1: <= 1ms</li><li>pulsar_compaction_latency_le_5: <= 5ms</li><li>pulsar_compaction_latency_le_10: <= 10ms</li><li>pulsar_compaction_latency_le_20: <= 20ms</li><li>pulsar_compaction_latency_le_50: <= 50ms</li><li>pulsar_compaction_latency_le_100: <= 100ms</li><li>pulsar_compact [...]
| pulsar_compaction_compacted_entries_count | Gauge | The total number of the compacted entries. |
| pulsar_compaction_compacted_entries_size |Gauge | The total size of the compacted entries. |
@@ -226,10 +227,10 @@ All the replication metrics are labelled with `remoteCluster=${pulsar_remote_clu
| Name | Type | Description |
|---|---|---|
-| pulsar_replication_rate_in | Gauge | The total message rate of the topic replicating from remote cluster (messages/second). |
-| pulsar_replication_rate_out | Gauge | The total message rate of the topic replicating to remote cluster (messages/second). |
-| pulsar_replication_throughput_in | Gauge | The total throughput of the topic replicating from remote cluster (bytes/second). |
-| pulsar_replication_throughput_out | Gauge | The total throughput of the topic replicating to remote cluster (bytes/second). |
+| pulsar_replication_rate_in | Gauge | The total message rate of the topic replicating from remote cluster (message per second). |
+| pulsar_replication_rate_out | Gauge | The total message rate of the topic replicating to remote cluster (message per second). |
+| pulsar_replication_throughput_in | Gauge | The total throughput of the topic replicating from remote cluster (byte per second). |
+| pulsar_replication_throughput_out | Gauge | The total throughput of the topic replicating to remote cluster (byte per second). |
| pulsar_replication_backlog | Gauge | The total backlog of the topic replicating to remote cluster (messages). |
#### Topic lookup metrics
@@ -272,8 +273,8 @@ All the managedLedger metrics are labelled with the following labels:
| Name | Type | Description |
| --- | --- | --- |
-| pulsar_ml_AddEntryBytesRate | Gauge | The bytes/s rate of messages added |
-| pulsar_ml_AddEntryWithReplicasBytesRate | Gauge | The bytes/s rate of messages added with replicas |
+| pulsar_ml_AddEntryBytesRate | Gauge | The B/s rate of messages added |
+| pulsar_ml_AddEntryWithReplicasBytesRate | Gauge | The B/s rate of messages added with replicas |
| pulsar_ml_AddEntryErrors | Gauge | The number of addEntry requests that failed |
| pulsar_ml_AddEntryLatencyBuckets | Histogram | The latency of adding a ledger entry with a given quantile (threshold), including time spent on waiting in queue on the broker side.<br> Available quantile: <br><ul><li> quantile="0.0_0.5" is AddEntryLatency between (0.0ms, 0.5ms]</li> <li>quantile="0.5_1.0" is AddEntryLatency between (0.5ms, 1.0ms]</li><li>quantile="1.0_5.0" is AddEntryLatency between (1ms, 5ms]</li><li>quantile="5.0_10.0" is AddEntryLatency between (5ms, 10ms]</li><li>qu [...]
| pulsar_ml_AddEntryLatencyBuckets_OVERFLOW | Gauge | The number of times the AddEntryLatency is longer than 1 second |
@@ -287,7 +288,7 @@ All the managedLedger metrics are labelled with the following labels:
| pulsar_ml_LedgerAddEntryLatencyBuckets_OVERFLOW | Gauge | The number of times the LedgerAddEntryLatency is longer than 1 second |
| pulsar_ml_MarkDeleteRate | Gauge | The rate of mark-delete ops/s |
| pulsar_ml_NumberOfMessagesInBacklog | Gauge | The number of backlog messages for all the consumers |
-| pulsar_ml_ReadEntriesBytesRate | Gauge | The bytes/s rate of messages read |
+| pulsar_ml_ReadEntriesBytesRate | Gauge | The B/s rate of messages read |
| pulsar_ml_ReadEntriesErrors | Gauge | The number of readEntries requests that failed |
| pulsar_ml_ReadEntriesRate | Gauge | The msg/s rate of messages read |
| pulsar_ml_ReadEntriesSucceeded | Gauge | The number of readEntries requests that succeeded |
@@ -358,13 +359,13 @@ All the bundle metrics are labelled with the following labels:
| Name | Type | Description |
| --- | --- | --- |
-| pulsar_bundle_msg_rate_in | Gauge | The total message rate coming into the topics in this bundle (messages/second). |
-| pulsar_bundle_msg_rate_out | Gauge | The total message rate going out from the topics in this bundle (messages/second). |
+| pulsar_bundle_msg_rate_in | Gauge | The total message rate coming into the topics in this bundle (message per second). |
+| pulsar_bundle_msg_rate_out | Gauge | The total message rate going out from the topics in this bundle (message per second). |
| pulsar_bundle_topics_count | Gauge | The topic count in this bundle. |
| pulsar_bundle_consumer_count | Gauge | The consumer count of the topics in this bundle. |
| pulsar_bundle_producer_count | Gauge | The producer count of the topics in this bundle. |
-| pulsar_bundle_msg_throughput_in | Gauge | The total throughput coming into the topics in this bundle (bytes/second). |
-| pulsar_bundle_msg_throughput_out | Gauge | The total throughput going out from the topics in this bundle (bytes/second). |
+| pulsar_bundle_msg_throughput_in | Gauge | The total throughput coming into the topics in this bundle (byte per second). |
+| pulsar_bundle_msg_throughput_out | Gauge | The total throughput going out from the topics in this bundle (byte per second). |
### Subscription metrics
@@ -382,11 +383,11 @@ All the subscription metrics are labelled with the following labels:
| pulsar_subscription_back_log | Gauge | The total backlog of a subscription (entries). |
| pulsar_subscription_back_log_no_delayed | Gauge | The backlog of a subscription that do not contain the delay messages (entries). |
| pulsar_subscription_delayed | Gauge | The total number of messages are delayed to be dispatched for a subscription (messages). |
-| pulsar_subscription_msg_rate_redeliver | Gauge | The total message rate for message being redelivered (messages/second). |
+| pulsar_subscription_msg_rate_redeliver | Gauge | The total message rate for message being redelivered (message per second). |
| pulsar_subscription_unacked_messages | Gauge | The total number of unacknowledged messages of a subscription (messages). |
-| pulsar_subscription_blocked_on_unacked_messages | Gauge | Indicate whether a subscription is blocked on unacknowledged messages or not. <br> <ul><li>1 means the subscription is blocked on waiting unacknowledged messages to be acked.</li><li>0 means the subscription is not blocked on waiting unacknowledged messages to be acked.</li></ul> |
-| pulsar_subscription_msg_rate_out | Gauge | The total message dispatch rate for a subscription (messages/second). |
-| pulsar_subscription_msg_throughput_out | Gauge | The total message dispatch throughput for a subscription (bytes/second). |
+| pulsar_subscription_blocked_on_unacked_messages | Gauge | Indicate whether a subscription is blocked on unacknowledged messages or not. <br /> <ul><li>1 means the subscription is blocked on waiting unacknowledged messages to be acked.</li><li>0 means the subscription is not blocked on waiting unacknowledged messages to be acked.</li></ul> |
+| pulsar_subscription_msg_rate_out | Gauge | The total message dispatch rate for a subscription (message per second). |
+| pulsar_subscription_msg_throughput_out | Gauge | The total message dispatch throughput for a subscription (byte per second). |
### Consumer metrics
@@ -403,11 +404,11 @@ All the consumer metrics are labelled with the following labels:
| Name | Type | Description |
|---|---|---|
-| pulsar_consumer_msg_rate_redeliver | Gauge | The total message rate for message being redelivered (messages/second). |
+| pulsar_consumer_msg_rate_redeliver | Gauge | The total message rate for message being redelivered (message per second). |
| pulsar_consumer_unacked_messages | Gauge | The total number of unacknowledged messages of a consumer (messages). |
-| pulsar_consumer_blocked_on_unacked_messages | Gauge | Indicate whether a consumer is blocked on unacknowledged messages or not. <br> <ul><li>1 means the consumer is blocked on waiting unacknowledged messages to be acked.</li><li>0 means the consumer is not blocked on waiting unacknowledged messages to be acked.</li></ul> |
-| pulsar_consumer_msg_rate_out | Gauge | The total message dispatch rate for a consumer (messages/second). |
-| pulsar_consumer_msg_throughput_out | Gauge | The total message dispatch throughput for a consumer (bytes/second). |
+| pulsar_consumer_blocked_on_unacked_messages | Gauge | Indicate whether a consumer is blocked on unacknowledged messages or not. <br /> <ul><li>1 means the consumer is blocked on waiting unacknowledged messages to be acked.</li><li>0 means the consumer is not blocked on waiting unacknowledged messages to be acked.</li></ul> |
+| pulsar_consumer_msg_rate_out | Gauge | The total message dispatch rate for a consumer (message per second). |
+| pulsar_consumer_msg_throughput_out | Gauge | The total message dispatch throughput for a consumer (byte per second). |
| pulsar_consumer_available_permits | Gauge | The available permits for for a consumer. |
### Managed ledger bookie client metrics
@@ -515,6 +516,27 @@ All the schema metrics are labelled with the following labels:
| pulsar_schema_get_ops_latency | Summary | Latency of successful operations to get schemas. |
| pulsar_schema_put_ops_latency | Summary | Latency of successful operations to send schemas. |
+### Offload metrics
+
+All the offload metrics are labelled with the following labels:
+
+- *cluster*: `cluster=${pulsar_cluster}`. `${pulsar_cluster}` is the cluster name that you configured in `broker.conf`.
+- *namespace*: `namespace=${pulsar_namespace}`. `${pulsar_namespace}` is the namespace name.
+- *topic*: `topic=${pulsar_topic}`. `${pulsar_topic}` is the topic name.
+
+| Name | Type | Description |
+|------------------------------------------------|---------|---------------------------------------------------------------------------------|
+| brk_ledgeroffloader_offload_error | Counter | The number of failed operations to offload. |
+| brk_ledgeroffloader_offload_rate | Gauge | The rate of offloading(byte per second). |
+| brk_ledgeroffloader_read_offload_error | Counter | The number of failed operations to read offload ledgers. |
+| brk_ledgeroffloader_read_offload_rate | Gauge | The rate of reading entries from offload ledgers(byte per second). |
+| brk_ledgeroffloader_write_storage_error | Counter | The number of failed operations to write to storage. |
+| brk_ledgeroffloader_read_offload_index_latency | Summary | The latency of reading index from offload ledgers. |
+| brk_ledgeroffloader_read_offload_data_latency | Summary | The latency of reading data from offload ledgers. |
+| brk_ledgeroffloader_read_ledger_latency | Summary | The latency of reading entries from BookKeeper. |
+| brk_ledgeroffloader_delete_offload_ops | Counter | The total number of successful and failed operations to delete offload ledgers. |
+
+
## Pulsar Functions
All the Pulsar Functions metrics are labelled with the following labels:
@@ -637,4 +659,4 @@ All the transaction metrics are labelled with the following labels:
| pulsar_txn_aborted_count | Counter | Number of aborted transactions of this coordinator. |
| pulsar_txn_timeout_count | Counter | Number of timeout transactions. |
| pulsar_txn_append_log_count | Counter | Number of append transaction logs. |
-| pulsar_txn_execution_latency_le_* | Histogram | Transaction execution latency. <br> Available latencies are as below: <br><ul><li> latency="10" is TransactionExecutionLatency between (0ms, 10ms]</li> <li>latency="20" is TransactionExecutionLatency between (10ms, 20ms]</li><li>latency="50" is TransactionExecutionLatency between (20ms, 50ms]</li><li>latency="100" is TransactionExecutionLatency between (50ms, 100ms]</li><li>latency="500" is TransactionExecutionLatency between (100ms, 500m [...]
+| pulsar_txn_execution_latency_le_* | Histogram | Transaction execution latency. <br /> Available latencies are as below: <br /><ul><li> latency="10" is TransactionExecutionLatency between (0ms, 10ms]</li> <li>latency="20" is TransactionExecutionLatency between (10ms, 20ms]</li><li>latency="50" is TransactionExecutionLatency between (20ms, 50ms]</li><li>latency="100" is TransactionExecutionLatency between (50ms, 100ms]</li><li>latency="500" is TransactionExecutionLatency between (100ms, [...]
diff --git a/site2/docs/tiered-storage-aliyun.md b/site2/docs/tiered-storage-aliyun.md
index c131842e9bc..183bd746feb 100644
--- a/site2/docs/tiered-storage-aliyun.md
+++ b/site2/docs/tiered-storage-aliyun.md
@@ -24,8 +24,6 @@ This example uses Pulsar 2.8.0.
**Output**
- As shown from the output, Pulsar uses [Apache jclouds](https://jclouds.apache.org) to support [AWS S3](https://aws.amazon.com/s3/), [GCS](https://cloud.google.com/storage/), [Azure](https://portal.azure.com/#home), and [Aliyun OSS](https://www.aliyun.com/product/oss) for long-term storage.
-
```
tiered-storage-file-system-2.8.0.nar
tiered-storage-jcloud-2.8.0.nar
@@ -64,8 +62,8 @@ You can configure the Aliyun OSS offloader driver in the configuration file `bro
| --- | --- | --- |
| `managedLedgerOffloadReadBufferSizeInBytes` | Size of block read | 1 MB |
| `managedLedgerOffloadMaxBlockSizeInBytes` | Size of block write | 64 MB |
- | `managedLedgerMinLedgerRolloverTimeMinutes` | Minimum time between ledger rollover for a topic<br><br>**Note**: it is not recommended that you set this configuration in the production environment. | 2 |
- | `managedLedgerMaxEntriesPerLedger` | Maximum number of entries to append to a ledger before triggering a rollover.<br><br>**Note**: it is not recommended that you set this configuration in the production environment. | 5000 |
+ | `managedLedgerMinLedgerRolloverTimeMinutes` | Minimum time between ledger rollover for a topic<br /><br />**Note**: it is not recommended that you set this configuration in the production environment. | 2 |
+ | `managedLedgerMaxEntriesPerLedger` | Maximum number of entries to append to a ledger before triggering a rollover.<br /><br />**Note**: it is not recommended that you set this configuration in the production environment. | 5000 |
#### Bucket (required)
diff --git a/site2/docs/tiered-storage-aws.md b/site2/docs/tiered-storage-aws.md
index 640c104464e..75b430732a7 100644
--- a/site2/docs/tiered-storage-aws.md
+++ b/site2/docs/tiered-storage-aws.md
@@ -47,8 +47,6 @@ This example uses Pulsar 2.5.1.
**Output**
- As shown from the output, Pulsar uses [Apache jclouds](https://jclouds.apache.org) to support [AWS S3](https://aws.amazon.com/s3/) and [GCS](https://cloud.google.com/storage/) for long term storage.
-
```
tiered-storage-file-system-2.5.1.nar
tiered-storage-jcloud-2.5.1.nar
@@ -76,7 +74,7 @@ You can configure the AWS S3 offloader driver in the configuration file `broker.
Required configuration | Description | Example value
|---|---|---
- `managedLedgerOffloadDriver` | Offloader driver name, which is case-insensitive. <br><br>**Note**: there is a third driver type, S3, which is identical to AWS S3, though S3 requires that you specify an endpoint URL using `s3ManagedLedgerOffloadServiceEndpoint`. This is useful if using an S3 compatible data store other than AWS S3. | aws-s3
+ `managedLedgerOffloadDriver` | Offloader driver name, which is case-insensitive. <br /><br />**Note**: there is a third driver type, S3, which is identical to AWS S3, though S3 requires that you specify an endpoint URL using `s3ManagedLedgerOffloadServiceEndpoint`. This is useful if using an S3 compatible data store other than AWS S3. | aws-s3
`offloadersDirectory` | Offloader directory | offloaders
`s3ManagedLedgerOffloadBucket` | Bucket | pulsar-topic-offload
@@ -84,11 +82,11 @@ You can configure the AWS S3 offloader driver in the configuration file `broker.
Optional | Description | Example value
|---|---|---
- `s3ManagedLedgerOffloadRegion` | Bucket region <br><br>**Note**: before specifying a value for this parameter, you need to set the following configurations. Otherwise, you might get an error.<br><br>- Set [`s3ManagedLedgerOffloadServiceEndpoint`](https://docs.aws.amazon.com/general/latest/gr/s3.html).<br><br>Example<br>`s3ManagedLedgerOffloadServiceEndpoint=https://s3.YOUR_REGION.amazonaws.com`<br><br>- Grant `GetBucketLocation` permission to a user.<br><br>For how to grant `GetBucke [...]
+ `s3ManagedLedgerOffloadRegion` | Bucket region <br /><br />**Note**: before specifying a value for this parameter, you need to set the following configurations. Otherwise, you might get an error.<br /><br />- Set [`s3ManagedLedgerOffloadServiceEndpoint`](https://docs.aws.amazon.com/general/latest/gr/s3.html).<br /><br />Example<br />`s3ManagedLedgerOffloadServiceEndpoint=https://s3.YOUR_REGION.amazonaws.com`<br /><br />- Grant `GetBucketLocation` permission to a user.<br /><br />For [...]
`s3ManagedLedgerOffloadReadBufferSizeInBytes`|Size of block read|1 MB
`s3ManagedLedgerOffloadMaxBlockSizeInBytes`|Size of block write|64 MB
- `managedLedgerMinLedgerRolloverTimeMinutes`|Minimum time between ledger rollover for a topic<br><br>**Note**: it is not recommended that you set this configuration in the production environment.|2
- `managedLedgerMaxEntriesPerLedger`|Maximum number of entries to append to a ledger before triggering a rollover.<br><br>**Note**: it is not recommended that you set this configuration in the production environment.|5000
+ `managedLedgerMinLedgerRolloverTimeMinutes`|Minimum time between ledger rollover for a topic<br /><br />**Note**: it is not recommended that you set this configuration in the production environment.|2
+ `managedLedgerMaxEntriesPerLedger`|Maximum number of entries to append to a ledger before triggering a rollover.<br /><br />**Note**: it is not recommended that you set this configuration in the production environment.|5000
#### Bucket (required)
diff --git a/site2/docs/tiered-storage-azure.md b/site2/docs/tiered-storage-azure.md
index 64feb69496b..019cafab7a6 100644
--- a/site2/docs/tiered-storage-azure.md
+++ b/site2/docs/tiered-storage-azure.md
@@ -47,8 +47,6 @@ This example uses Pulsar 2.6.2.
**Output**
- As shown from the output, Pulsar uses [Apache jclouds](https://jclouds.apache.org) to support [AWS S3](https://aws.amazon.com/s3/), [GCS](https://cloud.google.com/storage/) and [Azure](https://portal.azure.com/#home) for long term storage.
-
```
tiered-storage-file-system-2.6.2.nar
tiered-storage-jcloud-2.6.2.nar
@@ -86,8 +84,8 @@ You can configure the Azure BlobStore offloader driver in the configuration file
|---|---|---
`managedLedgerOffloadReadBufferSizeInBytes`|Size of block read|1 MB
`managedLedgerOffloadMaxBlockSizeInBytes`|Size of block write|64 MB
- `managedLedgerMinLedgerRolloverTimeMinutes`|Minimum time between ledger rollover for a topic<br><br>**Note**: it is not recommended that you set this configuration in the production environment.|2
- `managedLedgerMaxEntriesPerLedger`|Maximum number of entries to append to a ledger before triggering a rollover.<br><br>**Note**: it is not recommended that you set this configuration in the production environment.|5000
+ `managedLedgerMinLedgerRolloverTimeMinutes`|Minimum time between ledger rollover for a topic<br /><br />**Note**: it is not recommended that you set this configuration in the production environment.|2
+ `managedLedgerMaxEntriesPerLedger`|Maximum number of entries to append to a ledger before triggering a rollover.<br /><br />**Note**: it is not recommended that you set this configuration in the production environment.|5000
#### Bucket (required)
diff --git a/site2/docs/tiered-storage-gcs.md b/site2/docs/tiered-storage-gcs.md
index 02f96184c15..90d891f3443 100644
--- a/site2/docs/tiered-storage-gcs.md
+++ b/site2/docs/tiered-storage-gcs.md
@@ -54,9 +54,6 @@ This example uses Pulsar 2.5.1.
**Output**
- As shown in the output, Pulsar uses [Apache jclouds](https://jclouds.apache.org) to support GCS and AWS S3 for long term storage.
-
-
```
tiered-storage-file-system-2.5.1.nar
tiered-storage-jcloud-2.5.1.nar
@@ -166,8 +163,8 @@ You can configure the size of a request sent to or read from GCS in the configur
Configuration|Description
|---|---
-`gcsManagedLedgerOffloadReadBufferSizeInBytes`|Block size for each individual read when reading back data from GCS.<br><br>The **default** value is 1 MB.
-`gcsManagedLedgerOffloadMaxBlockSizeInBytes`|Maximum size of a "part" sent during a multipart upload to GCS. <br><br>It **can not** be smaller than 5 MB. <br><br>The **default** value is 64 MB.
+`gcsManagedLedgerOffloadReadBufferSizeInBytes`|Block size for each individual read when reading back data from GCS.<br /><br />The **default** value is 1 MB.
+`gcsManagedLedgerOffloadMaxBlockSizeInBytes`|Maximum size of a "part" sent during a multipart upload to GCS. <br /><br />It **can not** be smaller than 5 MB. <br /><br />The **default** value is 64 MB.
### Configure GCS offloader to run automatically
diff --git a/site2/docs/tiered-storage-overview.md b/site2/docs/tiered-storage-overview.md
index 320513764a6..aa1595d7040 100644
--- a/site2/docs/tiered-storage-overview.md
+++ b/site2/docs/tiered-storage-overview.md
@@ -7,7 +7,7 @@ sidebar_label: Overview
Pulsar's **Tiered Storage** feature allows older backlog data to be moved from BookKeeper to long term and cheaper storage, while still allowing clients to access the backlog as if nothing has changed.
* Tiered storage uses [Apache jclouds](https://jclouds.apache.org) to support
-[Amazon S3](https://aws.amazon.com/s3/) and [GCS (Google Cloud Storage)](https://cloud.google.com/storage/) for long term storage.
+[Amazon S3](https://aws.amazon.com/s3/), [GCS (Google Cloud Storage)](https://cloud.google.com/storage/), [Azure](https://azure.microsoft.com/en-us/services/storage/blobs/) and [Aliyun OSS](https://www.aliyun.com/product/oss) for long term storage.
With jclouds, it is easy to add support for more
[cloud storage providers](https://jclouds.apache.org/reference/providers/#blobstore-providers) in the future.
@@ -17,6 +17,10 @@ Pulsar's **Tiered Storage** feature allows older backlog data to be moved from B
> For more information about how to use the AWS S3 offloader with Pulsar, see [here](tiered-storage-aws.md).
>
> For more information about how to use the GCS offloader with Pulsar, see [here](tiered-storage-gcs.md).
+ >
+ > For more information about how to use the Azure offloader with Pulsar, see [here](tiered-storage-azure.md).
+ >
+ > For more information about how to use the Aliyun OSS offloader with Pulsar, see [here](tiered-storage-aliyun.md).
* Tiered storage uses [Apache Hadoop](http://hadoop.apache.org/) to support filesystems for long term storage.
@@ -47,3 +51,5 @@ Before offloading ledgers to long term storage, you need to configure buckets, c
After offloading ledgers to long term storage, you can still query data in the offloaded ledgers with Pulsar SQL.
For more information about tiered storage for Pulsar topics, see [here](https://github.com/apache/pulsar/wiki/PIP-17:-Tiered-storage-for-Pulsar-topics).
+
+For more information about offload metrics, see [here](reference-metrics.md#offload-metrics).
\ No newline at end of file