[D] Should we expose an IO‑thread level publish buffer backpressure metric? [pulsar]

["Radiancebobo (via GitHub)" <gi...@apache.org>]

GitHub user Radiancebobo created a discussion: Should we expose an IO‑thread level publish buffer backpressure metric?

Currently, connection‑level publish backpressure already has a metric: 
```
pulsar_broker_throttled_connections
```

This is useful for observing throttling caused by per‑connection limits.

However, the publish buffer protection controlled by `maxMessagePublishBufferSizeInMB` is applied at the IO‑thread level. When the pending publish bytes on an IO thread exceed the configured threshold, the broker pauses connections on that IO thread to protect memory usage. From what I understand, this state is currently maintained inside the broker, but it is **not directly visible as an independent metric**.

In production, this makes it harder to tell whether producers are being slowed down specifically because of IO‑thread publish buffer backpressure. Users may only see increased publish latency or throughput drops, but cannot directly distinguish whether the cause is:

- connection‑level throttling,
- topic/broker publish rate limiting,
- IO‑thread publish buffer pressure,
- slow BookKeeper writes,
- or another backpressure path.

I think exposing this as a dedicated metric could be useful. For example:

- `pulsar_broker_publish_buffer_throttled_connections`  
- or an OpenTelemetry‑style metric like:  
  `pulsar.broker.connection.rate_limit.active.count{reason="io_thread_publish_buffer"}`

Such a metric would help users directly observe whether IO‑thread publish buffer backpressure is active. This would make it easier to decide whether to:

- increase `maxMessagePublishBufferSizeInMB`,
- tune `numIOThreads`,
- investigate BookKeeper add‑entry latency,
- or look for other broker‑side publish bottlenecks.

My personal feeling is that this would fill an observability gap. The existing paused/resumed rate‑limit counters are useful, but they represent events and are not specific to the IO‑thread publish buffer reason. A current‑state metric for this specific backpressure condition may be easier to use in dashboards and alerts.

I would like to hear the community’s opinion:

1. Do you think this kind of metric is useful?
2. Should it be added as a dedicated publish‑buffer metric, or as a more generic active rate‑limit metric with a reason label?

If the community agrees that this is useful, I would be happy to help prepare a PR.

GitHub link: https://github.com/apache/pulsar/discussions/25904

----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org

Re: [D] Should we expose an IO‑thread level publish buffer backpressure metric? [pulsar]

["Radiancebobo (via GitHub)" <gi...@apache.org>]

GitHub user Radiancebobo closed the discussion with a comment: Should we expose an IO‑thread level publish buffer backpressure metric?

The master branch has been modified

GitHub link: https://github.com/apache/pulsar/discussions/25904#discussioncomment-17137168

----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org

Re: [D] Should we expose an IO‑thread level publish buffer backpressure metric? [pulsar]

["Radiancebobo (via GitHub)" <gi...@apache.org>]

GitHub user Radiancebobo closed a discussion: Should we expose an IO‑thread level publish buffer backpressure metric?

Currently, connection‑level publish backpressure already has a metric: 
```
pulsar_broker_throttled_connections
```

This is useful for observing throttling caused by per‑connection limits.

However, the publish buffer protection controlled by `maxMessagePublishBufferSizeInMB` is applied at the IO‑thread level. When the pending publish bytes on an IO thread exceed the configured threshold, the broker pauses connections on that IO thread to protect memory usage. From what I understand, this state is currently maintained inside the broker, but it is **not directly visible as an independent metric**.

In production, this makes it harder to tell whether producers are being slowed down specifically because of IO‑thread publish buffer backpressure. Users may only see increased publish latency or throughput drops, but cannot directly distinguish whether the cause is:

- connection‑level throttling,
- topic/broker publish rate limiting,
- IO‑thread publish buffer pressure,
- slow BookKeeper writes,
- or another backpressure path.

I think exposing this as a dedicated metric could be useful. For example:

- `pulsar_broker_publish_buffer_throttled_connections`  
- or an OpenTelemetry‑style metric like:  
  `pulsar.broker.connection.rate_limit.active.count{reason="io_thread_publish_buffer"}`

Such a metric would help users directly observe whether IO‑thread publish buffer backpressure is active. This would make it easier to decide whether to:

- increase `maxMessagePublishBufferSizeInMB`,
- tune `numIOThreads`,
- investigate BookKeeper add‑entry latency,
- or look for other broker‑side publish bottlenecks.

My personal feeling is that this would fill an observability gap. The existing paused/resumed rate‑limit counters are useful, but they represent events and are not specific to the IO‑thread publish buffer reason. A current‑state metric for this specific backpressure condition may be easier to use in dashboards and alerts.

I would like to hear the community’s opinion:

1. Do you think this kind of metric is useful?
2. Should it be added as a dedicated publish‑buffer metric, or as a more generic active rate‑limit metric with a reason label?

If the community agrees that this is useful, I would be happy to help prepare a PR.

GitHub link: https://github.com/apache/pulsar/discussions/25904

----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org