You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "coderzc (via GitHub)" <gi...@apache.org> on 2025/10/16 13:30:24 UTC
[PR] [feat][pip] PIP-447: Customizable Prometheus Labels for Topic Metrics [pulsar]
coderzc opened a new pull request, #24862:
URL: https://github.com/apache/pulsar/pull/24862
<!--
### Contribution Checklist
- PR title format should be *[type][component] summary*. For details, see *[Guideline - Pulsar PR Naming Convention](https://pulsar.apache.org/contribute/develop-semantic-title/)*.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
-->
<!-- Either this PR fixes an issue, -->
Fixes #xyz
<!-- or this PR is one task of an issue -->
Main Issue: #xyz
<!-- If the PR belongs to a PIP, please add the PIP link here -->
PIP: #xyz
<!-- Details of when a PIP is required and how the PIP process work, please see: https://github.com/apache/pulsar/blob/master/pip/README.md -->
### Motivation
<!-- Explain here the context, and why you're making that change. What is the problem you're trying to solve. -->
### Modifications
<!-- Describe the modifications you've done. -->
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
*(Please pick either of the following options)*
This change is a trivial rework / code cleanup without any test coverage.
*(or)*
This change is already covered by existing tests, such as *(please describe tests)*.
*(or)*
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads (10MB)*
- *Extended integration test for recovery after broker failure*
### Does this pull request potentially affect one of the following parts:
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
*If the box was checked, please highlight the changes*
- [ ] Dependencies (add or upgrade a dependency)
- [ ] The public API
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [ ] The metrics
- [ ] Anything that affects deployment
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
### Matching PR in forked repository
PR in forked repository: <!-- ENTER URL HERE -->
<!--
After opening this PR, the build in apache/pulsar will fail and instructions will
be provided for opening a PR in the PR author's forked repository.
apache/pulsar pull requests should be first tested in your own fork since the
apache/pulsar CI based on GitHub Actions has constrained resources and quota.
GitHub Actions provides separate quota for pull requests that are executed in
a forked repository.
The tests will be run in the forked repository until all PR review comments have
been handled, the tests pass and the PR is approved by a reviewer.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat][pip] PIP-447: Customizable Prometheus Labels for Topic Metrics [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24862:
URL: https://github.com/apache/pulsar/pull/24862#discussion_r2579694791
##########
pip/pip-447.md:
##########
@@ -0,0 +1,230 @@
+# PIP-447: Customizable Prometheus Labels for Topic Metrics
+
+# Abstract
+
+This PIP proposes a mechanism to add customizable Prometheus labels to Apache Pulsar topic-level metrics. Administrators will define a set of allowed custom metric label keys at the cluster or broker level. Users can then assign values to these predefined keys for specific topics. These key-value pairs will be exposed as Prometheus labels, enabling more granular metric filtering and precise alerting, while managing metric cardinality through centralized key governance.
+
+# Motivation
+
+Currently, Pulsar topic metrics exposed to Prometheus have a fixed set of labels: cluster, namespace, and topic. This limits users' ability to categorize topics for alerting and dashboarding based on custom criteria (e.g., SLA tier, data sensitivity, application owner) that are not easily expressed through topic names alone. Relying on regular expressions (regex) applied to the topic label in Prometheus for such grouping is often complex, inefficient, and error-prone.
+
+This limitation can lead to:
+
+Imprecise Alerting: Difficulty in setting distinct alerting thresholds for different categories of topics.
+
+Alert Fatigue or Missed Alerts: Overly broad or overly complex alerting rules.
+
+Operational Overhead: Increased effort in managing and maintaining Prometheus alert configurations.
+
+Users require a native Pulsar mechanism to inject queryable, custom metadata directly into topic metrics to improve alerting precision, simplify dashboarding, and enhance overall observability.
+
+# Goals
+
+The proposed solution allows administrators to define a list of permissible custom metric label keys (e.g., sla_tier, app_owner) in the broker configuration.
+
+Users can then use pulsar-admin commands or the REST API to set string values for these allowed keys on specific topics (e.g., sla_tier=gold).
+
+These key-value pairs will be stored as part of the topic's metadata, leveraging Pulsar's topic-level policy system.
+
+When topic metrics are generated for Prometheus, these custom key-value pairs will be added as labels to the existing set of standard labels.
+
+## In Scope
+
+The primary goals of this proposal are to:
+
+Allow administrators to define a configurable set of allowed custom metric label keys at the broker or cluster level.
+
+Enable users (or automated systems) to assign string values to these predefined keys for individual topics.
+
+Expose these user-defined key-value pairs as additional Prometheus labels on all topic-level metrics for the respective topics, provided exposeTopicLevelMetricsInPrometheus is true.
+
+Provide robust control over Prometheus metric cardinality by restricting the set of custom metric label keys and providing limits on their usage.
+
+Integrate this feature with Pulsar's existing topic-level policy framework, utilizing system topics for policy propagation.
+
+## Out of Scope
+
+This PIP does not propose changes to topic-level properties. Topic-level properties are distinct from topic-level custom metrics labels.
+
+It does not aim to replace existing standard metric labels (cluster, namespace, topic).
+
+Support for complex data types as label values; only string values will be supported.
+
+# High Level Design
+
+API and Data Structure Definition: Finalize the internal data structures for storing custom metric labels within topic policies and the public contracts for pulsar-admin commands and REST APIs.
+
+Broker Configuration Implementation: Add the new configuration parameters to broker.conf and implement the logic for brokers to read and use these settings.
+
+Admin Client and REST API Implementation: Develop the new pulsar-admin topics subcommands and their corresponding REST API endpoints in the broker, including validation logic against allowedCustomMetricLabelKeys.
+
+Broker Policy Handling Logic:
+
+Extend the topic-level policy framework to handle customMetricLabels.
+
+Ensure changes are published to and consumed from the `__change_events` system topic.
+
+Update broker policy caches accordingly.
+
+Metrics Generation Modification: Update the Prometheus metrics servlet (or equivalent OTel exporter logic in the future) to retrieve custom metric labels from the policy cache and add them to the outgoing topic metrics.
+
+# Detailed Design
+
+## Design & Implementation Details
+
+Topic Policy Storage: Custom metric labels will be stored as a Map<String, String> within the topic's policy data structure.
+
+Policy Propagation and Caching: Brokers will use the existing system topic mechanism (__change_events) for propagating custom metric label policy changes and updating their in-memory policy caches.
+
+### Metrics Generation Logic:
+The Prometheus metrics generation component in the broker (e.g., PrometheusMetricsServlet) will be modified.
+
+If `exposeCustomTopicMetricLabelsEnabled` is true, for each topic, it will retrieve the customMetricLabels map from its policy cache.
+
+Each key-value pair from this map will be added as an additional label to all Prometheus metrics emitted for that topic. The standard labels (cluster, namespace, topic) will remain.
+
+Validation: Brokers will enforce maxCustomMetricLabelValueLength during the set-custom-metric-labels operation. Keys will be validated against allowedCustomMetricLabelKeys.
+
+
+## Public-facing Changes
+
+### Configuration
+
+The following new configuration parameters will be introduced in broker.conf:
+
+exposeCustomTopicMetricLabelsEnabled=(true|false)
+
+Description: Enables or disables the custom topic metric labels feature.
+
+Default: false
+
+allowedCustomMetricLabelKeys=<key1>,<key2>,...
+
+Description: A comma-separated list of strings defining the custom metric label keys that administrators allow to be set on topics. Example: sla_tier,data_sensitivity,cost_center,app_owner.
+
+Default: Empty string (if the feature is enabled but no keys are defined, no custom metric labels can be set).
+
+maxCustomMetricLabelValueLength=<integer>
+
+Description: The maximum character length for a custom metric label value.
+
+Default: 128
+
+### Public API
+
+New pulsar-admin topics subcommands and corresponding REST API endpoints will be introduced:
+
+#### Set Custom Metric Labels:
+
+CLI: pulsar-admin topics set-custom-metric-labels <topic-name> --labels "key1=value1,key2=value2"
+
+REST API: POST /admin/v2/topics/{tenant}/{namespace}/{topic}/custom-metric-labels with a JSON payload {"labels": {"key1":"value1", "key2":"value2"}}
+
+Action: Sets or updates custom metric labels for the specified topic.
+
+The broker (or admin client before sending) will validate that all provided keys (e.g., key1, key2) are present in the allowedCustomMetricLabelKeys list defined in broker.conf.
+
+User cannot rewrite a label which already defined from Pulsar.
+
+Invalid keys will result in an error. This operation will update the topic's policy and publish a change event to the system topic (__change_events) for that namespace.
+
+#### Get Custom Metric Labels:
+
+CLI: pulsar-admin topics get-custom-metric-labels <topic-name>
+
+REST API: GET /admin/v2/topics/{tenant}/{namespace}/{topic}/custom-metric-labels
+
+Action: Retrieves the currently set custom metric labels for the topic.
+
+#### Remove Custom Metric Labels:
+
+CLI:
+pulsar-admin topics remove-custom-metric-labels <topic-name> --labels "key1,key2" (to remove specific labels)
+pulsar-admin topics remove-custom-metric-labels <topic-name> --all (to remove all custom metric labels from the topic)
+
+REST API: DELETE /admin/v2/topics/{tenant}/{namespace}/{topic}/custom-metric-labels with a query params keys=k1&keys=k2 or all=true.
+
+Action: Removes the specified custom metric labels or all custom metric labels from the topic. This also updates the topic policy.
+
+# Backward & Forward Compatibility
+
+## Backward Compatibility
+
+Disabled by Default: The feature will be disabled by default (exposeCustomTopicMetricLabelsEnabled=false).
+
+Existing Pulsar deployments will see no change in behavior or metric format.
+
+No Impact if Unused: If the feature is enabled but allowedCustomMetricLabelKeys is not configured or no labels are set on topics, metrics will remain unchanged.
+
+Existing APIs: Existing pulsar-admin commands and REST APIs are unaffected. The new commands and endpoints are additive.
+
+## Forward Compatibility
+
+Prometheus Systems: If a Pulsar broker with this feature enabled sends metrics with custom metric labels to an older Prometheus server or a monitoring system not expecting these additional labels, those systems will typically ignore the extra labels without issue.
+
+Future Enhancements: Future Pulsar versions could extend this feature, for example, by allowing more dynamic management of allowedCustomMetricLabelKeys if deemed safe and necessary.
+
+OpenTelemetry Alignment: The key-value structure of custom metric labels aligns well with OpenTelemetry attributes, ensuring that this feature remains relevant and compatible with Pulsar's evolving metrics infrastructure.
+
+# Testing Plan
+
+A comprehensive testing strategy will be required:
+
+Unit Tests: For new logic in pulsar-admin, REST API handlers, policy management, and metrics generation.
+
+Integration Tests:
+
+Verify correct setting, getting, and removing of custom metric labels via admin tools and REST APIs.
+
+Test validation logic for allowed keys, max labels per topic, and value length.
+
+Ensure correct propagation of custom metric label policy changes via system topics and updates to broker policy caches.
+
+Verify that Prometheus metrics output accurately includes the custom metric labels for relevant topics and does not include them when the feature is disabled or labels are not set.
+
+End-to-End Tests: Simulate a Pulsar cluster environment, set custom metric labels on topics, and scrape metrics using a Prometheus instance to confirm labels appear correctly and can be queried.
+
+Performance Tests: Assess any potential performance impact on brokers, particularly concerning policy updates and metrics generation, especially in environments with a large number of topics.
+
+# Documentation Plan
+
+The official Apache Pulsar documentation will be updated to include:
+
+Concepts Section: Explanation of the custom topic metric labels feature, its purpose, and how it helps with monitoring and alerting.
+
+Administrator Guide:
+
+Instructions on how to enable and configure the feature in broker.conf (e.g., exposeCustomTopicMetricLabelsEnabled, allowedCustomMetricLabelKeys, and other limits).
+
+Best practices for defining allowedCustomMetricLabelKeys with cardinality management in mind.
+
+User Guide / pulsar-admin Reference:
+
+Detailed syntax and examples for the new pulsar-admin topics set/get/remove-custom-metric-labels commands.
+
+Guidance on choosing appropriate values for predefined keys to manage cardinality.
+
+REST API Reference: Documentation for the new REST API endpoints.
+
+Monitoring Section: Notes on how these custom metric labels appear in Prometheus and how they can be used in PromQL queries, along with reminders about cardinality considerations for the Prometheus system.
+
+# Alternatives
+
+Briefly, two other approaches were considered and rejected:
Review Comment:
Were topic metadata (properties) added in [PIP-110](https://github.com/apache/pulsar/blob/master/pip/pip-110.md) considered as the data source for labels?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat][pip] PIP-447: Customizable Prometheus Labels for Topic Metrics [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #24862:
URL: https://github.com/apache/pulsar/pull/24862#issuecomment-3705385573
Vote has passed, https://lists.apache.org/thread/gotoy8oo2sghsckwtt6zh47obgbvnlzs , Marking as approved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat][pip] PIP-447: Customizable Prometheus Labels for Topic Metrics [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari merged PR #24862:
URL: https://github.com/apache/pulsar/pull/24862
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat][pip] PIP-447: Customizable Prometheus Labels for Topic Metrics [pulsar]
Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui commented on code in PR #24862:
URL: https://github.com/apache/pulsar/pull/24862#discussion_r2436772119
##########
pip/pip-447.md:
##########
@@ -0,0 +1,228 @@
+# PIP-447: Customizable Prometheus Labels for Topic Metrics
+
+# Abstract
+
+This PIP proposes a mechanism to add customizable Prometheus labels to Apache Pulsar topic-level metrics. Administrators will define a set of allowed custom metric label keys at the cluster or broker level. Users can then assign values to these predefined keys for specific topics. These key-value pairs will be exposed as Prometheus labels, enabling more granular metric filtering and precise alerting, while managing metric cardinality through centralized key governance.
+
+# Motivation
+
+Currently, Pulsar topic metrics exposed to Prometheus have a fixed set of labels: cluster, namespace, and topic. This limits users' ability to categorize topics for alerting and dashboarding based on custom criteria (e.g., SLA tier, data sensitivity, application owner) that are not easily expressed through topic names alone. Relying on regular expressions (regex) applied to the topic label in Prometheus for such grouping is often complex, inefficient, and error-prone.
+
+This limitation can lead to:
+
+Imprecise Alerting: Difficulty in setting distinct alerting thresholds for different categories of topics.
+
+Alert Fatigue or Missed Alerts: Overly broad or overly complex alerting rules.
+
+Operational Overhead: Increased effort in managing and maintaining Prometheus alert configurations.
+
+Users require a native Pulsar mechanism to inject queryable, custom metadata directly into topic metrics to improve alerting precision, simplify dashboarding, and enhance overall observability.
+
+# Goals
+
+The proposed solution allows administrators to define a list of permissible custom metric label keys (e.g., sla_tier, app_owner) in the broker configuration.
+
+Users can then use pulsar-admin commands or the REST API to set string values for these allowed keys on specific topics (e.g., sla_tier=gold).
+
+These key-value pairs will be stored as part of the topic's metadata, leveraging Pulsar's topic-level policy system.
+
+When topic metrics are generated for Prometheus, these custom key-value pairs will be added as labels to the existing set of standard labels.
+
+## In Scope
+
+The primary goals of this proposal are to:
+
+Allow administrators to define a configurable set of allowed custom metric label keys at the broker or cluster level.
+
+Enable users (or automated systems) to assign string values to these predefined keys for individual topics.
+
+Expose these user-defined key-value pairs as additional Prometheus labels on all topic-level metrics for the respective topics, provided exposeTopicLevelMetricsInPrometheus is true.
+
+Provide robust control over Prometheus metric cardinality by restricting the set of custom metric label keys and providing limits on their usage.
+
+Integrate this feature with Pulsar's existing topic-level policy framework, utilizing system topics for policy propagation.
+
+## Out of Scope
+
+This PIP does not propose changes to topic-level properties. Topic-level properties are distinct from topic-level custom metrics labels.
+
+It does not aim to replace existing standard metric labels (cluster, namespace, topic).
+
+Support for complex data types as label values; only string values will be supported.
+
+# High Level Design
+
+API and Data Structure Definition: Finalize the internal data structures for storing custom metric labels within topic policies and the public contracts for pulsar-admin commands and REST APIs.
+
+Broker Configuration Implementation: Add the new configuration parameters to broker.conf and implement the logic for brokers to read and use these settings.
+
+Admin Client and REST API Implementation: Develop the new pulsar-admin topics subcommands and their corresponding REST API endpoints in the broker, including validation logic against allowedCustomMetricLabelKeys.
+
+Broker Policy Handling Logic:
+
+Extend the topic-level policy framework to handle customMetricLabels.
+
+Ensure changes are published to and consumed from the `__change_events` system topic.
+
+Update broker policy caches accordingly.
+
+Metrics Generation Modification: Update the Prometheus metrics servlet (or equivalent OTel exporter logic in the future) to retrieve custom metric labels from the policy cache and add them to the outgoing topic metrics.
+
+# Detailed Design
+
+## Design & Implementation Details
+
+Topic Policy Storage: Custom metric labels will be stored as a Map<String, String> within the topic's policy data structure.
+
+Policy Propagation and Caching: Brokers will use the existing system topic mechanism (__change_events) for propagating custom metric label policy changes and updating their in-memory policy caches.
+
+### Metrics Generation Logic:
+The Prometheus metrics generation component in the broker (e.g., PrometheusMetricsServlet) will be modified.
+
+If `exposeCustomTopicMetricLabelsEnabled` is true, for each topic, it will retrieve the customMetricLabels map from its policy cache.
+
+Each key-value pair from this map will be added as an additional label to all Prometheus metrics emitted for that topic. The standard labels (cluster, namespace, topic) will remain.
+
+Validation: Brokers will enforce maxCustomMetricLabelValueLength during the set-custom-metric-labels operation. Keys will be validated against allowedCustomMetricLabelKeys.
+
+
+## Public-facing Changes
+
+### Configuration
+
+The following new configuration parameters will be introduced in broker.conf:
+
+exposeCustomTopicMetricLabelsEnabled=(true|false)
+
+Description: Enables or disables the custom topic metric labels feature.
+
+Default: false
+
+allowedCustomMetricLabelKeys=<key1>,<key2>,...
+
+Description: A comma-separated list of strings defining the custom metric label keys that administrators allow to be set on topics. Example: sla_tier,data_sensitivity,cost_center,app_owner.
+
+Default: Empty string (if the feature is enabled but no keys are defined, no custom metric labels can be set).
+
+maxCustomMetricLabelValueLength=<integer>
+
+Description: The maximum character length for a custom metric label value.
+
+Default: 128
+
+### Public API
+
+New pulsar-admin topics subcommands and corresponding REST API endpoints will be introduced:
+
+#### Set Custom Metric Labels:
+
+CLI: pulsar-admin topics set-custom-metric-labels <topic-name> --labels "key1=value1,key2=value2"
Review Comment:
Please mention that user cannot rewrite the label there, I mean do not add a label which already defined from Pulsar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org