You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "equanz (via GitHub)" <gi...@apache.org> on 2025/10/03 07:45:10 UTC
[PR] [feat][monitor] Add ML write latency histogram and entry size histogram as OTel metrics [pulsar]
equanz opened a new pull request, #24815:
URL: https://github.com/apache/pulsar/pull/24815
PIP: #21080
### Motivation
Please see the [PIP-264 doc](https://github.com/apache/pulsar/blob/v4.1.1/pip/pip-264.md).
ref.
* https://github.com/apache/pulsar/blob/v4.1.1/pip/pip-264.md#moving-topic-level-histograms-to-namespace-and-broker-level-only
* Note:
> Histograms are problematic in that sense, since they yet to have an asynchronous version - alas they are only synchronous. Hence, if we use them for topic instruments, used attributes can never be cleared from memory for them thus we’ll have an “attributes” (topic) leak, as over time it will only grow.
* Therefore, even if a namespace is deleted, the namespace-level histogram metrics persist
* https://github.com/apache/pulsar/blob/v4.1.1/pip/pip-264.md#which-summaries--histograms-are-effected
### Modifications
<!-- Describe the modifications you've done. -->
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
This change added tests and can be verified as follows:
- Added a simple E2E test to verify the expected metrics is exposed
### Does this pull request potentially affect one of the following parts:
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
*If the box was checked, please highlight the changes*
- [ ] Dependencies (add or upgrade a dependency)
- [ ] The public API
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [x] The metrics
- Introducing the new OTel metrics
- `pulsar.broker.managed_ledger.message.outgoing.latency`
- `pulsar.broker.managed_ledger.message.outgoing.ledger.latency`
- `pulsar.broker.managed_ledger.ledger.switch.latency`
- `pulsar.broker.managed_ledger.entry.size`
- [ ] Anything that affects deployment
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [x] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
### Matching PR in forked repository
PR in forked repository: <!-- ENTER URL HERE -->
<!--
After opening this PR, the build in apache/pulsar will fail and instructions will
be provided for opening a PR in the PR author's forked repository.
apache/pulsar pull requests should be first tested in your own fork since the
apache/pulsar CI based on GitHub Actions has constrained resources and quota.
GitHub Actions provides separate quota for pull requests that are executed in
a forked repository.
The tests will be run in the forked repository until all PR review comments have
been handled, the tests pass and the PR is approved by a reviewer.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat][monitor] Add ML write latency histogram and entry size histogram as OTel metrics [pulsar]
Posted by "equanz (via GitHub)" <gi...@apache.org>.
equanz commented on PR #24815:
URL: https://github.com/apache/pulsar/pull/24815#issuecomment-3364683536
Note:
In my understanding, implementing AutoConfigurationCustomizerProvider allows you to customize the histogram buckets.
Here is an example.
```java
import com.google.auto.service.AutoService;
import io.opentelemetry.sdk.autoconfigure.spi.AutoConfigurationCustomizer;
import io.opentelemetry.sdk.autoconfigure.spi.AutoConfigurationCustomizerProvider;
import io.opentelemetry.sdk.metrics.Aggregation;
import io.opentelemetry.sdk.metrics.InstrumentSelector;
import io.opentelemetry.sdk.metrics.View;
import java.util.Arrays;
@AutoService(AutoConfigurationCustomizerProvider.class)
public class MyCustomizerProvider implements AutoConfigurationCustomizerProvider {
@Override
public void customize(AutoConfigurationCustomizer customizer) {
customizer.addMeterProviderCustomizer(((sdkMeterProviderBuilder, configProperties) -> {
sdkMeterProviderBuilder.registerView(InstrumentSelector.builder()
.setName("pulsar.broker.topic.publish.latency").build(),
View.builder()
.setAggregation(Aggregation.explicitBucketHistogram(Arrays.asList(0.001, 30.0))).build());
return sdkMeterProviderBuilder;
}));
}
}
```
```
$ PULSAR_EXTRA_CLASSPATH='path/to/custom-provider.jar' PULSAR_EXTRA_OPTS=' -Dotel.sdk.disabled=false -Dotel.metrics.exporter=prometheus ' ./bin/pulsar standalone
```
ref.
* https://opentelemetry.io/docs/languages/java/sdk/#views
* https://opentelemetry.io/docs/languages/java/configuration/#zero-code-sdk-autoconfigure
* The `pulsar-opentelemetry` uses AutoConfiguredOpenTelemetrySdk. So, we can use AutoConfigurationCustomizerProvider
https://github.com/apache/pulsar/blob/v4.1.1/pulsar-opentelemetry/src/main/java/org/apache/pulsar/opentelemetry/OpenTelemetryService.java#L72
* https://opentelemetry.io/ja/docs/languages/java/configuration/#autoconfigurationcustomizerprovider
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat][monitor] Add ML write latency histogram and entry size histogram as OTel metrics [pulsar]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24815:
URL: https://github.com/apache/pulsar/pull/24815#issuecomment-3378443086
## [Codecov](https://app.codecov.io/gh/apache/pulsar/pull/24815?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 74.23%. Comparing base ([`771ce41`](https://app.codecov.io/gh/apache/pulsar/commit/771ce4185cfc479976d0fe8dd6bebacc9e701d1c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`14d025c`](https://app.codecov.io/gh/apache/pulsar/commit/14d025cc1899b33337afa6623113e69be4c7d12e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 6 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
[](https://app.codecov.io/gh/apache/pulsar/pull/24815?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
```diff
@@ Coverage Diff @@
## master #24815 +/- ##
============================================
- Coverage 74.50% 74.23% -0.28%
+ Complexity 33895 33275 -620
============================================
Files 1912 1912
Lines 149072 149128 +56
Branches 17299 17300 +1
============================================
- Hits 111071 110710 -361
- Misses 29218 29578 +360
- Partials 8783 8840 +57
```
| [Flag](https://app.codecov.io/gh/apache/pulsar/pull/24815/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [inttests](https://app.codecov.io/gh/apache/pulsar/pull/24815/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `26.45% <90.19%> (-0.30%)` | :arrow_down: |
| [systests](https://app.codecov.io/gh/apache/pulsar/pull/24815/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `22.78% <100.00%> (-0.14%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pulsar/pull/24815/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.75% <100.00%> (-0.28%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/pulsar/pull/24815?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...he/bookkeeper/mledger/ManagedLedgerAttributes.java](https://app.codecov.io/gh/apache/pulsar/pull/24815?src=pr&el=tree&filepath=managed-ledger%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbookkeeper%2Fmledger%2FManagedLedgerAttributes.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-bWFuYWdlZC1sZWRnZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2Jvb2trZWVwZXIvbWxlZGdlci9NYW5hZ2VkTGVkZ2VyQXR0cmlidXRlcy5qYXZh) | `100.00% <100.00%> (ø)` | |
| [...kkeeper/mledger/impl/ManagedLedgerFactoryImpl.java](https://app.codecov.io/gh/apache/pulsar/pull/24815?src=pr&el=tree&filepath=managed-ledger%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbookkeeper%2Fmledger%2Fimpl%2FManagedLedgerFactoryImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-bWFuYWdlZC1sZWRnZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2Jvb2trZWVwZXIvbWxlZGdlci9pbXBsL01hbmFnZWRMZWRnZXJGYWN0b3J5SW1wbC5qYXZh) | `58.86% <ø> (ø)` | |
| [...ookkeeper/mledger/impl/ManagedLedgerMBeanImpl.java](https://app.codecov.io/gh/apache/pulsar/pull/24815?src=pr&el=tree&filepath=managed-ledger%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbookkeeper%2Fmledger%2Fimpl%2FManagedLedgerMBeanImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-bWFuYWdlZC1sZWRnZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2Jvb2trZWVwZXIvbWxlZGdlci9pbXBsL01hbmFnZWRMZWRnZXJNQmVhbkltcGwuamF2YQ==) | `90.38% <100.00%> (+0.51%)` | :arrow_up: |
| [.../mledger/impl/OpenTelemetryManagedLedgerStats.java](https://app.codecov.io/gh/apache/pulsar/pull/24815?src=pr&el=tree&filepath=managed-ledger%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbookkeeper%2Fmledger%2Fimpl%2FOpenTelemetryManagedLedgerStats.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-bWFuYWdlZC1sZWRnZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2Jvb2trZWVwZXIvbWxlZGdlci9pbXBsL09wZW5UZWxlbWV0cnlNYW5hZ2VkTGVkZ2VyU3RhdHMuamF2YQ==) | `100.00% <100.00%> (ø)` | |
... and [89 files with indirect coverage changes](https://app.codecov.io/gh/apache/pulsar/pull/24815/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat][monitor] Add ML write latency histogram and entry size histogram as OTel metrics [pulsar]
Posted by "equanz (via GitHub)" <gi...@apache.org>.
equanz commented on PR #24815:
URL: https://github.com/apache/pulsar/pull/24815#issuecomment-3379342264
Rebased to the master to resolve conflicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat][monitor] Add ML write latency histogram and entry size histogram as OTel metrics [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24815:
URL: https://github.com/apache/pulsar/pull/24815#discussion_r2408434481
##########
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpenTelemetryManagedLedgerStats.java:
##########
@@ -151,4 +213,24 @@ private void recordMetrics(ManagedLedger ml) {
markDeleteCounter.record(stats.getMarkDeleteTotal(), attributes);
readEntryCacheMissCounter.record(stats.getReadEntriesOpsCacheMissesTotal(), attributes);
}
+
+ void recordAddEntryLatency(long latency, TimeUnit unit, ManagedLedger ml) {
+ final var attributes = ml.getManagedLedgerAttributes().getAttributesOnlyNamespace();
Review Comment:
Is there a specific reason to track add namespace level? Could you add comments in the source code to explain the reasoning?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat][monitor] Add ML write latency histogram and entry size histogram as OTel metrics [pulsar]
Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui merged PR #24815:
URL: https://github.com/apache/pulsar/pull/24815
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat][monitor] Add ML write latency histogram and entry size histogram as OTel metrics [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24815:
URL: https://github.com/apache/pulsar/pull/24815#discussion_r2408472557
##########
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpenTelemetryManagedLedgerStats.java:
##########
@@ -151,4 +213,24 @@ private void recordMetrics(ManagedLedger ml) {
markDeleteCounter.record(stats.getMarkDeleteTotal(), attributes);
readEntryCacheMissCounter.record(stats.getReadEntriesOpsCacheMissesTotal(), attributes);
}
+
+ void recordAddEntryLatency(long latency, TimeUnit unit, ManagedLedger ml) {
+ final var attributes = ml.getManagedLedgerAttributes().getAttributesOnlyNamespace();
Review Comment:
Now I got it from the referenced PIP-264 section. "In OTel, the API doesn't support remove() for an attribute set on an instrument (counter, histogram, etc.).". I didn't understand the asynchronous vs. synchronous description and the lack of being able to remove the instruments makes more sense to me. Perhaps summarizing the reason to use `getAttributesOnlyNamespace` in a code comment would be useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat][monitor] Add ML write latency histogram and entry size histogram as OTel metrics [pulsar]
Posted by "equanz (via GitHub)" <gi...@apache.org>.
equanz commented on code in PR #24815:
URL: https://github.com/apache/pulsar/pull/24815#discussion_r2409438733
##########
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpenTelemetryManagedLedgerStats.java:
##########
@@ -151,4 +213,24 @@ private void recordMetrics(ManagedLedger ml) {
markDeleteCounter.record(stats.getMarkDeleteTotal(), attributes);
readEntryCacheMissCounter.record(stats.getReadEntriesOpsCacheMissesTotal(), attributes);
}
+
+ void recordAddEntryLatency(long latency, TimeUnit unit, ManagedLedger ml) {
+ final var attributes = ml.getManagedLedgerAttributes().getAttributesOnlyNamespace();
Review Comment:
> Perhaps summarizing the reason to use getAttributesOnlyNamespace in a code comment would be useful.
Added. PTAL
https://github.com/apache/pulsar/compare/f65a473b30130cbd7bc21b6fe02c085473396b36..14d025cc1899b33337afa6623113e69be4c7d12e
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org