You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/03/23 14:17:46 UTC
[GitHub] [pulsar] frankjkelly opened a new issue #10016: pulsar_ml_cache_used_size Prometheus metric is negative (Pulsar 2.6.1)
frankjkelly opened a new issue #10016:
URL: https://github.com/apache/pulsar/issues/10016
**Describe the bug**
The prometheus metric `pulsar_ml_cache_used_size` exposed by Broker is negative. I would presume cache sizes should be 0 or above?
**To Reproduce**
1. Launch Pulsar on Kubernetes
1. Inspect metrics endpoint of Broker
```
$ curl -s http://platform-pulsar-broker:8080/metrics/ | grep pulsar_ml_cache_used_size
# TYPE pulsar_ml_cache_used_size gauge
pulsar_ml_cache_used_size{cluster="platform-pulsar"} -312 1616508954848
$ curl -s http://platform-pulsar-broker:8080/metrics/ | grep pulsar_ml_cache_used_size
# TYPE pulsar_ml_cache_used_size gauge
pulsar_ml_cache_used_size{cluster="platform-pulsar"} -829 1616508956674
$ curl -s http://platform-pulsar-broker:8080/metrics/ | grep pulsar_ml_cache_used_size
# TYPE pulsar_ml_cache_used_size gauge
pulsar_ml_cache_used_size{cluster="platform-pulsar"} -308 1616508958131
$ curl -s http://platform-pulsar-broker:8080/metrics/ | grep pulsar_ml_cache_used_size
# TYPE pulsar_ml_cache_used_size gauge
pulsar_ml_cache_used_size{cluster="platform-pulsar"} -1039 1616508958946
```
**Expected behavior**
Cache sizes are zero or above
**Screenshots**
**Additional context**
1. Pulsar 2.6.1
1. Kafkaesque Helm Chart 1.0.24
1. Probably has not bearing but exposeTopicLevelMetricsInPrometheus is set to false as we have a lot of topics
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frankjkelly commented on issue #10016: pulsar_ml_cache_used_size Prometheus metric is negative (Pulsar 2.6.1)
Posted by GitBox <gi...@apache.org>.
frankjkelly commented on issue #10016:
URL: https://github.com/apache/pulsar/issues/10016#issuecomment-819494682
Thanks @MarvinCai for the patch. For your last comment does this mean there should be an improvement ticket filed as I think understanding the size of this cache is super useful for performance/scaling decisions but I could be wrong. CC: @sijie @addisonj
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] merlimat closed issue #10016: pulsar_ml_cache_used_size Prometheus metric is negative (Pulsar 2.6.1)
Posted by GitBox <gi...@apache.org>.
merlimat closed issue #10016:
URL: https://github.com/apache/pulsar/issues/10016
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai edited a comment on issue #10016: pulsar_ml_cache_used_size Prometheus metric is negative (Pulsar 2.6.1)
Posted by GitBox <gi...@apache.org>.
MarvinCai edited a comment on issue #10016:
URL: https://github.com/apache/pulsar/issues/10016#issuecomment-819281424
The cache_used_size is get from [here](https://github.com/apache/pulsar/blob/f800fde9f27b2e14994f822b8ddac4d01583fb08/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryMBeanImpl.java#L74) which is stored in `EntryCacheManager` , the cacheSize is decrement [here](https://github.com/apache/pulsar/blob/f800fde9f27b2e14994f822b8ddac4d01583fb08/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCacheManager.java#L140), and the only place I find can possibly make it negative is [here](https://github.com/apache/pulsar/blob/f800fde9f27b2e14994f822b8ddac4d01583fb08/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/util/RangeCache.java#L238), when clearing a cache we're returning entry cache size before entries removal instead of the size of entries removed.
I'll have a patch for it.
But IMO this metrics is not very accurate as when it's calculated using Bytebuf.readableByte, and when we remove entry from cache after we consuming it, the readableByte will be smaller than before, which means we're not properly reducing the size of cached entries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai commented on issue #10016: pulsar_ml_cache_used_size Prometheus metric is negative (Pulsar 2.6.1)
Posted by GitBox <gi...@apache.org>.
MarvinCai commented on issue #10016:
URL: https://github.com/apache/pulsar/issues/10016#issuecomment-819281424
The cache_used_size is get from [here](https://github.com/apache/pulsar/blob/f800fde9f27b2e14994f822b8ddac4d01583fb08/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryMBeanImpl.java#L74) which is stored in `EntryCacheManager` , the cacheSize is decrement [here](https://github.com/apache/pulsar/blob/f800fde9f27b2e14994f822b8ddac4d01583fb08/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCacheManager.java#L140), and the only place I find can possibly make it negative is [here](https://github.com/apache/pulsar/blob/f800fde9f27b2e14994f822b8ddac4d01583fb08/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/util/RangeCache.java#L238), when clearing a cache we're not exactly returning entry cache size before entries removal instead of the size of entries removed.
I'll have a patch for it.
But IMO this metrics is not very accurate as when it's calculated using Bytebuf.readableByte, and when we remove entry from cache after we consuming it, the readableByte will be smaller than before, which means we're not properly reducing the size of cached entries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org