You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/02/19 14:16:38 UTC
[GitHub] [pulsar] frankjkelly opened a new issue #9634: bookie_journal_JOURNAL_QUEUE_SIZE Prometheus metric is negative (Pulsar 2.6.1)
frankjkelly opened a new issue #9634:
URL: https://github.com/apache/pulsar/issues/9634
**Describe the bug**
The prometheus metric `bookie_journal_JOURNAL_QUEUE_SIZE` exposed by bookie is negative. Shouldn't queue lengths be positive
**To Reproduce**
1. Launch Pulsar
2. Inspect metrics endpoint of Bookkeeper
```
$ curl http://platform-pulsar-bookkeeper:8000/metrics | grep QUEUE_SIZE
# TYPE bookie_journal_JOURNAL_CB_QUEUE_SIZE counter
bookie_journal_JOURNAL_CB_QUEUE_SIZE 57253790
# TYPE bookie_journal_JOURNAL_FORCE_WRITE_QUEUE_SIZE counter
bookie_journal_JOURNAL_FORCE_WRITE_QUEUE_SIZE 0
# TYPE bookie_journal_JOURNAL_QUEUE_SIZE counter
bookie_journal_JOURNAL_QUEUE_SIZE -57253790
```
**Expected behavior**
Queue lengths are zero or above
**Screenshots**
**Additional context**
- Pulsar `2.6.1`
- Kafkaesque Helm Chart `1.0.24`
- Probably has not bearing but `exposeTopicLevelMetricsInPrometheus` is set to `false` as we have a lot of topics
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frankjkelly commented on issue #9634: bookie_journal_JOURNAL_QUEUE_SIZE Prometheus metric is negative (Pulsar 2.6.1)
Posted by GitBox <gi...@apache.org>.
frankjkelly commented on issue #9634:
URL: https://github.com/apache/pulsar/issues/9634#issuecomment-784200556
@michaeljmarshall Yes only `bookie_journal_JOURNAL_QUEUE_SIZE` is negative - see below where `bookie_journal_JOURNAL_CB_QUEUE_SIZE` is positive (and almost the exact inverse value)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on issue #9634: bookie_journal_JOURNAL_QUEUE_SIZE Prometheus metric is negative (Pulsar 2.6.1)
Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on issue #9634:
URL: https://github.com/apache/pulsar/issues/9634#issuecomment-784277303
> bookie_journal_JOURNAL_CB_QUEUE_SIZE is positive (and almost the exact inverse value)
That's a helpful observation. When I was looking through the code last night, I was wondering if there was any type of relationship between those two metrics.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frankjkelly closed issue #9634: bookie_journal_JOURNAL_QUEUE_SIZE Prometheus metric is negative (Pulsar 2.6.1)
Posted by GitBox <gi...@apache.org>.
frankjkelly closed issue #9634:
URL: https://github.com/apache/pulsar/issues/9634
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on issue #9634: bookie_journal_JOURNAL_QUEUE_SIZE Prometheus metric is negative (Pulsar 2.6.1)
Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on issue #9634:
URL: https://github.com/apache/pulsar/issues/9634#issuecomment-783950066
I found some interesting things that I wanted to share, so I opened a copy of this issue in bookkeeper. I think we should continue all work against this issue there.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frankjkelly commented on issue #9634: bookie_journal_JOURNAL_QUEUE_SIZE Prometheus metric is negative (Pulsar 2.6.1)
Posted by GitBox <gi...@apache.org>.
frankjkelly commented on issue #9634:
URL: https://github.com/apache/pulsar/issues/9634#issuecomment-785084465
@michaeljmarshall Thanks so much for hunting this down. Closing this ticket . . . .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall edited a comment on issue #9634: bookie_journal_JOURNAL_QUEUE_SIZE Prometheus metric is negative (Pulsar 2.6.1)
Posted by GitBox <gi...@apache.org>.
michaeljmarshall edited a comment on issue #9634:
URL: https://github.com/apache/pulsar/issues/9634#issuecomment-784753496
@frankjkelly - I believe you're running with apache bookkeeper `4.10.0` (that is what shipped in the pulsar docker image for pulsar `2.6.1`). This bug was fixed for bookkeeper `4.10.1`, as seen in this PR: https://github.com/apache/bookkeeper/pull/2290. Pulsar `2.6.2` is `4.10.0` too, but Pulsar `2.6.3` comes with bookkeeper `4.11.1`, which has the fix to the metric.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on issue #9634: bookie_journal_JOURNAL_QUEUE_SIZE Prometheus metric is negative (Pulsar 2.6.1)
Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on issue #9634:
URL: https://github.com/apache/pulsar/issues/9634#issuecomment-783938100
I started looking into this tonight. I believe this issue belongs in the [apache/bookkeeper](https://github.com/apache/bookkeeper) repo because the metric is defined there.
Regarding the problematic metric, I already see some irregularities based on parameter names and things getting passed around.
@frankjkelly can you confirm that your only seeing this negative value for `bookie_journal_JOURNAL_QUEUE_SIZE`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on issue #9634: bookie_journal_JOURNAL_QUEUE_SIZE Prometheus metric is negative (Pulsar 2.6.1)
Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on issue #9634:
URL: https://github.com/apache/pulsar/issues/9634#issuecomment-784753496
@frankjkelly - I believe you're running with apache bookkeeper `4.10.0` (that is what shipped in the pulsar docker image for pulsar `2.6.1`. This bug was fixed for bookkeeper `4.10.1`, as seen in this PR: https://github.com/apache/bookkeeper/pull/2290. Pulsar `2.6.2` is `4.10.0` too, but Pulsar `2.6.3` comes with bookkeeper `4.11.1`, which has the fix to the metric.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frankjkelly edited a comment on issue #9634: bookie_journal_JOURNAL_QUEUE_SIZE Prometheus metric is negative (Pulsar 2.6.1)
Posted by GitBox <gi...@apache.org>.
frankjkelly edited a comment on issue #9634:
URL: https://github.com/apache/pulsar/issues/9634#issuecomment-784200556
@michaeljmarshall Yes only `bookie_journal_JOURNAL_QUEUE_SIZE` is negative - see below where `bookie_journal_JOURNAL_CB_QUEUE_SIZE` is positive (and almost the exact inverse value)
I also see `bookie_journal_JOURNAL_FORCE_WRITE_QUEUE_SIZE` as zeroes only
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org