You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/05/27 23:54:29 UTC
[GitHub] [pulsar] ahothan opened a new issue, #15819: pulsar_out_bytes_total and pulsar_out_messages_total metrics act as gauges instead of counters
ahothan opened a new issue, #15819:
URL: https://github.com/apache/pulsar/issues/15819
**Describe the bug**
pulsar_out_bytes_total and pulsar_out_messages_total metrics should normally be counters (always increasing) in the same way as pulsar_in_bytes_total and pulsar_in_messages_total metrics.
However they behave incorrectly as they go up and down.
Furthermore, when producer is idle, these 2 metrics are no longer reporting when pulsar is scraped.
**To Reproduce**
Steps to reproduce the behavior:
Create a topic, produce and consume from that topic for a long period of time.
Then check how these 2 counters evolve over time
**Expected behavior**
These 2 counters should continuously increase.
**Screenshots**
See attached grafana dashboard
**Desktop (please complete the following information):**
- OS: Linux/Kubernetes
- deployed with sn-platform and pulsar operator
streamnative/sn-platform:2.9.2.17
**Additional context**
See
- how the out metric goes up and down while the in metric behaves properly
- how the out metric stops reporting values when producer is idle while the in metric keeps reporting the last value as expected
<img width="1062" alt="Screen Shot 2022-05-27 at 4 08 34 PM" src="https://user-images.githubusercontent.com/4954645/170801095-076287b3-4487-474f-a5a8-6176c033ddd6.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on issue #15819: pulsar_out_bytes_total and pulsar_out_messages_total metrics act as gauges instead of counters
Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on issue #15819:
URL: https://github.com/apache/pulsar/issues/15819#issuecomment-1312074756
That's a very helpful observation @pgier. @dlg99 - any chance you're able to look into this again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #15819: pulsar_out_bytes_total and pulsar_out_messages_total metrics act as gauges instead of counters
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #15819:
URL: https://github.com/apache/pulsar/issues/15819#issuecomment-1166752883
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] pgier commented on issue #15819: pulsar_out_bytes_total and pulsar_out_messages_total metrics act as gauges instead of counters
Posted by GitBox <gi...@apache.org>.
pgier commented on issue #15819:
URL: https://github.com/apache/pulsar/issues/15819#issuecomment-1311967137
I'm not sure if this is exactly the same issue, but what I'm seeing is when I disconnect from a subscription, the message/byte count for that subscription goes back down to zero.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ahothan commented on issue #15819: pulsar_out_bytes_total and pulsar_out_messages_total metrics act as gauges instead of counters
Posted by GitBox <gi...@apache.org>.
ahothan commented on issue #15819:
URL: https://github.com/apache/pulsar/issues/15819#issuecomment-1313932506
@pgier thanks for the RCA and for fixing this!
> The issue seems to be that `pulsar_out_bytes_total` and `pulsar_out_messages_total` are calculated by adding up the values from the currently connected consumers, instead of just using the subscription stats directly. I'm not sure why it was done this way, but I created #18451 to hopefully address this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] pgier commented on issue #15819: pulsar_out_bytes_total and pulsar_out_messages_total metrics act as gauges instead of counters
Posted by GitBox <gi...@apache.org>.
pgier commented on issue #15819:
URL: https://github.com/apache/pulsar/issues/15819#issuecomment-1312039693
@michaeljmarshall I mean when the consumer disconnects. So, for example ctrl-c using pulsar-client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ahothan commented on issue #15819: pulsar_out_bytes_total and pulsar_out_messages_total metrics act as gauges instead of counters
Posted by GitBox <gi...@apache.org>.
ahothan commented on issue #15819:
URL: https://github.com/apache/pulsar/issues/15819#issuecomment-1140109546
corresponding stats from the cli shows correct values for msgOutCounter and msgInCounter,
```
root@pulsar-sn-platform-toolset-0:/pulsar# pulsar-admin topics stats benchmark/ns-cluster/pperf-test0-partition-0
{
"msgRateIn" : 5000.481725033992,
"msgThroughputIn" : 5206017.593981283,
"msgRateOut" : 5000.8455656363685,
"msgThroughputOut" : 5208596.899526573,
"bytesInCounter" : 6463689301,
"msgInCounter" : 6208551,
"bytesOutCounter" : 6466197060,
"msgOutCounter" : 6208417,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on issue #15819: pulsar_out_bytes_total and pulsar_out_messages_total metrics act as gauges instead of counters
Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on issue #15819:
URL: https://github.com/apache/pulsar/issues/15819#issuecomment-1311987751
@pgier - a similar issue was fixed here https://github.com/apache/pulsar/pull/10644. Can you clarify what you mean by "disconnect from a subscription"? That'll help me with looking into the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] pgier commented on issue #15819: pulsar_out_bytes_total and pulsar_out_messages_total metrics act as gauges instead of counters
Posted by GitBox <gi...@apache.org>.
pgier commented on issue #15819:
URL: https://github.com/apache/pulsar/issues/15819#issuecomment-1312793025
The issue seems to be that pulsar_out_bytes_total and pulsar_out_messages_total are calculated by adding up the values from the currently connected consumers, instead of just using the subscription stats directly. I'm not sure why it was done this way, but I created #18451 to hopefully address this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ahothan commented on issue #15819: pulsar_out_bytes_total and pulsar_out_messages_total metrics act as gauges instead of counters
Posted by GitBox <gi...@apache.org>.
ahothan commented on issue #15819:
URL: https://github.com/apache/pulsar/issues/15819#issuecomment-1140108986
related PR: https://github.com/apache/pulsar/pull/6918
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall closed issue #15819: pulsar_out_bytes_total and pulsar_out_messages_total metrics act as gauges instead of counters
Posted by GitBox <gi...@apache.org>.
michaeljmarshall closed issue #15819: pulsar_out_bytes_total and pulsar_out_messages_total metrics act as gauges instead of counters
URL: https://github.com/apache/pulsar/issues/15819
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org