[GitHub] [pulsar] zzzming opened a new issue #10857: pulsar_topics_count inaccurate after broker restarts

[GitBox <gi...@apache.org>]

zzzming opened a new issue #10857:
URL: https://github.com/apache/pulsar/issues/10857


   **Describe the bug**
   A newly created topic's Prometheus metrics does not survive broker restarts. The topic is being created by pulsar_admin with no traffic nor subscription. `pulsar_topics_count` is not accurate after the broker restarts.
   
   **To Reproduce**
   1. Create a persistent topic with pulsar-admin
   2. Verify the topic's prometheus metrics exist under pulsar_in_bytes_total,  pulsar_consumers_count, and a few others. 
   3. `pulsar_topics_count` is also incremented by 1
   4. Restart the broker
   5. After the broker has restarted, all metrics can be queried successfully.
   6. However, the newly created topic's metrics are missing from the prometheus query. `pulsar_topics_count` is also reset to the number before the topic was created.
   
   **Expected behavior**
   The new created topic metrics survive the broker restarts.
   
   **Screenshots**
   n/a
   
   **Desktop (please complete the following information):**
   Tested on 2.7.2
   
   **Additional context**
   Add any other context about the problem here.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] codelipenghui commented on issue #10857: A newly created topic metrics do not survive after broker restarts

[GitBox <gi...@apache.org>]

codelipenghui commented on issue #10857:
URL: https://github.com/apache/pulsar/issues/10857#issuecomment-1058889613


   The issue had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] michaeljmarshall commented on issue #10857: A newly created topic metrics do not survive after broker restarts

[GitBox <gi...@apache.org>]

michaeljmarshall commented on issue #10857:
URL: https://github.com/apache/pulsar/issues/10857#issuecomment-862073310


   > can you confirm config `brokerDeleteInactiveTopicsEnabled` is set to false?
   
   I believe the behavior described is independent of that setting. Topic level metrics are stored in memory, so when a broker restarts, it doesn't have the old metrics. Additionally, the broker might not own that topic any more, and if the topic doesn't have any producers/consumers trying to connect to it, I don't think the topic will be loaded on any broker, which means that it won't be reported in the metrics. (This is my current understanding, but it could be incorrect. I'd have to look at these metrics again to double check.)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] MarvinCai commented on issue #10857: A newly created topic metrics do not survive after broker restarts

[GitBox <gi...@apache.org>]

MarvinCai commented on issue #10857:
URL: https://github.com/apache/pulsar/issues/10857#issuecomment-862035760


   Hi, can you confirm config `brokerDeleteInactiveTopicsEnabled` is set to false?
   If it's true, inactive topic with no active traffic will be garbage collected by broker.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] michaeljmarshall commented on issue #10857: A newly created topic metrics do not survive after broker restarts

[GitBox <gi...@apache.org>]

michaeljmarshall commented on issue #10857:
URL: https://github.com/apache/pulsar/issues/10857#issuecomment-862080194


   I wonder if perhaps part of the problem here is that the `pulsar_topics_count` metric actually represents the number of topics loaded, not the total number of topics. I'd need to double check the implementation to be sure, but that could lead to the confusion seen here. If my understanding is correct, I see two options available:
   
   1. Introduce a new gauge metric that reports the total number of topics (not just the loaded ones). Then, only report this metric from the lead broker.
   2. Introduce the same metric as in 1, but instead of reporting it from the lead broker, report it from each bundle.
   
   In each solution, it'd be important to decrement the gauge as bundles are moved among brokers to ensure that topics are not double counted. I am pretty sure that a bundle doesn't have to be loaded if none of its topics are loaded, so that could lead to similar problems raised in this issue. Either way, the existing metric has a bit of a misleading name.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org