You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2021/05/12 17:54:13 UTC

[GitHub] [kafka] dielhennr opened a new pull request #10679: KAFKA-12697: Add Global Topic and Partition count metrics to the Quorum Controller

dielhennr opened a new pull request #10679:
URL: https://github.com/apache/kafka/pull/10679


   Added GlobalTopicCount and GlobalPartitionCount metrics to the QuorumControllerMetrics.
   
   The metrics are calculated by counting records as they are replayed e.g. replay(TopicRecord), replay(RemoveTopicRecord)
   
   This was unit tested using MockControllerMetrics.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] cmccabe commented on a change in pull request #10679: KAFKA-12697: Add Global Topic and Partition count metrics to the Quorum Controller

Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10679:
URL: https://github.com/apache/kafka/pull/10679#discussion_r631802943



##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumControllerMetrics.java
##########
@@ -30,14 +30,24 @@
         "kafka.controller", "ControllerEventManager", "EventQueueTimeMs", null);
     private final static MetricName EVENT_QUEUE_PROCESSING_TIME_MS = new MetricName(
         "kafka.controller", "ControllerEventManager", "EventQueueProcessingTimeMs", null);
+    private final static MetricName GLOBAL_TOPIC_COUNT = new MetricName(
+        "kafka.controller", "ReplicationControlManager", "GlobalTopicCount", null);
+    private final static MetricName GLOBAL_PARTITION_COUNT = new MetricName(

Review comment:
       For compatibility, the mbean needs to be `kafka.controller:type=KafkaController,name=GlobalPartitionCount`.  That means you need to pass "KafkaController" here instead of "ReplicationControlManager".




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] cmccabe merged pull request #10679: KAFKA-12697: Add Global Topic and Partition count metrics to the Quorum Controller

Posted by GitBox <gi...@apache.org>.
cmccabe merged pull request #10679:
URL: https://github.com/apache/kafka/pull/10679


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] cmccabe commented on a change in pull request #10679: KAFKA-12697: Add Global Topic and Partition count metrics to the Quorum Controller

Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10679:
URL: https://github.com/apache/kafka/pull/10679#discussion_r631804785



##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumControllerMetrics.java
##########
@@ -30,14 +30,24 @@
         "kafka.controller", "ControllerEventManager", "EventQueueTimeMs", null);
     private final static MetricName EVENT_QUEUE_PROCESSING_TIME_MS = new MetricName(
         "kafka.controller", "ControllerEventManager", "EventQueueProcessingTimeMs", null);
+    private final static MetricName GLOBAL_TOPIC_COUNT = new MetricName(
+        "kafka.controller", "ReplicationControlManager", "GlobalTopicCount", null);
+    private final static MetricName GLOBAL_PARTITION_COUNT = new MetricName(
+        "kafka.controller", "ReplicationControlManager", "GlobalPartitionCount", null);
 
     private volatile boolean active;
+    private volatile int topics;

Review comment:
       I think it would be preferrable to call this "globalTopicCount" and call the gauge "globalTopicCountGauge" or something similar.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] cmccabe commented on a change in pull request #10679: KAFKA-12697: Add Global Topic and Partition count metrics to the Quorum Controller

Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10679:
URL: https://github.com/apache/kafka/pull/10679#discussion_r631802796



##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumControllerMetrics.java
##########
@@ -30,14 +30,24 @@
         "kafka.controller", "ControllerEventManager", "EventQueueTimeMs", null);
     private final static MetricName EVENT_QUEUE_PROCESSING_TIME_MS = new MetricName(
         "kafka.controller", "ControllerEventManager", "EventQueueProcessingTimeMs", null);
+    private final static MetricName GLOBAL_TOPIC_COUNT = new MetricName(
+        "kafka.controller", "ReplicationControlManager", "GlobalTopicCount", null);

Review comment:
       For compatibility, the mbean needs to be `kafka.controller:type=KafkaController,name=GlobalTopicCount`.  That means you need to pass "KafkaController" here instead of "ReplicationControlManager".




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] cmccabe commented on a change in pull request #10679: KAFKA-12697: Add Global Topic and Partition count metrics to the Quorum Controller

Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10679:
URL: https://github.com/apache/kafka/pull/10679#discussion_r631806532



##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -279,6 +279,8 @@ public String toString() {
      */
     private final int defaultNumPartitions;
 
+    private int globalPartitionCount;

Review comment:
       Can you add a JavaDoc description here?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] cmccabe commented on a change in pull request #10679: KAFKA-12697: Add Global Topic and Partition count metrics to the Quorum Controller

Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10679:
URL: https://github.com/apache/kafka/pull/10679#discussion_r631804888



##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumControllerMetrics.java
##########
@@ -30,14 +30,24 @@
         "kafka.controller", "ControllerEventManager", "EventQueueTimeMs", null);
     private final static MetricName EVENT_QUEUE_PROCESSING_TIME_MS = new MetricName(
         "kafka.controller", "ControllerEventManager", "EventQueueProcessingTimeMs", null);
+    private final static MetricName GLOBAL_TOPIC_COUNT = new MetricName(
+        "kafka.controller", "ReplicationControlManager", "GlobalTopicCount", null);
+    private final static MetricName GLOBAL_PARTITION_COUNT = new MetricName(
+        "kafka.controller", "ReplicationControlManager", "GlobalPartitionCount", null);
 
     private volatile boolean active;
+    private volatile int topics;
+    private volatile int partitions;

Review comment:
       I think it would be preferrable to call this "globalPartitionCount" and call the gauge "globalPartitionCountGauge" or something similar.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] cmccabe commented on a change in pull request #10679: KAFKA-12697: Add Global Topic and Partition count metrics to the Quorum Controller

Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10679:
URL: https://github.com/apache/kafka/pull/10679#discussion_r631804785



##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumControllerMetrics.java
##########
@@ -30,14 +30,24 @@
         "kafka.controller", "ControllerEventManager", "EventQueueTimeMs", null);
     private final static MetricName EVENT_QUEUE_PROCESSING_TIME_MS = new MetricName(
         "kafka.controller", "ControllerEventManager", "EventQueueProcessingTimeMs", null);
+    private final static MetricName GLOBAL_TOPIC_COUNT = new MetricName(
+        "kafka.controller", "ReplicationControlManager", "GlobalTopicCount", null);
+    private final static MetricName GLOBAL_PARTITION_COUNT = new MetricName(
+        "kafka.controller", "ReplicationControlManager", "GlobalPartitionCount", null);
 
     private volatile boolean active;
+    private volatile int topics;

Review comment:
       I think it would be preferrable to call this "globalPartitionCount" and call the gauge "globalPartitionCountGauge" or something similar.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] cmccabe commented on a change in pull request #10679: KAFKA-12697: Add Global Topic and Partition count metrics to the Quorum Controller

Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10679:
URL: https://github.com/apache/kafka/pull/10679#discussion_r631807221



##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -343,12 +354,14 @@ public void replay(PartitionRecord record) {
             topicInfo.parts.put(record.partitionId(), newPartInfo);
             brokersToIsrs.update(record.topicId(), record.partitionId(), null,
                 newPartInfo.isr, NO_LEADER, newPartInfo.leader);
+            globalPartitionCount++;
         } else if (!newPartInfo.equals(prevPartInfo)) {
             newPartInfo.maybeLogPartitionChange(log, description, prevPartInfo);
             topicInfo.parts.put(record.partitionId(), newPartInfo);
             brokersToIsrs.update(record.topicId(), record.partitionId(), prevPartInfo.isr,
                 newPartInfo.isr, prevPartInfo.leader, newPartInfo.leader);
         }
+        controllerMetrics.setGlobalPartitionCount(globalPartitionCount);

Review comment:
       We only need to do this in the branch of the "if" where we  change `globalPartitionCount`, right?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org