You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2021/07/26 19:14:23 UTC
[GitHub] [kafka] rondagostino opened a new pull request #11131: KAFKA-13137: KRaft Controller Metric MBean names incorrectly quoted
rondagostino opened a new pull request #11131:
URL: https://github.com/apache/kafka/pull/11131
Controller metric names that are in common between the ZooKeeper-based and KRaft-based controller must remain the same, but they were not in the AK 2.8 early access release of KRaft. For example, the non-KRaft MBean name `kafka.controller:type=KafkaController,name=OfflinePartitionsCount` incorrectly became `"kafka.controller":type="KafkaController",name="OfflinePartitionsCount"` (note the added quotes). This patch fixes the issue and closes the test gap that allowed the divergence to occur.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hachikuji commented on a change in pull request #11131: KAFKA-13137: KRaft Controller Metric MBean names incorrectly quoted
Posted by GitBox <gi...@apache.org>.
hachikuji commented on a change in pull request #11131:
URL: https://github.com/apache/kafka/pull/11131#discussion_r676942838
##########
File path: metadata/src/test/java/org/apache/kafka/controller/QuorumControllerMetricsTest.java
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import com.yammer.metrics.core.MetricsRegistry;
+import org.junit.jupiter.api.Test;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class QuorumControllerMetricsTest {
+ @Test
+ public void testKafkaControllerMetricNames() {
+ String expectedGroup = "kafka.controller";
+ String expectedType = "KafkaController";
+ Set<String> expectedMetricNames = new HashSet<>(Arrays.asList(
+ "ActiveControllerCount",
+ "GlobalTopicCount",
+ "GlobalPartitionCount",
+ "OfflinePartitionCount",
+ "PreferredReplicaImbalanceCount"));
+ Set<String> missingMetrics = getMissingMetricNames(expectedMetricNames, expectedGroup, expectedType);
+ assertEquals(Collections.emptySet(), missingMetrics, "Expected metrics did not exist");
+ }
+
+ @Test
+ public void testControllerEventManagerMetricNames() {
+ String expectedGroup = "kafka.controller";
+ String expectedType = "ControllerEventManager";
+ Set<String> expectedMetricNames = new HashSet<>(Arrays.asList(
+ "EventQueueTimeMs",
+ "EventQueueProcessingTimeMs"));
+ Set<String> missingMetrics = getMissingMetricNames(expectedMetricNames, expectedGroup, expectedType);
+ assertEquals(Collections.emptySet(), missingMetrics, "Expected metrics did not exist");
+ }
+
+ private static Set<String> getMissingMetricNames(
+ Set<String> expectedMetricNames, String expectedGroup, String expectedType) {
+ MetricsRegistry registry = new MetricsRegistry();
+ new QuorumControllerMetrics(registry); // populates the registry
+ Set<String> foundMetricNames = expectedMetricNames.stream().filter(expectedMetricName ->
+ registry.allMetrics().keySet().stream().anyMatch(metricName -> {
+ if (metricName.getGroup().equals(expectedGroup) && metricName.getType().equals(expectedType)
+ && metricName.getScope() == null && metricName.getName().equals(expectedMetricName)) {
+ // It has to exist AND the MBean name has to be correct;
+ // fail right here if the MBean name doesn't match
+ String expectedMBeanPrefix = expectedGroup + ":type=" + expectedType + ",name=";
+ assertEquals(expectedMBeanPrefix + expectedMetricName, metricName.getMBeanName());
+ return true; // the expected metric name exists and the associated MBean name matches
+ } else {
+ return false; // this one didn't match
+ }
+ })).collect(Collectors.toSet());
+ if (foundMetricNames.size() == expectedMetricNames.size()) {
Review comment:
nit: is the size check sufficient? Any harm skipping this check and following the other branch in all cases?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumControllerMetrics.java
##########
@@ -22,22 +22,21 @@
import com.yammer.metrics.core.MetricName;
import com.yammer.metrics.core.MetricsRegistry;
-
public final class QuorumControllerMetrics implements ControllerMetrics {
- private final static MetricName ACTIVE_CONTROLLER_COUNT = new MetricName(
- "kafka.controller", "KafkaController", "ActiveControllerCount", null);
- private final static MetricName EVENT_QUEUE_TIME_MS = new MetricName(
- "kafka.controller", "ControllerEventManager", "EventQueueTimeMs", null);
- private final static MetricName EVENT_QUEUE_PROCESSING_TIME_MS = new MetricName(
- "kafka.controller", "ControllerEventManager", "EventQueueProcessingTimeMs", null);
- private final static MetricName GLOBAL_TOPIC_COUNT = new MetricName(
- "kafka.controller", "KafkaController", "GlobalTopicCount", null);
- private final static MetricName GLOBAL_PARTITION_COUNT = new MetricName(
- "kafka.controller", "KafkaController", "GlobalPartitionCount", null);
- private final static MetricName OFFLINE_PARTITION_COUNT = new MetricName(
- "kafka.controller", "KafkaController", "OfflinePartitionCount", null);
- private final static MetricName PREFERRED_REPLICA_IMBALANCE_COUNT = new MetricName(
- "kafka.controller", "KafkaController", "PreferredReplicaImbalanceCount", null);
+ private final static MetricName ACTIVE_CONTROLLER_COUNT = getMetricName(
+ "kafka.controller", "KafkaController", "ActiveControllerCount");
+ private final static MetricName EVENT_QUEUE_TIME_MS = getMetricName(
+ "kafka.controller", "ControllerEventManager", "EventQueueTimeMs");
+ private final static MetricName EVENT_QUEUE_PROCESSING_TIME_MS = getMetricName(
+ "kafka.controller", "ControllerEventManager", "EventQueueProcessingTimeMs");
+ private final static MetricName GLOBAL_TOPIC_COUNT = getMetricName(
+ "kafka.controller", "KafkaController", "GlobalTopicCount");
+ private final static MetricName GLOBAL_PARTITION_COUNT = getMetricName(
+ "kafka.controller", "KafkaController", "GlobalPartitionCount");
Review comment:
nit: since the context is already controller metrics, do we need to pass through `kafka.controller` or could this be implicit?
##########
File path: metadata/src/test/java/org/apache/kafka/controller/QuorumControllerMetricsTest.java
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import com.yammer.metrics.core.MetricsRegistry;
+import org.junit.jupiter.api.Test;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class QuorumControllerMetricsTest {
+ @Test
+ public void testKafkaControllerMetricNames() {
+ String expectedGroup = "kafka.controller";
+ String expectedType = "KafkaController";
+ Set<String> expectedMetricNames = new HashSet<>(Arrays.asList(
Review comment:
nit: could use Utils.mkSet
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumControllerMetrics.java
##########
@@ -22,22 +22,21 @@
import com.yammer.metrics.core.MetricName;
import com.yammer.metrics.core.MetricsRegistry;
-
public final class QuorumControllerMetrics implements ControllerMetrics {
- private final static MetricName ACTIVE_CONTROLLER_COUNT = new MetricName(
- "kafka.controller", "KafkaController", "ActiveControllerCount", null);
- private final static MetricName EVENT_QUEUE_TIME_MS = new MetricName(
- "kafka.controller", "ControllerEventManager", "EventQueueTimeMs", null);
- private final static MetricName EVENT_QUEUE_PROCESSING_TIME_MS = new MetricName(
- "kafka.controller", "ControllerEventManager", "EventQueueProcessingTimeMs", null);
- private final static MetricName GLOBAL_TOPIC_COUNT = new MetricName(
- "kafka.controller", "KafkaController", "GlobalTopicCount", null);
- private final static MetricName GLOBAL_PARTITION_COUNT = new MetricName(
- "kafka.controller", "KafkaController", "GlobalPartitionCount", null);
- private final static MetricName OFFLINE_PARTITION_COUNT = new MetricName(
- "kafka.controller", "KafkaController", "OfflinePartitionCount", null);
- private final static MetricName PREFERRED_REPLICA_IMBALANCE_COUNT = new MetricName(
- "kafka.controller", "KafkaController", "PreferredReplicaImbalanceCount", null);
+ private final static MetricName ACTIVE_CONTROLLER_COUNT = getMetricName(
+ "kafka.controller", "KafkaController", "ActiveControllerCount");
Review comment:
Can we introduce some constants for the metric names? They could be used in the test case as well as de-registration (which is a gap at the moment).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on pull request #11131: KAFKA-13137: KRaft Controller Metric MBean names incorrectly quoted
Posted by GitBox <gi...@apache.org>.
rondagostino commented on pull request #11131:
URL: https://github.com/apache/kafka/pull/11131#issuecomment-887525250
Thanks for the reviews @hachikuji and @showuon. The test is revamped so that the method is `private static void assertExpectedMetrics(Set<String> expectedMetricNames, String expectedType)` as suggested, and all of the assertion messages now make sense if there is a failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hachikuji commented on pull request #11131: KAFKA-13137: KRaft Controller Metric MBean names incorrectly quoted
Posted by GitBox <gi...@apache.org>.
hachikuji commented on pull request #11131:
URL: https://github.com/apache/kafka/pull/11131#issuecomment-887924795
@rondagostino I've seen this test failing consistently: `kafka.controller.ControllerEventManagerTest.testMetricsCleanedOnClose()`. I tried it locally and it passes when run by itself. I have a suspicion that now that the jmx names have been fixed, there is kind of registration conflict. Perhaps the new tests or some other test which involves the KRaft controller is leaving behind the mbeans. As we noted above, the de-registration logic has not been implemented for these metrics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on a change in pull request #11131: KAFKA-13137: KRaft Controller Metric MBean names incorrectly quoted
Posted by GitBox <gi...@apache.org>.
rondagostino commented on a change in pull request #11131:
URL: https://github.com/apache/kafka/pull/11131#discussion_r676982713
##########
File path: metadata/src/test/java/org/apache/kafka/controller/QuorumControllerMetricsTest.java
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import com.yammer.metrics.core.MetricsRegistry;
+import org.junit.jupiter.api.Test;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class QuorumControllerMetricsTest {
+ @Test
+ public void testKafkaControllerMetricNames() {
+ String expectedGroup = "kafka.controller";
+ String expectedType = "KafkaController";
+ Set<String> expectedMetricNames = new HashSet<>(Arrays.asList(
+ "ActiveControllerCount",
+ "GlobalTopicCount",
+ "GlobalPartitionCount",
+ "OfflinePartitionCount",
+ "PreferredReplicaImbalanceCount"));
+ Set<String> missingMetrics = getMissingMetricNames(expectedMetricNames, expectedGroup, expectedType);
+ assertEquals(Collections.emptySet(), missingMetrics, "Expected metrics did not exist");
+ }
+
+ @Test
+ public void testControllerEventManagerMetricNames() {
+ String expectedGroup = "kafka.controller";
+ String expectedType = "ControllerEventManager";
+ Set<String> expectedMetricNames = new HashSet<>(Arrays.asList(
+ "EventQueueTimeMs",
+ "EventQueueProcessingTimeMs"));
+ Set<String> missingMetrics = getMissingMetricNames(expectedMetricNames, expectedGroup, expectedType);
+ assertEquals(Collections.emptySet(), missingMetrics, "Expected metrics did not exist");
+ }
+
+ private static Set<String> getMissingMetricNames(
+ Set<String> expectedMetricNames, String expectedGroup, String expectedType) {
+ MetricsRegistry registry = new MetricsRegistry();
+ new QuorumControllerMetrics(registry); // populates the registry
+ Set<String> foundMetricNames = expectedMetricNames.stream().filter(expectedMetricName ->
+ registry.allMetrics().keySet().stream().anyMatch(metricName -> {
+ if (metricName.getGroup().equals(expectedGroup) && metricName.getType().equals(expectedType)
+ && metricName.getScope() == null && metricName.getName().equals(expectedMetricName)) {
+ // It has to exist AND the MBean name has to be correct;
+ // fail right here if the MBean name doesn't match
+ String expectedMBeanPrefix = expectedGroup + ":type=" + expectedType + ",name=";
+ assertEquals(expectedMBeanPrefix + expectedMetricName, metricName.getMBeanName());
+ return true; // the expected metric name exists and the associated MBean name matches
+ } else {
+ return false; // this one didn't match
+ }
+ })).collect(Collectors.toSet());
+ if (foundMetricNames.size() == expectedMetricNames.size()) {
Review comment:
The size check is sufficient, but there's really no significant harm in following the other branch in all cases (just unnecessarily copying a small set), so I'll go with the cleaner single solution as you suggest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on pull request #11131: KAFKA-13137: KRaft Controller Metric MBean names incorrectly quoted
Posted by GitBox <gi...@apache.org>.
rondagostino commented on pull request #11131:
URL: https://github.com/apache/kafka/pull/11131#issuecomment-888509504
@hachikuji I added the deregistration logic; hopefully it looks good and resolves the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on a change in pull request #11131: KAFKA-13137: KRaft Controller Metric MBean names incorrectly quoted
Posted by GitBox <gi...@apache.org>.
rondagostino commented on a change in pull request #11131:
URL: https://github.com/apache/kafka/pull/11131#discussion_r676985024
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumControllerMetrics.java
##########
@@ -22,22 +22,21 @@
import com.yammer.metrics.core.MetricName;
import com.yammer.metrics.core.MetricsRegistry;
-
public final class QuorumControllerMetrics implements ControllerMetrics {
- private final static MetricName ACTIVE_CONTROLLER_COUNT = new MetricName(
- "kafka.controller", "KafkaController", "ActiveControllerCount", null);
- private final static MetricName EVENT_QUEUE_TIME_MS = new MetricName(
- "kafka.controller", "ControllerEventManager", "EventQueueTimeMs", null);
- private final static MetricName EVENT_QUEUE_PROCESSING_TIME_MS = new MetricName(
- "kafka.controller", "ControllerEventManager", "EventQueueProcessingTimeMs", null);
- private final static MetricName GLOBAL_TOPIC_COUNT = new MetricName(
- "kafka.controller", "KafkaController", "GlobalTopicCount", null);
- private final static MetricName GLOBAL_PARTITION_COUNT = new MetricName(
- "kafka.controller", "KafkaController", "GlobalPartitionCount", null);
- private final static MetricName OFFLINE_PARTITION_COUNT = new MetricName(
- "kafka.controller", "KafkaController", "OfflinePartitionCount", null);
- private final static MetricName PREFERRED_REPLICA_IMBALANCE_COUNT = new MetricName(
- "kafka.controller", "KafkaController", "PreferredReplicaImbalanceCount", null);
+ private final static MetricName ACTIVE_CONTROLLER_COUNT = getMetricName(
+ "kafka.controller", "KafkaController", "ActiveControllerCount");
Review comment:
I'm wondering if using constants in the test case would be counter-productive -- the test would still succeed if the constant were to change. It feels to me that hard-coding the expected string inside the test case as is done now is the safest thing. Having a constant for de-registration makes sense, but maybe we should leave the rafactor for later if we aren't going to use the constant in multiple places right now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hachikuji commented on a change in pull request #11131: KAFKA-13137: KRaft Controller Metric MBean names incorrectly quoted
Posted by GitBox <gi...@apache.org>.
hachikuji commented on a change in pull request #11131:
URL: https://github.com/apache/kafka/pull/11131#discussion_r677033977
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumControllerMetrics.java
##########
@@ -22,22 +22,21 @@
import com.yammer.metrics.core.MetricName;
import com.yammer.metrics.core.MetricsRegistry;
-
public final class QuorumControllerMetrics implements ControllerMetrics {
- private final static MetricName ACTIVE_CONTROLLER_COUNT = new MetricName(
- "kafka.controller", "KafkaController", "ActiveControllerCount", null);
- private final static MetricName EVENT_QUEUE_TIME_MS = new MetricName(
- "kafka.controller", "ControllerEventManager", "EventQueueTimeMs", null);
- private final static MetricName EVENT_QUEUE_PROCESSING_TIME_MS = new MetricName(
- "kafka.controller", "ControllerEventManager", "EventQueueProcessingTimeMs", null);
- private final static MetricName GLOBAL_TOPIC_COUNT = new MetricName(
- "kafka.controller", "KafkaController", "GlobalTopicCount", null);
- private final static MetricName GLOBAL_PARTITION_COUNT = new MetricName(
- "kafka.controller", "KafkaController", "GlobalPartitionCount", null);
- private final static MetricName OFFLINE_PARTITION_COUNT = new MetricName(
- "kafka.controller", "KafkaController", "OfflinePartitionCount", null);
- private final static MetricName PREFERRED_REPLICA_IMBALANCE_COUNT = new MetricName(
- "kafka.controller", "KafkaController", "PreferredReplicaImbalanceCount", null);
+ private final static MetricName ACTIVE_CONTROLLER_COUNT = getMetricName(
+ "kafka.controller", "KafkaController", "ActiveControllerCount");
Review comment:
I don't feel too strongly about it. I guess rewriting the name might prevent some accidental modification case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on a change in pull request #11131: KAFKA-13137: KRaft Controller Metric MBean names incorrectly quoted
Posted by GitBox <gi...@apache.org>.
showuon commented on a change in pull request #11131:
URL: https://github.com/apache/kafka/pull/11131#discussion_r677159652
##########
File path: metadata/src/test/java/org/apache/kafka/controller/QuorumControllerMetricsTest.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import com.yammer.metrics.core.MetricsRegistry;
+import org.apache.kafka.common.utils.Utils;
+import org.junit.jupiter.api.Test;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class QuorumControllerMetricsTest {
+ @Test
+ public void testKafkaControllerMetricNames() {
+ String expectedGroup = "kafka.controller";
+ String expectedType = "KafkaController";
+ Set<String> expectedMetricNames = Utils.mkSet(
+ "ActiveControllerCount",
+ "GlobalTopicCount",
+ "GlobalPartitionCount",
+ "OfflinePartitionsCount",
+ "PreferredReplicaImbalanceCount");
+ Set<String> missingMetrics = getMissingMetricNames(expectedMetricNames, expectedGroup, expectedType);
+ assertEquals(Collections.emptySet(), missingMetrics, "Expected metrics did not exist");
+ }
+
+ @Test
+ public void testControllerEventManagerMetricNames() {
+ String expectedGroup = "kafka.controller";
+ String expectedType = "ControllerEventManager";
+ Set<String> expectedMetricNames = new HashSet<>(Arrays.asList(
+ "EventQueueTimeMs",
+ "EventQueueProcessingTimeMs"));
+ Set<String> missingMetrics = getMissingMetricNames(expectedMetricNames, expectedGroup, expectedType);
+ assertEquals(Collections.emptySet(), missingMetrics, "Expected metrics did not exist");
Review comment:
nit: When this assert failed, we'll see the error messages:
`Expected metrics did not exist` ==> expected: `emptySet`, but was: `missingMetrics`
I think we should change the error messages, ex:
`Expect no missing metrics` ==> expected: `emptySet`, but was: `missingMetrics`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on pull request #11131: KAFKA-13137: KRaft Controller Metric MBean names incorrectly quoted
Posted by GitBox <gi...@apache.org>.
rondagostino commented on pull request #11131:
URL: https://github.com/apache/kafka/pull/11131#issuecomment-888721165
Thanks, @hachikuji -- I think it's good now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hachikuji merged pull request #11131: KAFKA-13137: KRaft Controller Metric MBean names incorrectly quoted
Posted by GitBox <gi...@apache.org>.
hachikuji merged pull request #11131:
URL: https://github.com/apache/kafka/pull/11131
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hachikuji commented on a change in pull request #11131: KAFKA-13137: KRaft Controller Metric MBean names incorrectly quoted
Posted by GitBox <gi...@apache.org>.
hachikuji commented on a change in pull request #11131:
URL: https://github.com/apache/kafka/pull/11131#discussion_r676993092
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumControllerMetrics.java
##########
@@ -34,7 +34,7 @@
private final static MetricName GLOBAL_PARTITION_COUNT = getMetricName(
"KafkaController", "GlobalPartitionCount");
private final static MetricName OFFLINE_PARTITION_COUNT = getMetricName(
- "KafkaController", "OfflinePartitionCount");
+ "KafkaController", "OfflinePartitionsCount");
Review comment:
Good catch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hachikuji commented on a change in pull request #11131: KAFKA-13137: KRaft Controller Metric MBean names incorrectly quoted
Posted by GitBox <gi...@apache.org>.
hachikuji commented on a change in pull request #11131:
URL: https://github.com/apache/kafka/pull/11131#discussion_r678716922
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ControllerMetrics.java
##########
@@ -42,4 +42,6 @@
void setPreferredReplicaImbalanceCount(int replicaImbalances);
int preferredReplicaImbalanceCount();
+
+ void close();
Review comment:
nit: shall we extend `AutoCloseable`?
##########
File path: metadata/src/test/java/org/apache/kafka/controller/QuorumControllerMetricsTest.java
##########
@@ -45,26 +45,36 @@ public void testControllerEventManagerMetricNames() {
Set<String> expectedMetricNames = Utils.mkSet(
"EventQueueTimeMs",
"EventQueueProcessingTimeMs");
- assertExpectedMetrics(expectedMetricNames, expectedType);
+ assertExpectedMetricsCreatedAndRemovedUponClose(expectedMetricNames, expectedType);
}
- private static void assertExpectedMetrics(Set<String> expectedMetricNames, String expectedType) {
+ private static void assertExpectedMetricsCreatedAndRemovedUponClose(Set<String> expectedMetricNames, String expectedType) {
String expectedGroup = "kafka.controller";
MetricsRegistry registry = new MetricsRegistry();
- new QuorumControllerMetrics(registry); // populates the registry
- expectedMetricNames.stream().forEach(expectedMetricName -> assertTrue(
- registry.allMetrics().keySet().stream().anyMatch(metricName -> {
- if (metricName.getGroup().equals(expectedGroup) && metricName.getType().equals(expectedType)
- && metricName.getScope() == null && metricName.getName().equals(expectedMetricName)) {
- // It has to exist AND the MBean name has to be correct;
- // fail right here if the MBean name doesn't match
- String expectedMBeanPrefix = expectedGroup + ":type=" + expectedType + ",name=";
- assertEquals(expectedMBeanPrefix + expectedMetricName, metricName.getMBeanName(),
- "Incorrect MBean name");
- return true; // the expected metric name exists and the associated MBean name matches
- } else {
- return false; // this one didn't match
- }
- }), "Missing metric: " + expectedMetricName));
+ QuorumControllerMetrics quorumControllerMetrics = new QuorumControllerMetrics(registry); // populates the registry
+ Arrays.asList(true, false).forEach(checkMetricsExist -> {
Review comment:
This seems overly complicated. An easier structure to follow would be something like this:
```java
String expectedType = "KafkaController";
Set<String> expectedMetricNames = Utils.mkSet(
"ActiveControllerCount",
"GlobalTopicCount",
"GlobalPartitionCount",
"OfflinePartitionsCount",
"PreferredReplicaImbalanceCount"
);
MetricsRegistry registry = new MetricsRegistry();
try (QuorumControllerMetrics quorumControllerMetrics = new QuorumControllerMetrics(registry)) {
assertMetricsCreated(registry, expectedMetricNames);
}
assertMetricsRemoved(registry, expectedMetricNames);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hachikuji commented on a change in pull request #11131: KAFKA-13137: KRaft Controller Metric MBean names incorrectly quoted
Posted by GitBox <gi...@apache.org>.
hachikuji commented on a change in pull request #11131:
URL: https://github.com/apache/kafka/pull/11131#discussion_r677032066
##########
File path: metadata/src/test/java/org/apache/kafka/controller/QuorumControllerMetricsTest.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import com.yammer.metrics.core.MetricsRegistry;
+import org.apache.kafka.common.utils.Utils;
+import org.junit.jupiter.api.Test;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class QuorumControllerMetricsTest {
+ @Test
+ public void testKafkaControllerMetricNames() {
+ String expectedGroup = "kafka.controller";
+ String expectedType = "KafkaController";
+ Set<String> expectedMetricNames = Utils.mkSet(
+ "ActiveControllerCount",
+ "GlobalTopicCount",
+ "GlobalPartitionCount",
+ "OfflinePartitionsCount",
+ "PreferredReplicaImbalanceCount");
+ Set<String> missingMetrics = getMissingMetricNames(expectedMetricNames, expectedGroup, expectedType);
+ assertEquals(Collections.emptySet(), missingMetrics, "Expected metrics did not exist");
+ }
+
+ @Test
+ public void testControllerEventManagerMetricNames() {
+ String expectedGroup = "kafka.controller";
+ String expectedType = "ControllerEventManager";
+ Set<String> expectedMetricNames = new HashSet<>(Arrays.asList(
+ "EventQueueTimeMs",
+ "EventQueueProcessingTimeMs"));
+ Set<String> missingMetrics = getMissingMetricNames(expectedMetricNames, expectedGroup, expectedType);
+ assertEquals(Collections.emptySet(), missingMetrics, "Expected metrics did not exist");
+ }
+
+ private static Set<String> getMissingMetricNames(
+ Set<String> expectedMetricNames, String expectedGroup, String expectedType) {
+ MetricsRegistry registry = new MetricsRegistry();
+ new QuorumControllerMetrics(registry); // populates the registry
+ Set<String> foundMetricNames = expectedMetricNames.stream().filter(expectedMetricName ->
+ registry.allMetrics().keySet().stream().anyMatch(metricName -> {
+ if (metricName.getGroup().equals(expectedGroup) && metricName.getType().equals(expectedType)
+ && metricName.getScope() == null && metricName.getName().equals(expectedMetricName)) {
+ // It has to exist AND the MBean name has to be correct;
+ // fail right here if the MBean name doesn't match
+ String expectedMBeanPrefix = expectedGroup + ":type=" + expectedType + ",name=";
+ assertEquals(expectedMBeanPrefix + expectedMetricName, metricName.getMBeanName());
+ return true; // the expected metric name exists and the associated MBean name matches
+ } else {
+ return false; // this one didn't match
Review comment:
Could we fail the test right here? It doesn't seem like there is much benefit to returning the missing metrics from the method. That would let us simplify this a little. Instead of this:
```java
Set<String> missingMetrics = getMissingMetricNames(expectedMetricNames, expectedGroup, expectedType);
assertEquals(Collections.emptySet(), missingMetrics, "Expected metrics did not exist");
```
we could have:
```java
assertRegisteredMetrics(expectedMetricNames, expectedGroup, expectedType);
```
We could probably also drop `expectedGroup` since we only have `kafka.controller`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org