You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "hudeqi (via GitHub)" <gi...@apache.org> on 2023/03/29 08:22:49 UTC
[GitHub] [kafka] hudeqi opened a new pull request, #13473: KAFKA-14866:Remove controller module metrics when broker is shutting down
hudeqi opened a new pull request, #13473:
URL: https://github.com/apache/kafka/pull/13473
In the current situation, KafkaController started 12 related metrics at startup, but did not remove these metrics when broker shutdown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13473: KAFKA-14866:Remove controller module metrics when broker is shutting down
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi commented on PR #13473:
URL: https://github.com/apache/kafka/pull/13473#issuecomment-1543326829
> Hey @hudeqi I had a similar PR merge recently (#13623) and the comments I received in that PR could be applied to this one also. Specifically,
>
> 1. the metric names could be verified individually in the test.
> 2. the metric names could be moved in a const and we can use companion object to define those.
>
> Perhaps, you are make those changes in this PR as request the same committer to review this PR as well?
thanks! divijvaidya!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13473: KAFKA-14866:Remove controller module metrics when broker is shutting down
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi commented on PR #13473:
URL: https://github.com/apache/kafka/pull/13473#issuecomment-1494104389
> Added a nit, otherwise code looks good to me. As a general suggestion for this and future PRs, it helps the reviewer if your description has the following sections:
>
> ```
> Motivation (explaining why this change is required)
> Code change (what actually is being changed in the code)
> Testing (how did you validate that the code changes are correct)
> ```
>
> This PR is ready for a committer to take a look at it.
I see, thanks a lot!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13473: KAFKA-14866:Remove controller module metrics when broker is shutting down
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi commented on PR #13473:
URL: https://github.com/apache/kafka/pull/13473#issuecomment-1498629991
Do other committers have time to review this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13473: KAFKA-14866:Remove controller module metrics when broker is shutting down
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi commented on PR #13473:
URL: https://github.com/apache/kafka/pull/13473#issuecomment-1493517525
Hello, please help to review this PR, thank you! @divijvaidya
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13473: KAFKA-14866:Remove controller module metrics when broker is shutting down
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi commented on PR #13473:
URL: https://github.com/apache/kafka/pull/13473#issuecomment-1540199229
> bump this pr, Is there any other commiters to help review and merge this pr?
I'm sorry I didn't notice the comment ("We're past code freeze for 3.5 so moving this to the next release") in the corresponding [jira](https://issues.apache.org/jira/browse/KAFKA-14866)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13473: KAFKA-14866:Remove controller module metrics when broker is shutting down
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi commented on PR #13473:
URL: https://github.com/apache/kafka/pull/13473#issuecomment-1514011903
Hello, can you help to review this PR? @mimaison
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13473: KAFKA-14866:Remove controller module metrics when broker is shutting down
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi commented on PR #13473:
URL: https://github.com/apache/kafka/pull/13473#issuecomment-1569906469
> @dajac When you get a chance, please take a look. Thanks!
Hello, pin again. @dajac
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] divijvaidya commented on a diff in pull request #13473: KAFKA-14866:Remove controller module metrics when broker is shutting down
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #13473:
URL: https://github.com/apache/kafka/pull/13473#discussion_r1151651343
##########
core/src/main/scala/kafka/controller/KafkaController.scala:
##########
@@ -198,6 +198,7 @@ class KafkaController(val config: KafkaConfig,
def shutdown(): Unit = {
eventManager.close()
onControllerResignation()
+ removeMetrics()
Review Comment:
please do this in try {} finally {}
##########
core/src/main/scala/kafka/controller/KafkaController.scala:
##########
@@ -502,6 +503,21 @@ class KafkaController(val config: KafkaConfig,
info("Resigned")
}
+ private def removeMetrics(): Unit = {
+ metricsGroup.removeMetric("ActiveControllerCount")
Review Comment:
Perhaps we should store the metric names in an Array and then close all of them with:
```
for (String metricName : metrics)
metricsGroup.removeMetric(metricName);
```
this will ensure that new metrics added in the future here are cleaned up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13473: KAFKA-14866:Remove controller module metrics when broker is shutting down
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi commented on PR #13473:
URL: https://github.com/apache/kafka/pull/13473#issuecomment-1501643812
> Do other committers have time to review this PR?
@guozhangwang Hello, can you help to review these two PRs? this and https://github.com/apache/kafka/pull/13471
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13473: KAFKA-14866:Remove controller module metrics when broker is shutting down
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi commented on PR #13473:
URL: https://github.com/apache/kafka/pull/13473#issuecomment-1495258058
And this pr. thx! @showuon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13473: KAFKA-14866:Remove controller module metrics when broker is shutting down
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi commented on PR #13473:
URL: https://github.com/apache/kafka/pull/13473#issuecomment-1520101597
> Hello, can you help to review this PR? @mimaison
and pin here @mimaison
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mimaison commented on a diff in pull request #13473: KAFKA-14866:Remove controller module metrics when broker is shutting down
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on code in PR #13473:
URL: https://github.com/apache/kafka/pull/13473#discussion_r1218269590
##########
core/src/test/scala/unit/kafka/controller/KafkaControllerTest.scala:
##########
@@ -0,0 +1,77 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package kafka.controller
+
+import kafka.server.metadata.ZkMetadataCache
+import kafka.server.{BrokerFeatures, DelegationTokenManager, KafkaConfig}
+import kafka.utils.TestUtils
+import kafka.zk.{BrokerInfo, KafkaZkClient}
+import org.apache.kafka.common.metrics.Metrics
+import org.apache.kafka.common.utils.Time
+import org.apache.kafka.server.metrics.KafkaMetricsGroup
+import org.junit.jupiter.api.{BeforeEach, Test}
+import org.mockito.ArgumentMatchers
+import org.mockito.ArgumentMatchers.{any, anyString}
+import org.mockito.Mockito.{mock, mockConstruction, times, verify, verifyNoMoreInteractions}
+
+class KafkaControllerTest {
+ var config: KafkaConfig = _
+
+ @BeforeEach
+ def setUp(): Unit = {
+ val props = TestUtils.createBrokerConfig(1, TestUtils.MockZkConnect)
+ config = KafkaConfig.fromProps(props)
+ }
+
+ @Test
+ def checkMetricsMatchInKafkaController(): Unit = {
+ val mockMetricsGroupCtor = mockConstruction(classOf[KafkaMetricsGroup])
+ try {
+ val kafkaController = new KafkaController(
+ config = config,
+ zkClient = mock(classOf[KafkaZkClient]),
+ time = Time.SYSTEM,
+ metrics = mock(classOf[Metrics]),
+ initialBrokerInfo = mock(classOf[BrokerInfo]),
+ initialBrokerEpoch = 0,
+ tokenManager = mock(classOf[DelegationTokenManager]),
+ brokerFeatures = mock(classOf[BrokerFeatures]),
+ featureCache = mock(classOf[ZkMetadataCache])
+ )
+
+ // shutdown kafkaController so that metrics are removed
+ kafkaController.shutdown()
+
+ val mockMetricsGroup = mockMetricsGroupCtor.constructed.get(0)
+ val numMetricsRegistered = KafkaController.MetricNames.size
+ verify(mockMetricsGroup, times(numMetricsRegistered)).newGauge(anyString(), any())
+ KafkaController.MetricNames.foreach(metricName => verify(mockMetricsGroup).newGauge(ArgumentMatchers.eq(metricName), any()))
+ // verify that each metric is removed
+ verify(mockMetricsGroup, times(numMetricsRegistered)).removeMetric(anyString())
+ KafkaController.MetricNames.foreach(verify(mockMetricsGroup).removeMetric(_))
+
+ // assert that we have verified all invocations on
+ verifyNoMoreInteractions(mockMetricsGroup)
+ } finally {
+ if (mockMetricsGroupCtor != null) {
Review Comment:
Can `mockMetricsGroupCtor` actually be null here?
##########
core/src/test/scala/unit/kafka/controller/KafkaControllerTest.scala:
##########
@@ -0,0 +1,77 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package kafka.controller
+
+import kafka.server.metadata.ZkMetadataCache
+import kafka.server.{BrokerFeatures, DelegationTokenManager, KafkaConfig}
+import kafka.utils.TestUtils
+import kafka.zk.{BrokerInfo, KafkaZkClient}
+import org.apache.kafka.common.metrics.Metrics
+import org.apache.kafka.common.utils.Time
+import org.apache.kafka.server.metrics.KafkaMetricsGroup
+import org.junit.jupiter.api.{BeforeEach, Test}
+import org.mockito.ArgumentMatchers
+import org.mockito.ArgumentMatchers.{any, anyString}
+import org.mockito.Mockito.{mock, mockConstruction, times, verify, verifyNoMoreInteractions}
+
+class KafkaControllerTest {
+ var config: KafkaConfig = _
+
+ @BeforeEach
+ def setUp(): Unit = {
+ val props = TestUtils.createBrokerConfig(1, TestUtils.MockZkConnect)
+ config = KafkaConfig.fromProps(props)
+ }
+
+ @Test
+ def checkMetricsMatchInKafkaController(): Unit = {
Review Comment:
What about naming this `testRemoveMetricsOnClose`?
##########
core/src/test/scala/unit/kafka/controller/KafkaControllerTest.scala:
##########
@@ -0,0 +1,77 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package kafka.controller
+
+import kafka.server.metadata.ZkMetadataCache
+import kafka.server.{BrokerFeatures, DelegationTokenManager, KafkaConfig}
+import kafka.utils.TestUtils
+import kafka.zk.{BrokerInfo, KafkaZkClient}
+import org.apache.kafka.common.metrics.Metrics
+import org.apache.kafka.common.utils.Time
+import org.apache.kafka.server.metrics.KafkaMetricsGroup
+import org.junit.jupiter.api.{BeforeEach, Test}
+import org.mockito.ArgumentMatchers
+import org.mockito.ArgumentMatchers.{any, anyString}
+import org.mockito.Mockito.{mock, mockConstruction, times, verify, verifyNoMoreInteractions}
+
+class KafkaControllerTest {
+ var config: KafkaConfig = _
+
+ @BeforeEach
+ def setUp(): Unit = {
+ val props = TestUtils.createBrokerConfig(1, TestUtils.MockZkConnect)
+ config = KafkaConfig.fromProps(props)
+ }
+
+ @Test
+ def checkMetricsMatchInKafkaController(): Unit = {
+ val mockMetricsGroupCtor = mockConstruction(classOf[KafkaMetricsGroup])
+ try {
+ val kafkaController = new KafkaController(
+ config = config,
+ zkClient = mock(classOf[KafkaZkClient]),
+ time = Time.SYSTEM,
Review Comment:
Can we use `new MockTime()` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] divijvaidya commented on a diff in pull request #13473: KAFKA-14866:Remove controller module metrics when broker is shutting down
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #13473:
URL: https://github.com/apache/kafka/pull/13473#discussion_r1155781010
##########
core/src/test/scala/unit/kafka/controller/KafkaControllerTest.scala:
##########
@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package kafka.controller
+
+import kafka.server.metadata.ZkMetadataCache
+import kafka.server.{BrokerFeatures, DelegationTokenManager, KafkaConfig}
+import kafka.utils.TestUtils
+import kafka.zk.{BrokerInfo, KafkaZkClient}
+import org.apache.kafka.common.metrics.Metrics
+import org.apache.kafka.common.utils.Time
+import org.apache.kafka.server.metrics.KafkaMetricsGroup
+import org.junit.jupiter.api.{AfterEach, BeforeEach, Test}
+import org.mockito.ArgumentMatchers.{any, anyString}
+import org.mockito.Mockito.{mock, mockConstruction, times, verify, verifyNoMoreInteractions}
+
+class KafkaControllerTest {
+ var config: KafkaConfig = _
+
+ @BeforeEach
+ def setUp(): Unit = {
+ val props = TestUtils.createBrokerConfig(1, TestUtils.MockZkConnect)
+ config = KafkaConfig.fromProps(props)
+ }
+
+ @AfterEach
+ def tearDown(): Unit = {
+
+ }
Review Comment:
nit
This could perhaps be removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13473: KAFKA-14866:Remove controller module metrics when broker is shutting down
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi commented on PR #13473:
URL: https://github.com/apache/kafka/pull/13473#issuecomment-1577789264
> Thanks for the PR! I left a few comments
Thanks for your review, I have updated it based on your comments. @mimaison
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13473: KAFKA-14866:Remove controller module metrics when broker is shutting down
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi commented on PR #13473:
URL: https://github.com/apache/kafka/pull/13473#issuecomment-1540185668
bump this pr, Is there any other commiters to help review and merge this pr?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mimaison merged pull request #13473: KAFKA-14866:Remove controller module metrics when broker is shutting down
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison merged PR #13473:
URL: https://github.com/apache/kafka/pull/13473
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] divijvaidya commented on pull request #13473: KAFKA-14866:Remove controller module metrics when broker is shutting down
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on PR #13473:
URL: https://github.com/apache/kafka/pull/13473#issuecomment-1542365763
Hey @hudeqi
I had a similar PR merge recently (https://github.com/apache/kafka/pull/13623) and the comments I received in that PR could be applied to this one also.
Specifically,
1. the metric names could be verified individually in the test.
2. the metric names could be moved in a const and we can use companion object to define those.
Perhaps, you are make those changes in this PR as request the same committer to review this PR as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13473: KAFKA-14866:Remove controller module metrics when broker is shutting down
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi commented on PR #13473:
URL: https://github.com/apache/kafka/pull/13473#issuecomment-1543327504
@dajac When you get a chance, please take a look. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org