You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "showuon (via GitHub)" <gi...@apache.org> on 2023/12/16 06:36:18 UTC
[PR] KAFKA-16014: add RemoteLogMetadataCount metric [kafka]
showuon opened a new pull request, #15026:
URL: https://github.com/apache/kafka/pull/15026
Add `RemoteLogMetadataCount` count metric as described in KIP-963.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16014: add RemoteLogMetadataCount metric [kafka]
Posted by "clolov (via GitHub)" <gi...@apache.org>.
clolov commented on PR #15026:
URL: https://github.com/apache/kafka/pull/15026#issuecomment-1860888646
Just to put my thoughts down in answer to what Kamal wrote. In my head, there are a couple of places on the copy and delete paths where we obtain the segment metadata for topic partitions the current broker is a leader for. I was imagining that regardless of whether a broker goes up or down it will report whatever it knows about the aggregated topic partitions it is the leader for. If one would want to know the information about the full topic then they would need to aggregate the values across all brokers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16014: add RemoteLogMetadataCount metric [kafka]
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd merged PR #15026:
URL: https://github.com/apache/kafka/pull/15026
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16014: add RemoteLogMetadataCount metric [kafka]
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on PR #15026:
URL: https://github.com/apache/kafka/pull/15026#issuecomment-1864082356
There are a few test failures unrelated to this PR, merging it to trunk and 3.7.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16014: add RemoteLogMetadataCount metric [kafka]
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on code in PR #15026:
URL: https://github.com/apache/kafka/pull/15026#discussion_r1432205258
##########
core/src/test/scala/kafka/server/KafkaRequestHandlerTest.scala:
##########
@@ -321,4 +323,24 @@ class KafkaRequestHandlerTest {
assertEquals(0, brokerTopicMetrics.remoteCopyBytesLag)
}
+ @Test
+ def testRemoteLogMetadataCount(): Unit = {
+ val brokerTopicMetrics = setupBrokerTopicMetrics()
+
+ assertEquals(0, brokerTopicMetrics.remoteLogMetadataCount)
+ brokerTopicMetrics.recordRemoteLogMetadataCount(0, 1)
+ assertEquals(1, brokerTopicMetrics.remoteLogMetadataCount)
+
+ brokerTopicMetrics.recordRemoteLogMetadataCount(1, 2)
+ brokerTopicMetrics.recordRemoteLogMetadataCount(2, 3)
+ assertEquals(3, brokerTopicMetrics.remoteLogMetadataCount)
+
+
+ // verify there will be no minus value for a partition
Review Comment:
Oh, I forgot to remove it.
##########
core/src/test/scala/kafka/server/KafkaRequestHandlerTest.scala:
##########
@@ -321,4 +323,24 @@ class KafkaRequestHandlerTest {
assertEquals(0, brokerTopicMetrics.remoteCopyBytesLag)
}
+ @Test
+ def testRemoteLogMetadataCount(): Unit = {
+ val brokerTopicMetrics = setupBrokerTopicMetrics()
+
+ assertEquals(0, brokerTopicMetrics.remoteLogMetadataCount)
+ brokerTopicMetrics.recordRemoteLogMetadataCount(0, 1)
+ assertEquals(1, brokerTopicMetrics.remoteLogMetadataCount)
+
+ brokerTopicMetrics.recordRemoteLogMetadataCount(1, 2)
+ brokerTopicMetrics.recordRemoteLogMetadataCount(2, 3)
+ assertEquals(3, brokerTopicMetrics.remoteLogMetadataCount)
Review Comment:
Oops!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16014: add RemoteLogMetadataCount metric [kafka]
Posted by "kamalcph (via GitHub)" <gi...@apache.org>.
kamalcph commented on code in PR #15026:
URL: https://github.com/apache/kafka/pull/15026#discussion_r1432264764
##########
core/src/main/scala/kafka/server/KafkaRequestHandler.scala:
##########
@@ -289,14 +289,14 @@ class BrokerTopicMetrics(name: Option[String], configOpt: java.util.Optional[Kaf
def gauge(): Gauge[Long] = gaugeLock synchronized {
if (gaugeObject == null) {
- gaugeObject = metricsGroup.newGauge(metricType, () => brokerTopicAggregatedMetric.value())
+ gaugeObject = metricsGroup.newGauge(metricType, () => brokerTopicAggregatedMetric.value(), tags)
Review Comment:
Nice catch, `tags` was missing for other metrics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16014: add RemoteLogMetadataCount metric [kafka]
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on PR #15026:
URL: https://github.com/apache/kafka/pull/15026#issuecomment-1862324346
Just saw the comments. Thanks all. I'll update it soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16014: add RemoteLogMetadataCount metric [kafka]
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #15026:
URL: https://github.com/apache/kafka/pull/15026#discussion_r1431232848
##########
core/src/main/scala/kafka/server/KafkaRequestHandler.scala:
##########
@@ -467,6 +480,12 @@ class BrokerTopicAggregatedMetric() {
partitionMetricValues.put(partition, partitionValue)
}
+ def incrementPartitionMetricValue(partition: Int): Unit = partitionMetricValues.compute(partition, (_, value) => value + 1)
+
+
+ def decrementPartitionMetricValue(partition: Int): Unit = partitionMetricValues.compute(partition, (_, value) => Math.max(0, value - 1))
Review Comment:
Can we remove this method also?
##########
core/src/main/scala/kafka/server/KafkaRequestHandler.scala:
##########
@@ -467,6 +480,12 @@ class BrokerTopicAggregatedMetric() {
partitionMetricValues.put(partition, partitionValue)
}
+ def incrementPartitionMetricValue(partition: Int): Unit = partitionMetricValues.compute(partition, (_, value) => value + 1)
Review Comment:
Can we remove this method as it is not used anymore?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16014: add RemoteLogMetadataCount metric [kafka]
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #15026:
URL: https://github.com/apache/kafka/pull/15026#discussion_r1431542018
##########
core/src/test/scala/kafka/server/KafkaRequestHandlerTest.scala:
##########
@@ -321,4 +323,24 @@ class KafkaRequestHandlerTest {
assertEquals(0, brokerTopicMetrics.remoteCopyBytesLag)
}
+ @Test
+ def testRemoteLogMetadataCount(): Unit = {
+ val brokerTopicMetrics = setupBrokerTopicMetrics()
+
+ assertEquals(0, brokerTopicMetrics.remoteLogMetadataCount)
+ brokerTopicMetrics.recordRemoteLogMetadataCount(0, 1)
+ assertEquals(1, brokerTopicMetrics.remoteLogMetadataCount)
+
+ brokerTopicMetrics.recordRemoteLogMetadataCount(1, 2)
+ brokerTopicMetrics.recordRemoteLogMetadataCount(2, 3)
+ assertEquals(3, brokerTopicMetrics.remoteLogMetadataCount)
Review Comment:
Should not this be 6 instead of 3?
##########
core/src/test/scala/kafka/server/KafkaRequestHandlerTest.scala:
##########
@@ -321,4 +323,24 @@ class KafkaRequestHandlerTest {
assertEquals(0, brokerTopicMetrics.remoteCopyBytesLag)
}
+ @Test
+ def testRemoteLogMetadataCount(): Unit = {
+ val brokerTopicMetrics = setupBrokerTopicMetrics()
+
+ assertEquals(0, brokerTopicMetrics.remoteLogMetadataCount)
+ brokerTopicMetrics.recordRemoteLogMetadataCount(0, 1)
+ assertEquals(1, brokerTopicMetrics.remoteLogMetadataCount)
+
+ brokerTopicMetrics.recordRemoteLogMetadataCount(1, 2)
+ brokerTopicMetrics.recordRemoteLogMetadataCount(2, 3)
+ assertEquals(3, brokerTopicMetrics.remoteLogMetadataCount)
+
+
+ // verify there will be no minus value for a partition
Review Comment:
What does this mean here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16014: add RemoteLogMetadataCount metric [kafka]
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on PR #15026:
URL: https://github.com/apache/kafka/pull/15026#issuecomment-1858739478
@clolov @satishd , call for review. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16014: add RemoteLogMetadataCount metric [kafka]
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on code in PR #15026:
URL: https://github.com/apache/kafka/pull/15026#discussion_r1432207715
##########
core/src/test/java/kafka/log/remote/RemoteLogManagerTest.java:
##########
@@ -701,16 +702,22 @@ void testRemoteLogManagerTasksAvgIdlePercentAndMetadataCountMetrics() throws Exc
CountDownLatch copyLogSegmentLatch = new CountDownLatch(1);
doAnswer(ans -> {
// waiting for verification
- copyLogSegmentLatch.await();
+ copyLogSegmentLatch.await(5000, TimeUnit.MILLISECONDS);
Review Comment:
Put await timeout to avoid to wait forever.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16014: add RemoteLogMetadataCount metric [kafka]
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on PR #15026:
URL: https://github.com/apache/kafka/pull/15026#issuecomment-1860033624
@clolov , that's a good point. I've updated the PR to address your comment. Please check again. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16014: add RemoteLogMetadataCount metric [kafka]
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on code in PR #15026:
URL: https://github.com/apache/kafka/pull/15026#discussion_r1431292250
##########
core/src/main/scala/kafka/server/KafkaRequestHandler.scala:
##########
@@ -467,6 +480,12 @@ class BrokerTopicAggregatedMetric() {
partitionMetricValues.put(partition, partitionValue)
}
+ def incrementPartitionMetricValue(partition: Int): Unit = partitionMetricValues.compute(partition, (_, value) => value + 1)
Review Comment:
Ah, right, just updated.
##########
core/src/main/scala/kafka/server/KafkaRequestHandler.scala:
##########
@@ -467,6 +480,12 @@ class BrokerTopicAggregatedMetric() {
partitionMetricValues.put(partition, partitionValue)
}
+ def incrementPartitionMetricValue(partition: Int): Unit = partitionMetricValues.compute(partition, (_, value) => value + 1)
+
+
+ def decrementPartitionMetricValue(partition: Int): Unit = partitionMetricValues.compute(partition, (_, value) => Math.max(0, value - 1))
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16014: add RemoteLogMetadataCount metric [kafka]
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on PR #15026:
URL: https://github.com/apache/kafka/pull/15026#issuecomment-1862456789
@clolov @satishd @kamalcph , I've updated the PR. Please take a look again. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16014: add RemoteLogMetadataCount metric [kafka]
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on code in PR #15026:
URL: https://github.com/apache/kafka/pull/15026#discussion_r1431177323
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -988,13 +985,24 @@ void cleanupExpiredRemoteLogSegments() throws RemoteStorageException, ExecutionE
return;
}
+ // Cleanup remote log segments and update the log start offset if applicable.
+ final Iterator<RemoteLogSegmentMetadata> segmentMetadataIter = remoteLogMetadataManager.listRemoteLogSegments(topicIdPartition);
+ if (!segmentMetadataIter.hasNext()) {
+ brokerTopicStats.topicStats(topicIdPartition.topic()).recordRemoteLogMetadataCount(topicIdPartition.partition(), 0);
+ logger.debug("No remote log segments available on remote storage for partition: {}", topicIdPartition);
+ return;
+ }
Review Comment:
Move the `listRemoteLogSegments` here so that we can have some cases return earlier without running `listRemoteLogSegments`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org