You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "tinaselenge (via GitHub)" <gi...@apache.org> on 2023/09/12 11:04:04 UTC
[GitHub] [kafka] tinaselenge opened a new pull request, #14375: KAFKA-15158: Add metrics for RemoteRequestsPerSec
tinaselenge opened a new pull request, #14375:
URL: https://github.com/apache/kafka/pull/14375
*More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.*
*Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.*
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15158: Add metrics for RemoteRequestsPerSec [kafka]
Posted by "tinaselenge (via GitHub)" <gi...@apache.org>.
tinaselenge commented on code in PR #14375:
URL: https://github.com/apache/kafka/pull/14375#discussion_r1428032730
##########
core/src/main/java/kafka/server/ReplicaFetcherTierStateMachine.java:
##########
@@ -229,6 +229,9 @@ private Long buildRemoteLogAuxState(TopicPartition topicPartition,
Partition partition = replicaMgr.getPartitionOrException(topicPartition);
partition.truncateFullyAndStartAt(nextOffset, false, Option.apply(leaderLogStartOffset));
+ replicaMgr.brokerTopicStats().topicStats(topicPartition.topic()).buildRemoteLogAuxStateRequestRate().mark();
+ replicaMgr.brokerTopicStats().allTopicsStats().buildRemoteLogAuxStateRequestRate().mark();
+
Review Comment:
The original reason was not to emit the error metrics, when buildRemoteLogAuxState fails because of the remote log storage not being enabled. However, thinking about this again, this makes more sense. We will always emit request metrics when we attempt to build log aux, and emit error metrics when there is an exception.
I made the suggested change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15158: Add metrics for RemoteRequestsPerSec [kafka]
Posted by "tinaselenge (via GitHub)" <gi...@apache.org>.
tinaselenge commented on code in PR #14375:
URL: https://github.com/apache/kafka/pull/14375#discussion_r1428034292
##########
core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala:
##########
@@ -3205,13 +3220,52 @@ class ReplicaManagerTest {
threadNamePrefix: Option[String],
quotaManager: ReplicationQuotaManager
): ReplicaFetcherManager = {
- mockReplicaFetcherManager.getOrElse {
- super.createReplicaFetcherManager(
- metrics,
- time,
- threadNamePrefix,
- quotaManager
- )
+ mockReplicaFetcherManager.getOrElse {
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15158: Add metrics for RemoteRequestsPerSec [kafka]
Posted by "tinaselenge (via GitHub)" <gi...@apache.org>.
tinaselenge commented on code in PR #14375:
URL: https://github.com/apache/kafka/pull/14375#discussion_r1360770031
##########
core/src/main/java/kafka/server/ReplicaFetcherTierStateMachine.java:
##########
@@ -68,14 +68,15 @@
*/
public class ReplicaFetcherTierStateMachine implements TierStateMachine {
private static final Logger log = LoggerFactory.getLogger(ReplicaFetcherTierStateMachine.class);
-
+ private BrokerTopicStats brokerTopicStats;
private LeaderEndPoint leader;
private ReplicaManager replicaMgr;
public ReplicaFetcherTierStateMachine(LeaderEndPoint leader,
ReplicaManager replicaMgr) {
this.leader = leader;
this.replicaMgr = replicaMgr;
+ this.brokerTopicStats = replicaMgr.brokerTopicStats();
Review Comment:
Good catch. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15158: Add metrics for RemoteRequestsPerSec [kafka]
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on PR #14375:
URL: https://github.com/apache/kafka/pull/14375#issuecomment-1862219804
Failed tests are unrelated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15158: Add metrics for RemoteRequestsPerSec [kafka]
Posted by "clolov (via GitHub)" <gi...@apache.org>.
clolov commented on PR #14375:
URL: https://github.com/apache/kafka/pull/14375#issuecomment-1772757204
Yup, I believe these metrics are useful and I will add them as part of KIP-963, thanks for bringing this to my attention!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] divijvaidya commented on pull request #14375: KAFKA-15158: Add metrics for RemoteRequestsPerSec
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on PR #14375:
URL: https://github.com/apache/kafka/pull/14375#issuecomment-1725125479
Thank you for adding these metrics. When this is ready for review, please add the associated KIP discussing these metrics in the description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15158: Add metrics for RemoteRequestsPerSec [kafka]
Posted by "clolov (via GitHub)" <gi...@apache.org>.
clolov commented on PR #14375:
URL: https://github.com/apache/kafka/pull/14375#issuecomment-1780714845
I have republished KIP-963 (https://lists.apache.org/thread/40vsyc240hyody37mf2f0pn90shkzb45). Would people manage to find some time to comment on it (@tinaselenge @nikramakrishnan @showuon @kamalcph @satishd @divijvaidya)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] tinaselenge commented on a diff in pull request #14375: KAFKA-15158: Add metrics for RemoteRequestsPerSec
Posted by "tinaselenge (via GitHub)" <gi...@apache.org>.
tinaselenge commented on code in PR #14375:
URL: https://github.com/apache/kafka/pull/14375#discussion_r1335624219
##########
core/src/main/java/kafka/server/ReplicaFetcherTierStateMachine.java:
##########
@@ -68,14 +68,16 @@
*/
public class ReplicaFetcherTierStateMachine implements TierStateMachine {
private static final Logger log = LoggerFactory.getLogger(ReplicaFetcherTierStateMachine.class);
-
+ private BrokerTopicStats brokerTopicStats;
private LeaderEndPoint leader;
private ReplicaManager replicaMgr;
public ReplicaFetcherTierStateMachine(LeaderEndPoint leader,
- ReplicaManager replicaMgr) {
+ ReplicaManager replicaMgr,
+ BrokerTopicStats brokerTopicStats) {
Review Comment:
Thank you for pointing this out! So I don't need to change as many classes. Fixed now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15158: Add metrics for RemoteRequestsPerSec [kafka]
Posted by "tinaselenge (via GitHub)" <gi...@apache.org>.
tinaselenge commented on PR #14375:
URL: https://github.com/apache/kafka/pull/14375#issuecomment-1857973003
@showuon thank you so much for reviewing the PR. I believe I address the comments that I can. Please review again, when you get a chance. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15158: Add metrics for RemoteRequestsPerSec [kafka]
Posted by "tinaselenge (via GitHub)" <gi...@apache.org>.
tinaselenge commented on PR #14375:
URL: https://github.com/apache/kafka/pull/14375#issuecomment-1860352086
thanks @showuon, @nikramakrishnan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15158: Add metrics for RemoteRequestsPerSec [kafka]
Posted by "nikramakrishnan (via GitHub)" <gi...@apache.org>.
nikramakrishnan commented on code in PR #14375:
URL: https://github.com/apache/kafka/pull/14375#discussion_r1357082765
##########
core/src/main/java/kafka/server/ReplicaFetcherTierStateMachine.java:
##########
@@ -68,14 +68,16 @@
*/
public class ReplicaFetcherTierStateMachine implements TierStateMachine {
private static final Logger log = LoggerFactory.getLogger(ReplicaFetcherTierStateMachine.class);
-
+ private BrokerTopicStats brokerTopicStats;
private LeaderEndPoint leader;
private ReplicaManager replicaMgr;
public ReplicaFetcherTierStateMachine(LeaderEndPoint leader,
- ReplicaManager replicaMgr) {
+ ReplicaManager replicaMgr,
+ BrokerTopicStats brokerTopicStats) {
Review Comment:
Note that this would currently break a lot of tests because `MockTierStateMachine` passes `null` for the `replicaMgr` parameter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15158: Add metrics for RemoteDelete and BuildRemoteLogAuxState [kafka]
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon merged PR #14375:
URL: https://github.com/apache/kafka/pull/14375
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15158: Add metrics for RemoteRequestsPerSec [kafka]
Posted by "tinaselenge (via GitHub)" <gi...@apache.org>.
tinaselenge commented on code in PR #14375:
URL: https://github.com/apache/kafka/pull/14375#discussion_r1428033100
##########
core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala:
##########
@@ -3205,13 +3220,52 @@ class ReplicaManagerTest {
threadNamePrefix: Option[String],
quotaManager: ReplicationQuotaManager
): ReplicaFetcherManager = {
- mockReplicaFetcherManager.getOrElse {
- super.createReplicaFetcherManager(
- metrics,
- time,
- threadNamePrefix,
- quotaManager
- )
+ mockReplicaFetcherManager.getOrElse {
+ if (buildRemoteLogAuxState) {
+ super.createReplicaFetcherManager(
+ metrics,
+ time,
+ threadNamePrefix,
+ quotaManager
+ )
+ val config = this.config
+ val metadataCache = this.metadataCache
+ new ReplicaFetcherManager(config, this, metrics, time, threadNamePrefix, quotaManager, () => metadataCache.metadataVersion(), () => 1) {
+ override def createFetcherThread(fetcherId: Int, sourceBroker: BrokerEndPoint): ReplicaFetcherThread = {
+ val prefix = threadNamePrefix.map(tp => s"$tp:").getOrElse("")
+ val threadName = s"${prefix}ReplicaFetcherThread-$fetcherId-${sourceBroker.id}"
+
+ val tp = new TopicPartition(topic, 0)
+ val leader = new MockLeaderEndPoint() {
+ override def fetch(fetchRequest: FetchRequest.Builder): Map[TopicPartition, FetchData] = {
+ Map(tp -> new FetchData().setErrorCode(Errors.OFFSET_MOVED_TO_TIERED_STORAGE.code))
+ }
+ }
+ leader.setLeaderState(tp, PartitionState(leaderEpoch = 0))
+ leader.setReplicaPartitionStateCallback(tp => PartitionState(leaderEpoch = 0))
+
+ val fetcher = new ReplicaFetcherThread(threadName, leader, config, failedPartitions, replicaManager,
+ quotaManager, "", () => config.interBrokerProtocolVersion)
+
+ val initialFetchState = InitialFetchState(
+ topicId = Some(Uuid.randomUuid()),
+ leader = leader.brokerEndPoint(),
+ currentLeaderEpoch = 0,
+ initOffset = 0)
+
+ fetcher.addPartitions(Map(tp -> initialFetchState))
+
+ fetcher
+ }
+ }
+ } else {
+ super.createReplicaFetcherManager(
+ metrics,
+ time,
+ threadNamePrefix,
+ quotaManager
+ )
Review Comment:
This is not a duplication but intended change. It cannot be done that way due to the mismatching return type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15158: Add metrics for RemoteRequestsPerSec [kafka]
Posted by "tinaselenge (via GitHub)" <gi...@apache.org>.
tinaselenge commented on code in PR #14375:
URL: https://github.com/apache/kafka/pull/14375#discussion_r1428033100
##########
core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala:
##########
@@ -3205,13 +3220,52 @@ class ReplicaManagerTest {
threadNamePrefix: Option[String],
quotaManager: ReplicationQuotaManager
): ReplicaFetcherManager = {
- mockReplicaFetcherManager.getOrElse {
- super.createReplicaFetcherManager(
- metrics,
- time,
- threadNamePrefix,
- quotaManager
- )
+ mockReplicaFetcherManager.getOrElse {
+ if (buildRemoteLogAuxState) {
+ super.createReplicaFetcherManager(
+ metrics,
+ time,
+ threadNamePrefix,
+ quotaManager
+ )
+ val config = this.config
+ val metadataCache = this.metadataCache
+ new ReplicaFetcherManager(config, this, metrics, time, threadNamePrefix, quotaManager, () => metadataCache.metadataVersion(), () => 1) {
+ override def createFetcherThread(fetcherId: Int, sourceBroker: BrokerEndPoint): ReplicaFetcherThread = {
+ val prefix = threadNamePrefix.map(tp => s"$tp:").getOrElse("")
+ val threadName = s"${prefix}ReplicaFetcherThread-$fetcherId-${sourceBroker.id}"
+
+ val tp = new TopicPartition(topic, 0)
+ val leader = new MockLeaderEndPoint() {
+ override def fetch(fetchRequest: FetchRequest.Builder): Map[TopicPartition, FetchData] = {
+ Map(tp -> new FetchData().setErrorCode(Errors.OFFSET_MOVED_TO_TIERED_STORAGE.code))
+ }
+ }
+ leader.setLeaderState(tp, PartitionState(leaderEpoch = 0))
+ leader.setReplicaPartitionStateCallback(tp => PartitionState(leaderEpoch = 0))
+
+ val fetcher = new ReplicaFetcherThread(threadName, leader, config, failedPartitions, replicaManager,
+ quotaManager, "", () => config.interBrokerProtocolVersion)
+
+ val initialFetchState = InitialFetchState(
+ topicId = Some(Uuid.randomUuid()),
+ leader = leader.brokerEndPoint(),
+ currentLeaderEpoch = 0,
+ initOffset = 0)
+
+ fetcher.addPartitions(Map(tp -> initialFetchState))
+
+ fetcher
+ }
+ }
+ } else {
+ super.createReplicaFetcherManager(
+ metrics,
+ time,
+ threadNamePrefix,
+ quotaManager
+ )
Review Comment:
This cannot be done, due to the mismatching return type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] divijvaidya commented on a diff in pull request #14375: KAFKA-15158: Add metrics for RemoteRequestsPerSec
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #14375:
URL: https://github.com/apache/kafka/pull/14375#discussion_r1332594386
##########
core/src/main/java/kafka/server/ReplicaFetcherTierStateMachine.java:
##########
@@ -68,14 +68,16 @@
*/
public class ReplicaFetcherTierStateMachine implements TierStateMachine {
private static final Logger log = LoggerFactory.getLogger(ReplicaFetcherTierStateMachine.class);
-
+ private BrokerTopicStats brokerTopicStats;
private LeaderEndPoint leader;
private ReplicaManager replicaMgr;
public ReplicaFetcherTierStateMachine(LeaderEndPoint leader,
- ReplicaManager replicaMgr) {
+ ReplicaManager replicaMgr,
+ BrokerTopicStats brokerTopicStats) {
Review Comment:
you can get brokerTopicStats as replicaMgr.brokerTopicStats()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15158: Add metrics for RemoteRequestsPerSec [kafka]
Posted by "tinaselenge (via GitHub)" <gi...@apache.org>.
tinaselenge commented on code in PR #14375:
URL: https://github.com/apache/kafka/pull/14375#discussion_r1428029215
##########
core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala:
##########
@@ -3884,6 +3938,116 @@ class ReplicaManagerTest {
}
}
+ @Test
+ def testSuccessfulBuildRemoteLogAuxStateMetrics(): Unit = {
+ val tp0 = new TopicPartition(topic, 0)
+
+ val remoteLogManager = mock(classOf[RemoteLogManager])
+ val remoteLogSegmentMetadata = mock(classOf[RemoteLogSegmentMetadata])
+ when(remoteLogManager.fetchRemoteLogSegmentMetadata(any(), anyInt(), anyLong())).thenReturn(
+ Optional.of(remoteLogSegmentMetadata)
+ )
+ val storageManager = mock(classOf[RemoteStorageManager])
+ when(storageManager.fetchIndex(any(), any())).thenReturn(new ByteArrayInputStream("0".getBytes()))
+ when(remoteLogManager.storageManager()).thenReturn(storageManager)
+
+ val replicaManager = setupReplicaManagerWithMockedPurgatories(new MockTimer(time), aliveBrokerIds = Seq(0, 1, 2), enableRemoteStorage = true, shouldMockLog = true, remoteLogManager = Some(remoteLogManager), buildRemoteLogAuxState = true)
+ try {
+
+ val offsetCheckpoints = new LazyOffsetCheckpoints(replicaManager.highWatermarkCheckpoints)
+ replicaManager.createPartition(tp0).createLogIfNotExists(isNew = false, isFutureReplica = false, offsetCheckpoints, None)
+ val partition0Replicas = Seq[Integer](0, 1).asJava
+ val topicIds = Map(tp0.topic -> topicId).asJava
+ val leaderAndIsrRequest = new LeaderAndIsrRequest.Builder(ApiKeys.LEADER_AND_ISR.latestVersion, 0, 0, brokerEpoch,
+ Seq(
+ new LeaderAndIsrPartitionState()
+ .setTopicName(tp0.topic)
+ .setPartitionIndex(tp0.partition)
+ .setControllerEpoch(0)
+ .setLeader(1)
+ .setLeaderEpoch(0)
+ .setIsr(partition0Replicas)
+ .setPartitionEpoch(0)
+ .setReplicas(partition0Replicas)
+ .setIsNew(true)
+ ).asJava,
+ topicIds,
+ Set(new Node(0, "host1", 0), new Node(1, "host2", 1)).asJava).build()
+
+ // Verify the metrics for build remote log state and for failures is zero before replicas start to fetch
+ assertEquals(0, brokerTopicStats.topicStats(tp0.topic()).buildRemoteLogAuxStateRequestRate.count)
+ assertEquals(0, brokerTopicStats.topicStats(tp0.topic()).failedBuildRemoteLogAuxStateRate.count)
+ // Verify aggregate metrics
+ assertEquals(0, brokerTopicStats.allTopicsStats.buildRemoteLogAuxStateRequestRate.count)
+ assertEquals(0, brokerTopicStats.allTopicsStats.failedBuildRemoteLogAuxStateRate.count)
+
+ replicaManager.becomeLeaderOrFollower(0, leaderAndIsrRequest, (_, _) => ())
+
+ // Replicas fetch from the leader periodically, therefore we check that the metric value is increasing
+ assertTrue(brokerTopicStats.topicStats(tp0.topic()).buildRemoteLogAuxStateRequestRate.count > 0)
+ assertEquals(0, brokerTopicStats.topicStats(tp0.topic()).failedBuildRemoteLogAuxStateRate.count)
+ // Verify aggregate metrics
+ assertTrue(brokerTopicStats.allTopicsStats.buildRemoteLogAuxStateRequestRate.count > 0)
+ assertEquals(0, brokerTopicStats.allTopicsStats.failedBuildRemoteLogAuxStateRate.count)
+ } finally {
+ replicaManager.shutdown(checkpointHW = false)
+ }
+ }
+
+ @Test
+ def testFailedBuildRemoteLogAuxStateMetrics(): Unit = {
Review Comment:
`
when(remoteLogManager.fetchRemoteLogSegmentMetadata(any(), anyInt(), anyLong())).thenReturn(
Optional.empty())
`
This is basically what testFailedBuildRemoteLogAuxStateMetrics() already does. Because we are not mocking remoteLogSegmentMetadata, fetchRemoteLogSegmentMetadata returns an empty result.
However, I added another test for the exception part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15158: Add metrics for RemoteRequestsPerSec [kafka]
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on PR #14375:
URL: https://github.com/apache/kafka/pull/14375#issuecomment-1858633479
There is a compilation failure in java 8 scala 12. Please help fix it.
https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14375/16/pipeline
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15158: Add metrics for RemoteRequestsPerSec [kafka]
Posted by "nikramakrishnan (via GitHub)" <gi...@apache.org>.
nikramakrishnan commented on code in PR #14375:
URL: https://github.com/apache/kafka/pull/14375#discussion_r1358162028
##########
core/src/main/java/kafka/server/ReplicaFetcherTierStateMachine.java:
##########
@@ -229,6 +230,9 @@ private Long buildRemoteLogAuxState(TopicPartition topicPartition,
Partition partition = replicaMgr.getPartitionOrException(topicPartition);
partition.truncateFullyAndStartAt(nextOffset, false, Option.apply(leaderLogStartOffset));
+ brokerTopicStats.topicStats(topicPartition.topic()).buildRemoteLogAuxStateRequestRate().mark();
+ brokerTopicStats.allTopicsStats().buildRemoteLogAuxStateRequestRate().mark();
Review Comment:
Instead, we could call `replicaMgr.brokerTopicStats().topicStats(...)` here directly to avoid a null pointer exception in the constructor.
##########
core/src/main/java/kafka/server/ReplicaFetcherTierStateMachine.java:
##########
@@ -68,14 +68,15 @@
*/
public class ReplicaFetcherTierStateMachine implements TierStateMachine {
private static final Logger log = LoggerFactory.getLogger(ReplicaFetcherTierStateMachine.class);
-
+ private BrokerTopicStats brokerTopicStats;
private LeaderEndPoint leader;
private ReplicaManager replicaMgr;
public ReplicaFetcherTierStateMachine(LeaderEndPoint leader,
ReplicaManager replicaMgr) {
this.leader = leader;
this.replicaMgr = replicaMgr;
+ this.brokerTopicStats = replicaMgr.brokerTopicStats();
Review Comment:
This would currently break a lot of tests because `MockTierStateMachine` passes `null` for the `replicaMgr` parameter of the `ReplicaFetcherTierStateMachine` constructor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15158: Add metrics for RemoteRequestsPerSec [kafka]
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on code in PR #14375:
URL: https://github.com/apache/kafka/pull/14375#discussion_r1426129619
##########
core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala:
##########
@@ -3205,13 +3220,52 @@ class ReplicaManagerTest {
threadNamePrefix: Option[String],
quotaManager: ReplicationQuotaManager
): ReplicaFetcherManager = {
- mockReplicaFetcherManager.getOrElse {
- super.createReplicaFetcherManager(
- metrics,
- time,
- threadNamePrefix,
- quotaManager
- )
+ mockReplicaFetcherManager.getOrElse {
Review Comment:
nit: did we have the indent correct here?
##########
core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala:
##########
@@ -3884,6 +3938,116 @@ class ReplicaManagerTest {
}
}
+ @Test
+ def testSuccessfulBuildRemoteLogAuxStateMetrics(): Unit = {
+ val tp0 = new TopicPartition(topic, 0)
+
+ val remoteLogManager = mock(classOf[RemoteLogManager])
+ val remoteLogSegmentMetadata = mock(classOf[RemoteLogSegmentMetadata])
+ when(remoteLogManager.fetchRemoteLogSegmentMetadata(any(), anyInt(), anyLong())).thenReturn(
+ Optional.of(remoteLogSegmentMetadata)
+ )
+ val storageManager = mock(classOf[RemoteStorageManager])
+ when(storageManager.fetchIndex(any(), any())).thenReturn(new ByteArrayInputStream("0".getBytes()))
+ when(remoteLogManager.storageManager()).thenReturn(storageManager)
+
+ val replicaManager = setupReplicaManagerWithMockedPurgatories(new MockTimer(time), aliveBrokerIds = Seq(0, 1, 2), enableRemoteStorage = true, shouldMockLog = true, remoteLogManager = Some(remoteLogManager), buildRemoteLogAuxState = true)
+ try {
+
+ val offsetCheckpoints = new LazyOffsetCheckpoints(replicaManager.highWatermarkCheckpoints)
+ replicaManager.createPartition(tp0).createLogIfNotExists(isNew = false, isFutureReplica = false, offsetCheckpoints, None)
+ val partition0Replicas = Seq[Integer](0, 1).asJava
+ val topicIds = Map(tp0.topic -> topicId).asJava
+ val leaderAndIsrRequest = new LeaderAndIsrRequest.Builder(ApiKeys.LEADER_AND_ISR.latestVersion, 0, 0, brokerEpoch,
+ Seq(
+ new LeaderAndIsrPartitionState()
+ .setTopicName(tp0.topic)
+ .setPartitionIndex(tp0.partition)
+ .setControllerEpoch(0)
+ .setLeader(1)
+ .setLeaderEpoch(0)
+ .setIsr(partition0Replicas)
+ .setPartitionEpoch(0)
+ .setReplicas(partition0Replicas)
+ .setIsNew(true)
+ ).asJava,
+ topicIds,
+ Set(new Node(0, "host1", 0), new Node(1, "host2", 1)).asJava).build()
+
+ // Verify the metrics for build remote log state and for failures is zero before replicas start to fetch
+ assertEquals(0, brokerTopicStats.topicStats(tp0.topic()).buildRemoteLogAuxStateRequestRate.count)
+ assertEquals(0, brokerTopicStats.topicStats(tp0.topic()).failedBuildRemoteLogAuxStateRate.count)
+ // Verify aggregate metrics
+ assertEquals(0, brokerTopicStats.allTopicsStats.buildRemoteLogAuxStateRequestRate.count)
+ assertEquals(0, brokerTopicStats.allTopicsStats.failedBuildRemoteLogAuxStateRate.count)
+
+ replicaManager.becomeLeaderOrFollower(0, leaderAndIsrRequest, (_, _) => ())
+
+ // Replicas fetch from the leader periodically, therefore we check that the metric value is increasing
+ assertTrue(brokerTopicStats.topicStats(tp0.topic()).buildRemoteLogAuxStateRequestRate.count > 0)
+ assertEquals(0, brokerTopicStats.topicStats(tp0.topic()).failedBuildRemoteLogAuxStateRate.count)
+ // Verify aggregate metrics
+ assertTrue(brokerTopicStats.allTopicsStats.buildRemoteLogAuxStateRequestRate.count > 0)
+ assertEquals(0, brokerTopicStats.allTopicsStats.failedBuildRemoteLogAuxStateRate.count)
+ } finally {
+ replicaManager.shutdown(checkpointHW = false)
+ }
+ }
+
+ @Test
+ def testFailedBuildRemoteLogAuxStateMetrics(): Unit = {
Review Comment:
I'm thinking we should add 1 more test case that when `remoteLogManager.fetchRemoteLogSegmentMetadata` throws exception, we should also mark the error rate. So, maybe we use parameter in the test? ex:
```
@Test
def testFailedBuildRemoteLogAuxStateMetrics(boolean shouldThrowException): Unit = {
if (!shouldThrowException)
when(remoteLogManager.fetchRemoteLogSegmentMetadata(any(), anyInt(), anyLong())).thenReturn(
Optional.empty())
else {
when(remoteLogManager.fetchRemoteLogSegmentMetadata(any(), anyInt(), anyLong())).thenThrow(new Exception...)
}
WDYT?
```
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -1042,8 +1053,9 @@ void cleanupExpiredRemoteLogSegments() throws RemoteStorageException, ExecutionE
// If the follower HAS NOT picked up the changes, and they become the leader then they will go through this process
// again and delete them with the original deletion reason i.e. size, time or log start offset breach.
List<String> undeletedSegments = new ArrayList<>();
+ String topicName = log.topicPartition().topic();
Review Comment:
nit: I think using `topicIdPartition.topic()` is more straight-forward and consistent, like in L1045, we use `topicIdPartition.topicPartition()` to get partition.
##########
core/src/main/java/kafka/server/ReplicaFetcherTierStateMachine.java:
##########
@@ -229,6 +229,9 @@ private Long buildRemoteLogAuxState(TopicPartition topicPartition,
Partition partition = replicaMgr.getPartitionOrException(topicPartition);
partition.truncateFullyAndStartAt(nextOffset, false, Option.apply(leaderLogStartOffset));
+ replicaMgr.brokerTopicStats().topicStats(topicPartition.topic()).buildRemoteLogAuxStateRequestRate().mark();
+ replicaMgr.brokerTopicStats().allTopicsStats().buildRemoteLogAuxStateRequestRate().mark();
+
Review Comment:
Question: Why do we update request rate metric here?
I'm thinking we could mark the request rate and error rate at the caller of `buildRemoteLogAuxState` method. That is, in `ReplicaFetcherTierStateMachine#start`
```
long offsetToFetch = 0;
// mark request rate here
try {
offsetToFetch = buildRemoteLogAuxState(topicPartition, currentFetchState.currentLeaderEpoch(), leaderLocalStartOffset, epoch, fetchPartitionData.logStartOffset());
} catch (RemoteStorageException e) {
// mark error rate here
throw e;
}
```
This way, if there are exception thrown from other places, like in L219 `fetchRemoteLogSegmentMetadata` method, we can still catch that and update the metrics. WDYT?
##########
core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala:
##########
@@ -3205,13 +3220,52 @@ class ReplicaManagerTest {
threadNamePrefix: Option[String],
quotaManager: ReplicationQuotaManager
): ReplicaFetcherManager = {
- mockReplicaFetcherManager.getOrElse {
- super.createReplicaFetcherManager(
- metrics,
- time,
- threadNamePrefix,
- quotaManager
- )
+ mockReplicaFetcherManager.getOrElse {
+ if (buildRemoteLogAuxState) {
+ super.createReplicaFetcherManager(
+ metrics,
+ time,
+ threadNamePrefix,
+ quotaManager
+ )
+ val config = this.config
+ val metadataCache = this.metadataCache
+ new ReplicaFetcherManager(config, this, metrics, time, threadNamePrefix, quotaManager, () => metadataCache.metadataVersion(), () => 1) {
+ override def createFetcherThread(fetcherId: Int, sourceBroker: BrokerEndPoint): ReplicaFetcherThread = {
+ val prefix = threadNamePrefix.map(tp => s"$tp:").getOrElse("")
+ val threadName = s"${prefix}ReplicaFetcherThread-$fetcherId-${sourceBroker.id}"
+
+ val tp = new TopicPartition(topic, 0)
+ val leader = new MockLeaderEndPoint() {
+ override def fetch(fetchRequest: FetchRequest.Builder): Map[TopicPartition, FetchData] = {
+ Map(tp -> new FetchData().setErrorCode(Errors.OFFSET_MOVED_TO_TIERED_STORAGE.code))
+ }
+ }
+ leader.setLeaderState(tp, PartitionState(leaderEpoch = 0))
+ leader.setReplicaPartitionStateCallback(tp => PartitionState(leaderEpoch = 0))
+
+ val fetcher = new ReplicaFetcherThread(threadName, leader, config, failedPartitions, replicaManager,
+ quotaManager, "", () => config.interBrokerProtocolVersion)
+
+ val initialFetchState = InitialFetchState(
+ topicId = Some(Uuid.randomUuid()),
+ leader = leader.brokerEndPoint(),
+ currentLeaderEpoch = 0,
+ initOffset = 0)
+
+ fetcher.addPartitions(Map(tp -> initialFetchState))
+
+ fetcher
+ }
+ }
+ } else {
+ super.createReplicaFetcherManager(
+ metrics,
+ time,
+ threadNamePrefix,
+ quotaManager
+ )
Review Comment:
We duplicated these lines with L3224. Could we move them before if/else block?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15158: Add metrics for RemoteRequestsPerSec [kafka]
Posted by "tinaselenge (via GitHub)" <gi...@apache.org>.
tinaselenge commented on PR #14375:
URL: https://github.com/apache/kafka/pull/14375#issuecomment-1772998045
thank you @clolov!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15158: Add metrics for RemoteRequestsPerSec [kafka]
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on PR #14375:
URL: https://github.com/apache/kafka/pull/14375#issuecomment-1853767041
I'll take a look tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15158: Add metrics for RemoteRequestsPerSec [kafka]
Posted by "tinaselenge (via GitHub)" <gi...@apache.org>.
tinaselenge commented on code in PR #14375:
URL: https://github.com/apache/kafka/pull/14375#discussion_r1428029820
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -1042,8 +1053,9 @@ void cleanupExpiredRemoteLogSegments() throws RemoteStorageException, ExecutionE
// If the follower HAS NOT picked up the changes, and they become the leader then they will go through this process
// again and delete them with the original deletion reason i.e. size, time or log start offset breach.
List<String> undeletedSegments = new ArrayList<>();
+ String topicName = log.topicPartition().topic();
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15158: Add metrics for RemoteRequestsPerSec [kafka]
Posted by "kamalcph (via GitHub)" <gi...@apache.org>.
kamalcph commented on code in PR #14375:
URL: https://github.com/apache/kafka/pull/14375#discussion_r1428646042
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -926,16 +926,27 @@ private boolean deleteLogSegmentsDueToLeaderEpochCacheTruncation(EpochEntry earl
return isSegmentDeleted;
}
- private boolean deleteRemoteLogSegment(RemoteLogSegmentMetadata segmentMetadata, Predicate<RemoteLogSegmentMetadata> predicate)
+ private boolean deleteRemoteLogSegment(String topic, RemoteLogSegmentMetadata segmentMetadata, Predicate<RemoteLogSegmentMetadata> predicate)
Review Comment:
Can we take the `topic` name from the segmentMetadata instead of propagating it from the caller?
```
String topic = segmentMetadata.topicIdPartition().topic();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15158: Add metrics for RemoteRequestsPerSec [kafka]
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on PR #14375:
URL: https://github.com/apache/kafka/pull/14375#issuecomment-1855078026
Also, please resolve the conflict. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15158: Add metrics for RemoteRequestsPerSec [kafka]
Posted by "nikramakrishnan (via GitHub)" <gi...@apache.org>.
nikramakrishnan commented on code in PR #14375:
URL: https://github.com/apache/kafka/pull/14375#discussion_r1357082765
##########
core/src/main/java/kafka/server/ReplicaFetcherTierStateMachine.java:
##########
@@ -68,14 +68,16 @@
*/
public class ReplicaFetcherTierStateMachine implements TierStateMachine {
private static final Logger log = LoggerFactory.getLogger(ReplicaFetcherTierStateMachine.class);
-
+ private BrokerTopicStats brokerTopicStats;
private LeaderEndPoint leader;
private ReplicaManager replicaMgr;
public ReplicaFetcherTierStateMachine(LeaderEndPoint leader,
- ReplicaManager replicaMgr) {
+ ReplicaManager replicaMgr,
+ BrokerTopicStats brokerTopicStats) {
Review Comment:
Note that this would currently break a lot of tests because `MockTierStateMachine` passes `null` for the `replicaMgr` parameter of the `ReplicaFetcherTierStateMachine` constructor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org