You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "aheev (via GitHub)" <gi...@apache.org> on 2025/07/29 13:12:29 UTC
[PR] KAFKA-15038: Replace topicIdByPartitionMap with MetadataCache instance in RemoteLogManager [kafka]
aheev opened a new pull request, #20256:
URL: https://github.com/apache/kafka/pull/20256
reopening https://github.com/apache/kafka/pull/14136. Rebased with latest trunk and accounted for new changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15038: Replace topicIdByPartitionMap with MetadataCache instance in RemoteLogManager [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20256:
URL: https://github.com/apache/kafka/pull/20256#issuecomment-3157293701
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15038: Replace topicIdByPartitionMap with MetadataCache instance in RemoteLogManager [kafka]
Posted by "aheev (via GitHub)" <gi...@apache.org>.
aheev commented on code in PR #20256:
URL: https://github.com/apache/kafka/pull/20256#discussion_r2304022058
##########
storage/src/test/java/org/apache/kafka/server/log/remote/storage/RemoteLogManagerTest.java:
##########
@@ -3746,18 +3738,6 @@ long findLogStartOffset(TopicIdPartition topicIdPartition, UnifiedLog log) {
verifyNoMoreInteractions(remoteStorageManager);
}
- @Test
- public void testIsPartitionReady() throws InterruptedException {
Review Comment:
This test checks if the `topicIds`(previous metadata cache in RLM) are cached in `onLeadershipChange`, which is no longer the case now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15038: Replace topicIdByPartitionMap with MetadataCache instance in RemoteLogManager [kafka]
Posted by "aheev (via GitHub)" <gi...@apache.org>.
aheev commented on PR #20256:
URL: https://github.com/apache/kafka/pull/20256#issuecomment-3205314494
@kamalcph found a new issue. Since metadataCache is [first updated](https://github.com/apache/kafka/blob/f81853ca8858fe1dfbb038ce39c8595a2a42c484/core/src/main/scala/kafka/server/metadata/BrokerMetadataPublisher.scala#L132-L150) before `ReplicaManager.applyDelta`, [stopPartitions](https://github.com/apache/kafka/blob/42644b3045eb0be3d8646496ca43dfaca75b6923/core/src/main/scala/kafka/server/ReplicaManager.scala#L2360), inturn `rlm.stopPartitions` is not able to [find topicId](https://github.com/aheev/kafka/blob/42644b3045eb0be3d8646496ca43dfaca75b6923/storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManager.java#L506). Why can't we just use `TopicIdPartition` in [StopPartition](https://github.com/apache/kafka/blob/5d2074e8f26a0573fd7d66cc4dfa68a9dae58dc4/server-common/src/main/java/org/apache/kafka/server/common/StopPartition.java#L26) class :thinking:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15038: Replace topicIdByPartitionMap with MetadataCache instance in RemoteLogManager [kafka]
Posted by "aheev (via GitHub)" <gi...@apache.org>.
aheev commented on code in PR #20256:
URL: https://github.com/apache/kafka/pull/20256#discussion_r2304022058
##########
storage/src/test/java/org/apache/kafka/server/log/remote/storage/RemoteLogManagerTest.java:
##########
@@ -3746,18 +3738,6 @@ long findLogStartOffset(TopicIdPartition topicIdPartition, UnifiedLog log) {
verifyNoMoreInteractions(remoteStorageManager);
}
- @Test
- public void testIsPartitionReady() throws InterruptedException {
Review Comment:
This test checks if the `topicIdByPartitionMap` is cached in `onLeadershipChange`, which is no longer the case now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15038: Replace topicIdByPartitionMap with MetadataCache instance in RemoteLogManager [kafka]
Posted by "aheev (via GitHub)" <gi...@apache.org>.
aheev commented on PR #20256:
URL: https://github.com/apache/kafka/pull/20256#issuecomment-3195408795
will look at the test failures today
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15038: Replace topicIdByPartitionMap with MetadataCache instance in RemoteLogManager [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20256:
URL: https://github.com/apache/kafka/pull/20256#issuecomment-3186663045
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15038: Replace topicIdByPartitionMap with MetadataCache instance in RemoteLogManager [kafka]
Posted by "kamalcph (via GitHub)" <gi...@apache.org>.
kamalcph commented on code in PR #20256:
URL: https://github.com/apache/kafka/pull/20256#discussion_r2296603590
##########
storage/src/test/java/org/apache/kafka/server/log/remote/storage/RemoteLogManagerTest.java:
##########
@@ -3746,18 +3738,6 @@ long findLogStartOffset(TopicIdPartition topicIdPartition, UnifiedLog log) {
verifyNoMoreInteractions(remoteStorageManager);
}
- @Test
- public void testIsPartitionReady() throws InterruptedException {
Review Comment:
Why this test was removed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15038: Replace topicIdByPartitionMap with MetadataCache instance in RemoteLogManager [kafka]
Posted by "kamalcph (via GitHub)" <gi...@apache.org>.
kamalcph commented on PR #20256:
URL: https://github.com/apache/kafka/pull/20256#issuecomment-3132891415
@aheev
Thanks for the PR! Could you add a comment to understand why PR#14136 was abandoned and how this PR addresses it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15038: Replace topicIdByPartitionMap with MetadataCache instance in RemoteLogManager [kafka]
Posted by "aheev (via GitHub)" <gi...@apache.org>.
aheev commented on code in PR #20256:
URL: https://github.com/apache/kafka/pull/20256#discussion_r2319418482
##########
storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManager.java:
##########
@@ -490,8 +479,8 @@ public void onLeadershipChange(Set<TopicPartitionLog> partitionsBecomeLeader,
public void stopLeaderCopyRLMTasks(Set<TopicPartitionLog> partitions) {
for (TopicPartitionLog partition : partitions) {
TopicPartition tp = partition.topicPartition();
- if (topicIdByPartitionMap.containsKey(tp)) {
- TopicIdPartition tpId = new TopicIdPartition(topicIdByPartitionMap.get(tp), tp);
+ if (metadataCache.contains(tp)) {
Review Comment:
Yes. Any ETA for when that would be introduced? If there a parent ticket, I can create a sub-ticket for Remote storage components under it
##########
core/src/main/scala/kafka/server/ConfigHandler.scala:
##########
@@ -90,12 +90,14 @@ class TopicConfigHandler(private val replicaManager: ReplicaManager,
val stopPartitions: java.util.HashSet[StopPartition] = new java.util.HashSet[StopPartition]()
leaderPartitions.foreach(partition => {
// delete remote logs and stop RemoteLogMetadataManager
- stopPartitions.add(new StopPartition(partition.topicPartition, false, true, true))
+ stopPartitions.add(
+ new StopPartition(new TopicIdPartition(partition.topicId.get, partition.topicPartition), false, true, true))
Review Comment:
Also, in future, TopicIdPartition would be added in TopicPartitionLog right?
##########
core/src/main/scala/kafka/server/ConfigHandler.scala:
##########
@@ -90,12 +90,14 @@ class TopicConfigHandler(private val replicaManager: ReplicaManager,
val stopPartitions: java.util.HashSet[StopPartition] = new java.util.HashSet[StopPartition]()
leaderPartitions.foreach(partition => {
// delete remote logs and stop RemoteLogMetadataManager
- stopPartitions.add(new StopPartition(partition.topicPartition, false, true, true))
+ stopPartitions.add(
+ new StopPartition(new TopicIdPartition(partition.topicId.get, partition.topicPartition), false, true, true))
Review Comment:
Curious. When do we get an empty topicId?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15038: Replace topicIdByPartitionMap with MetadataCache instance in RemoteLogManager [kafka]
Posted by "aheev (via GitHub)" <gi...@apache.org>.
aheev commented on PR #20256:
URL: https://github.com/apache/kafka/pull/20256#issuecomment-3178879527
@kamalcph gentle ping for the review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15038: Replace topicIdByPartitionMap with MetadataCache instance in RemoteLogManager [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20256:
URL: https://github.com/apache/kafka/pull/20256#issuecomment-3212913051
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15038: Replace topicIdByPartitionMap with MetadataCache instance in RemoteLogManager [kafka]
Posted by "kamalcph (via GitHub)" <gi...@apache.org>.
kamalcph commented on code in PR #20256:
URL: https://github.com/apache/kafka/pull/20256#discussion_r2307889453
##########
storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManager.java:
##########
@@ -490,8 +479,8 @@ public void onLeadershipChange(Set<TopicPartitionLog> partitionsBecomeLeader,
public void stopLeaderCopyRLMTasks(Set<TopicPartitionLog> partitions) {
for (TopicPartitionLog partition : partitions) {
TopicPartition tp = partition.topicPartition();
- if (topicIdByPartitionMap.containsKey(tp)) {
- TopicIdPartition tpId = new TopicIdPartition(topicIdByPartitionMap.get(tp), tp);
+ if (metadataCache.contains(tp)) {
Review Comment:
This `if` check can be removed once TopicIdPartition gets introduced in the TopicPartitionLog. This can be taken up in the next / separate PR.
##########
core/src/main/scala/kafka/server/ReplicaManager.scala:
##########
@@ -2347,6 +2347,7 @@ class ReplicaManager(val config: KafkaConfig,
// Before taking the lock, compute the local changes
val localChanges = delta.localChanges(config.nodeId)
val metadataVersion = newImage.features().metadataVersionOrThrow()
+ val deletedTopicIdMap = delta.image().topicNameToIdView()
Review Comment:
nit:
Can this be renamed to `deletedTopicIdMap` -> `topicNamesToIds` for clarity?
##########
core/src/main/scala/kafka/server/ConfigHandler.scala:
##########
@@ -90,12 +90,14 @@ class TopicConfigHandler(private val replicaManager: ReplicaManager,
val stopPartitions: java.util.HashSet[StopPartition] = new java.util.HashSet[StopPartition]()
leaderPartitions.foreach(partition => {
// delete remote logs and stop RemoteLogMetadataManager
- stopPartitions.add(new StopPartition(partition.topicPartition, false, true, true))
+ stopPartitions.add(
+ new StopPartition(new TopicIdPartition(partition.topicId.get, partition.topicPartition), false, true, true))
Review Comment:
To rule out empty topicId, can we take the topicId from the metadataCache itself?
```
val topicId = replicaManager.metadataCache.getTopicId(partition.topic)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15038: Replace topicIdByPartitionMap with MetadataCache instance in RemoteLogManager [kafka]
Posted by "aheev (via GitHub)" <gi...@apache.org>.
aheev commented on PR #20256:
URL: https://github.com/apache/kafka/pull/20256#issuecomment-3136837389
> @aheev
>
> Thanks for the PR! Could you add a comment to understand why PR#14136 was abandoned and how this PR addresses it?
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15038: Replace topicIdByPartitionMap with MetadataCache instance in RemoteLogManager [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20256:
URL: https://github.com/apache/kafka/pull/20256#issuecomment-3204053717
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15038: Replace topicIdByPartitionMap with MetadataCache instance in RemoteLogManager [kafka]
Posted by "aheev (via GitHub)" <gi...@apache.org>.
aheev commented on PR #20256:
URL: https://github.com/apache/kafka/pull/20256#issuecomment-3229075386
> Thanks for the patch!
>
> > Since metadataCache is [first updated](https://github.com/apache/kafka/blob/f81853ca8858fe1dfbb038ce39c8595a2a42c484/core/src/main/scala/kafka/server/metadata/BrokerMetadataPublisher.scala#L132-L150) before ReplicaManager.applyDelta, [stopPartitions](https://github.com/apache/kafka/blob/42644b3045eb0be3d8646496ca43dfaca75b6923/core/src/main/scala/kafka/server/ReplicaManager.scala#L2360), inturn rlm.stopPartitions is not able to [find topicId](https://github.com/aheev/kafka/blob/42644b3045eb0be3d8646496ca43dfaca75b6923/storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManager.java#L506). Why can't we just use TopicIdPartition in [StopPartition](https://github.com/apache/kafka/blob/5d2074e8f26a0573fd7d66cc4dfa68a9dae58dc4/server-common/src/main/java/org/apache/kafka/server/common/StopPartition.java#L26) class
>
> yeah, I think topicId can be set in the StopPartition object to propagate the value to the RemoteLogManager. The deletedTopicIds is available in the TopicsDelta.
Replaced `TopicPartition` with `TopicIdPartition` in `StopPartition`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15038: Replace topicIdByPartitionMap with MetadataCache instance in RemoteLogManager [kafka]
Posted by "aheev (via GitHub)" <gi...@apache.org>.
aheev commented on PR #20256:
URL: https://github.com/apache/kafka/pull/20256#issuecomment-3132875940
@divijvaidya @kamalcph could you please take a look at this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15038: Replace topicIdByPartitionMap with MetadataCache instance in RemoteLogManager [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20256:
URL: https://github.com/apache/kafka/pull/20256#issuecomment-3704416050
This PR has been closed since it has not had any activity in 120 days. If you feel like this
was a mistake, or you would like to continue working on it, please feel free to re-open the
PR and ask for a review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15038: Replace topicIdByPartitionMap with MetadataCache instance in RemoteLogManager [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #20256: KAFKA-15038: Replace topicIdByPartitionMap with MetadataCache instance in RemoteLogManager
URL: https://github.com/apache/kafka/pull/20256
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15038: Replace topicIdByPartitionMap with MetadataCache instance in RemoteLogManager [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20256:
URL: https://github.com/apache/kafka/pull/20256#issuecomment-3604953241
This PR is being marked as stale since it has not had any activity in 90 days. If you
would like to keep this PR alive, please leave a comment asking for a review. If the PR has
merge conflicts, update it with the latest from the base branch.
<p>
If you are having difficulty finding a reviewer, please reach out on the
[mailing list](https://kafka.apache.org/contact).
<p>
If this PR is no longer valid or desired, please feel free to close it. If no activity
occurs in the next 30 days, it will be automatically closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15038: Replace topicIdByPartitionMap with MetadataCache instance in RemoteLogManager [kafka]
Posted by "kamalcph (via GitHub)" <gi...@apache.org>.
kamalcph commented on code in PR #20256:
URL: https://github.com/apache/kafka/pull/20256#discussion_r2307900595
##########
core/src/main/scala/kafka/server/ReplicaManager.scala:
##########
@@ -2347,6 +2347,7 @@ class ReplicaManager(val config: KafkaConfig,
// Before taking the lock, compute the local changes
val localChanges = delta.localChanges(config.nodeId)
val metadataVersion = newImage.features().metadataVersionOrThrow()
+ val deletedTopicIdMap = delta.image().topicNameToIdView()
Review Comment:
nit:
Can this be renamed to `deletedTopicIdMap` -> `topicNamesToIds` or `topicIdMap`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15038: Replace topicIdByPartitionMap with MetadataCache instance in RemoteLogManager [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #20256:
URL: https://github.com/apache/kafka/pull/20256#issuecomment-3217261664
@aheev could you please rebase code to include CI changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org