You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "fujian (Jira)" <ji...@apache.org> on 2025/06/04 13:29:00 UTC
[jira] [Updated] (KAFKA-19371) When a broker restarts, it should not attempt to create the __remote_log_metadata topic if it already exists.
[ https://issues.apache.org/jira/browse/KAFKA-19371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
fujian updated KAFKA-19371:
---------------------------
Description:
*[Precondition]*
Kafka cluster already enabled the remote storage feature based on inner topic's implementation. The core inner topic "__remote_log_metadata" already created.
*[Steps]*
1. Restart one broker of the Kafka cluster.
2. Check the log and the code logic for the "__remote_log_metadata"s creating when broker restarting
*[Expect result]*
The broker shouldn't attempt to call API to create the topic due to that it already existed.
*[Actual result]*
The results are different which depend on the start process' duration for broker:
*Case 1: Happy Path when restarting take a short time*
[2025-06-03 22:35:11,648] INFO Topic __remote_log_metadata {color:#00875a}exists{color}. TopicId: 4CT2TTC-R6u7fNo_njYlDA, numPartitions: 50,
*Case 2: Unhappy path 1 when restarting take some time*
[2025-06-03 23:59:40,505] INFO Topic __remote_log_metadata{color:#de350b} does not exist{color}. Error: Timed out waiting for a node assignment. Call: listNodes
[2025-06-04 00:00:36,938] INFO Topic [__remote_log_metadata] {color:#de350b}already exists
{color}
*Case 3: Unhappy path 2 when restarting take a long time.*
[2025-06-03 21:57:21,151] INFO Topic __remote_log_metadata {color:#de350b}does not exist{color}. Error: {color:#de350b}Timed out waiting{color} for a node assignment. Call: {color:#de350b}listNodes {color}at
[2025-06-03 21:58:21,153] ERROR Encountered error while creating __remote_log_metadata topic. java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.{color:#de350b}TimeoutException{color}: Timed out waiting for a node assignment. Call: {color:#de350b}createTopics {color}at
From the log and current code. we can know that {color:#de350b}case 2 and case 3 both give the prompt "the topic does not exist" and try to call topic creating API. In actually. it is useless and contradict the fact that the topic already existed. Especially. the case 2's log prompt the topic existed and not existed at the same time.{color}
*[Root Cause analyst]*
After reviewing the related code (TopicBasedRemoteLogMetadataManager#doesTopicExist). It is one {color:#de350b}wrong implement{color} to judge one topic existed or not.
So let me create one PR to fix this minor bug. Thanks
FYI:
Why we got the timeout exception?
It is normal case based on the fact:
When restarting broker. The connection to query/create topic in "TopicBasedRemoteLogMetadataManager#initializeResources"will fail until the broker get ready.
[2025-06-03 23:21:20,752] WARN [AdminClient clientId=adminclient-1] Connection to node -1 ([10.20.1.125:9559)|https://10-20-1-125/] could not be established. Node may not be available.
[2025-06-03 23:21:21,282] INFO [BrokerServer id=2] Transition from STARTING to STARTED (kafka.server.BrokerServer)
was:
*[Precondition]*
Kafka cluster already enabled the remote storage feature based on inner topic's implementation. The core inner topic "__remote_log_metadata" already created.
*[Steps]*
1. Restart one broker of the Kafka cluster.
2. Check the log and the code logic for the "__remote_log_metadata"s creating when broker restarting
*[Expect result]*
The broker shouldn't attempt to call API to create the topic due to that it already existed.
*[Actual result]*
The results are different which depend on the start process' duration for broker:
*Case 1: Happy Path when restarting take a short time*
[2025-06-03 22:35:11,648] INFO Topic __remote_log_metadata {color:#00875a}exists{color}. TopicId: 4CT2TTC-R6u7fNo_njYlDA, numPartitions: 50,
*Case 2: Unhappy path 1 when restarting take some time*
[2025-06-03 23:59:40,505] INFO Topic __remote_log_metadata{color:#de350b} does not exist{color}. Error: Timed out waiting for a node assignment. Call: listNodes
[2025-06-04 00:00:36,938] INFO Topic [__remote_log_metadata] {color:#de350b}already exists
{color}
*Case 3: Unhappy path 2 when restarting take a long time.*
[2025-06-03 21:57:21,151] INFO Topic __remote_log_metadata {color:#de350b}does not exist{color}. Error: {color:#de350b}Timed out waiting{color} for a node assignment. Call: {color:#de350b}listNodes {color}at
[2025-06-03 21:58:21,153] ERROR Encountered error while creating __remote_log_metadata topic. java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.{color:#de350b}TimeoutException{color}: Timed out waiting for a node assignment. Call: {color:#de350b}createTopics {color}at
From the log and current code. we can know that {color:#de350b}case 2 and case 3 both give the prompt "the topic does not exist" and try to call topic creating API. In actually. it is useless and contradict the fact that the topic already existed. Especially. the case 2's log prompt the topic existed and not existed at the same time.{color}
[Root Cause analyst]
After reviewing the related code (TopicBasedRemoteLogMetadataManager#doesTopicExist). It is one {color:#de350b}wrong implement{color} to judge one topic existed or not.
So let me create one PR to fix this minor bug. Thanks
FYI:
Why we got the timeout exception?
It is normal case based on the fact:
When restarting broker. The connection to query/create topic in "TopicBasedRemoteLogMetadataManager#initializeResources"will fail until the broker get ready.
[2025-06-03 23:21:20,752] WARN [AdminClient clientId=adminclient-1] Connection to node -1 ([10.20.1.125:9559)|https://10-20-1-125/] could not be established. Node may not be available.
[2025-06-03 23:21:21,282] INFO [BrokerServer id=2] Transition from STARTING to STARTED (kafka.server.BrokerServer)
> When a broker restarts, it should not attempt to create the __remote_log_metadata topic if it already exists.
> -------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-19371
> URL: https://issues.apache.org/jira/browse/KAFKA-19371
> Project: Kafka
> Issue Type: Bug
> Components: Tiered-Storage
> Affects Versions: 3.9.0, 4.0.0
> Reporter: fujian
> Priority: Major
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> *[Precondition]*
> Kafka cluster already enabled the remote storage feature based on inner topic's implementation. The core inner topic "__remote_log_metadata" already created.
>
> *[Steps]*
> 1. Restart one broker of the Kafka cluster.
> 2. Check the log and the code logic for the "__remote_log_metadata"s creating when broker restarting
>
> *[Expect result]*
> The broker shouldn't attempt to call API to create the topic due to that it already existed.
>
> *[Actual result]*
> The results are different which depend on the start process' duration for broker:
> *Case 1: Happy Path when restarting take a short time*
> [2025-06-03 22:35:11,648] INFO Topic __remote_log_metadata {color:#00875a}exists{color}. TopicId: 4CT2TTC-R6u7fNo_njYlDA, numPartitions: 50,
> *Case 2: Unhappy path 1 when restarting take some time*
> [2025-06-03 23:59:40,505] INFO Topic __remote_log_metadata{color:#de350b} does not exist{color}. Error: Timed out waiting for a node assignment. Call: listNodes
> [2025-06-04 00:00:36,938] INFO Topic [__remote_log_metadata] {color:#de350b}already exists
> {color}
> *Case 3: Unhappy path 2 when restarting take a long time.*
> [2025-06-03 21:57:21,151] INFO Topic __remote_log_metadata {color:#de350b}does not exist{color}. Error: {color:#de350b}Timed out waiting{color} for a node assignment. Call: {color:#de350b}listNodes {color}at
> [2025-06-03 21:58:21,153] ERROR Encountered error while creating __remote_log_metadata topic. java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.{color:#de350b}TimeoutException{color}: Timed out waiting for a node assignment. Call: {color:#de350b}createTopics {color}at
>
> From the log and current code. we can know that {color:#de350b}case 2 and case 3 both give the prompt "the topic does not exist" and try to call topic creating API. In actually. it is useless and contradict the fact that the topic already existed. Especially. the case 2's log prompt the topic existed and not existed at the same time.{color}
>
> *[Root Cause analyst]*
> After reviewing the related code (TopicBasedRemoteLogMetadataManager#doesTopicExist). It is one {color:#de350b}wrong implement{color} to judge one topic existed or not.
> So let me create one PR to fix this minor bug. Thanks
>
> FYI:
> Why we got the timeout exception?
> It is normal case based on the fact:
> When restarting broker. The connection to query/create topic in "TopicBasedRemoteLogMetadataManager#initializeResources"will fail until the broker get ready.
> [2025-06-03 23:21:20,752] WARN [AdminClient clientId=adminclient-1] Connection to node -1 ([10.20.1.125:9559)|https://10-20-1-125/] could not be established. Node may not be available.
> [2025-06-03 23:21:21,282] INFO [BrokerServer id=2] Transition from STARTING to STARTED (kafka.server.BrokerServer)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)