[jira] [Commented] (KAFKA-15041) Source Connector auto topic creation fails when topic is deleted and brokers don't support auto topic creation

["Sagar Rao (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17727894#comment-17727894 ] 

Sagar Rao commented on KAFKA-15041:
-----------------------------------

Ideally a worker restart should fix this but it didn't quite happen on my local testing. 

> Source Connector auto topic creation fails when topic is deleted and brokers don't support auto topic creation
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-15041
>                 URL: https://issues.apache.org/jira/browse/KAFKA-15041
>             Project: Kafka
>          Issue Type: Bug
>          Components: KafkaConnect
>            Reporter: Sagar Rao
>            Assignee: Sagar Rao
>            Priority: Major
>
> [KIP-158|https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics]  allows the source connectors to create topics even when the broker doesn't allow to do so. It does so by checking for every record if a topic needs to be created [https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractWorkerSourceTask.java#L500.] To not always keep checking for topic presence via admin topics, it also maintains a cache of the topics that it has created and doesn't create those anymore. This helps to create topics when brokers don't support automatic topic creation.
> However, lets say the topic gets created initially and later on gets deleted while the connector is still running and the brokers don't support automatic topic creation. For such cases, the connector has cached the topic it has already created and wouldn't recreate it because the cache never updates and since the broker doesn't support topic creation, the logs would just be full of messages like 
> ```
>  Error while fetching metadata with correlation id 3260 : \{connect-test=UNKNOWN_TOPIC_OR_PARTITION} 
> ```
>  
> This can become a problem on enviroments where brokers don't allow topic creation. We need a way to refresh the topics cache for such cases.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)