[jira] [Commented] (KAFKA-10340) Source connectors should report error when trying to produce records to non-existent topics instead of hanging forever

["Chris Egerton (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-10340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276498#comment-17276498 ] 

Chris Egerton commented on KAFKA-10340:
---------------------------------------

The logging changes are certainly an improvement, but don't address the underlying issue: source tasks and their producers hang and remain active (i.e., are not closed) indefinitely, even after the task is scheduled for shutdown. If several task instances are scheduled for shutdown and then recreated (which may happen as a result of rebalance or connector reconfiguration), any of those instances in this bad state will keep sending metadata requests to the broker from their producers, and any resources allocated by the {{SourceTask}} instances will not be freed.

This creates a resource leak and the impact is substantial enough that I believe the ticket should be reopened until it has been addressed with a functional fix.

> Source connectors should report error when trying to produce records to non-existent topics instead of hanging forever
> ----------------------------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-10340
>                 URL: https://issues.apache.org/jira/browse/KAFKA-10340
>             Project: Kafka
>          Issue Type: Bug
>          Components: KafkaConnect
>            Reporter: Arjun Satish
>            Assignee: Luke Chen
>            Priority: Major
>             Fix For: 2.7.0, 2.6.1
>
>
> Currently, a source connector will blindly attempt to write a record to a Kafka topic. When the topic does not exist, its creation is controlled by the {{auto.create.topics.enable}} config on the brokers. When auto.create is disabled, the producer.send() call on the Connect worker will hang indefinitely (due to the "infinite retries" configuration for said producer). In setups where this config is usually disabled, the source connector simply appears to hang and not produce any output.
> It is desirable to either log an info or an error message (or inform the user somehow) that the connector is simply stuck waiting for the destination topic to be created. When the worker has permissions to inspect the broker settings, it can use the {{listTopics}} and {{describeConfigs}} API in AdminClient to check if the topic exists, the broker can {{auto.create.topics.enable}} topics, and if these cases do not exist, either throw an error.
> With the recently merged [KIP-158|https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics], this becomes even more specific a corner case: when topic creation settings are enabled, the worker should handle the corner case where topic creation is disabled, {{auto.create.topics.enable}} is set to false and topic does not exist.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)