[jira] [Commented] (KAFKA-10340) Source connectors should report error when trying to produce records to non-existent topics instead of hanging forever

["Randall Hauch (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-10340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17292993#comment-17292993 ] 

Randall Hauch commented on KAFKA-10340:
---------------------------------------

[~xakassi] wrote:

{quote}
And also I cannot understand, why we even have the property "topic.creation.enable" and do not create all topics by AdminClient? Isn't it a better approach to create topics always explicitly using AdminClient than to rely on auto creation?
{quote}

When we introduced KIP-158 to allow Connect to source connectors to specify the rules for creating topics, it was important that we maintained backward compatibility. Many people upgrade Kafka and Connect simply to get fixes, and do not want new features they don't know about nor wish to use enabled by default. This is an essential aspect of all KIPs and AK features, even when by default most users *probably* would want to use the feature.  

{quote}
According to KIP-158 we have an option to set replication.factor and partitions to -1 to use the broker's default value. So why we cannot always create topics by AdminClient and use these broker's default values in case connector does not have any "topic.creation" properties?

As far as I understand this will completely solve the problem described in this ticket.
{quote}
It actually does not solve the problem. If Connect were to have the behavior you suggest, then we'd still run into the issue when the broker has autocreation of topics disabled *and* the particular source connector did not specify the topic creation rules.

The bottom line is that many things would be made easier *for some people* if we could easily change defaults, but upgrading would become painfully difficult for those that don't wish to use those new features or behaviors. By maintaining backward compatibility, we dramatically reduce the overhead and risk of upgrades to get fixes, which is an important characteristic that the AK community cherishes.

We will have the option in 3.0 to *change* defaults and behaviors -- it is a major release that may not be entirely backward compatible. Obviously, we'll want to not break things unnecessarily, but changing the default `topic.creation.enable = true` will be one such change we are very much looking forward to making.

> Source connectors should report error when trying to produce records to non-existent topics instead of hanging forever
> ----------------------------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-10340
>                 URL: https://issues.apache.org/jira/browse/KAFKA-10340
>             Project: Kafka
>          Issue Type: Bug
>          Components: KafkaConnect
>    Affects Versions: 2.5.1, 2.7.0, 2.6.1, 2.8.0
>            Reporter: Arjun Satish
>            Assignee: Chris Egerton
>            Priority: Major
>             Fix For: 3.0.0, 2.7.1
>
>
> Currently, a source connector will blindly attempt to write a record to a Kafka topic. When the topic does not exist, its creation is controlled by the {{auto.create.topics.enable}} config on the brokers. When auto.create is disabled, the producer.send() call on the Connect worker will hang indefinitely (due to the "infinite retries" configuration for said producer). In setups where this config is usually disabled, the source connector simply appears to hang and not produce any output.
> It is desirable to either log an info or an error message (or inform the user somehow) that the connector is simply stuck waiting for the destination topic to be created. When the worker has permissions to inspect the broker settings, it can use the {{listTopics}} and {{describeConfigs}} API in AdminClient to check if the topic exists, the broker can {{auto.create.topics.enable}} topics, and if these cases do not exist, either throw an error.
> With the recently merged [KIP-158|https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics], this becomes even more specific a corner case: when topic creation settings are enabled, the worker should handle the corner case where topic creation is disabled, {{auto.create.topics.enable}} is set to false and topic does not exist.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)