[jira] [Commented] (KAFKA-6890) Add connector level configurability for producer/consumer client configs

["Randall Hauch (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16671976#comment-16671976 ] 

Randall Hauch commented on KAFKA-6890:
--------------------------------------

First of all, we're going to need a KIP for this, since it will change the configurations exposed to users. We need to make sure that this doesn't break backward compatibility, etc.

Second, I like this change and the ability to override some of the producer/consumer behavior. I also think that generally it's better to give users more control, but I'm actually very concerned about allowing connector configurations to override {{bootstrap servers}}. Yes, I can totally see the benefits, but this also gives users that don't know what they're doing to really mess things up. Some people will undoubtably think they *have* to define these properties in every connector, and then when they change their bootstrap URL in the worker they might forget it in their connector configs. Or, worse yet, having the internal topics in one cluster and the connector records in a different cluster will very much conflict with any hopes of adding EOS to source connectors.

> Add connector level configurability for producer/consumer client configs
> ------------------------------------------------------------------------
>
>                 Key: KAFKA-6890
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6890
>             Project: Kafka
>          Issue Type: New Feature
>          Components: KafkaConnect
>            Reporter: Allen Tang
>            Priority: Minor
>
> Right now, each source connector and sink connector inherit their client configurations from the worker properties. Within the worker properties, all configurations that have a prefix of "producer." or "consumer." are applied to all source connectors and sink connectors respectively.
> We should also provide connector-level overrides whereby connector properties that are prefixed with "producer." and "consumer." are used to feed into the producer and consumer clients embedded within source and sink connectors respectively. The prefixes will be removed via a String#substring() call, and the remainder of the connector property key will be used as the client configuration key. The value is fed directly to the client as the configuration value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)