You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Jordan Moore (JIRA)" <ji...@apache.org> on 2018/10/25 17:05:00 UTC
[jira] [Commented] (KAFKA-6890) Add connector level configurability
for producer/consumer client configs
[ https://issues.apache.org/jira/browse/KAFKA-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664027#comment-16664027 ]
Jordan Moore commented on KAFKA-6890:
-------------------------------------
[~rhauch], nice to meet you at Kafka Summit,
When we spoke, I had mentioned that we currently have one large connect cluster, primarily for S3 Connect, in each of our development environments, and for some of the 100+ connector configurations that we have loaded, the topics they are sinking have variable amounts of throughput, so I think it would be beneficial to be able to tune at least some of the properties for those APIs.
I know you brought up the concerns about at least being able to set "consumer.bootstrap.servers", for example, so perhaps setting "bootstrap.servers" (and other, similar properties, such as SSL certs for those servers) could be blacklisted in some way?
Also, as I mentioned on GitHub, this looks like it might duplicate KAFKA-4159
> Add connector level configurability for producer/consumer client configs
> ------------------------------------------------------------------------
>
> Key: KAFKA-6890
> URL: https://issues.apache.org/jira/browse/KAFKA-6890
> Project: Kafka
> Issue Type: New Feature
> Components: KafkaConnect
> Reporter: Allen Tang
> Priority: Minor
>
> Right now, each source connector and sink connector inherit their client configurations from the worker properties. Within the worker properties, all configurations that have a prefix of "producer." or "consumer." are applied to all source connectors and sink connectors respectively.
> We should also provide connector-level overrides whereby connector properties that are prefixed with "producer." and "consumer." are used to feed into the producer and consumer clients embedded within source and sink connectors respectively. The prefixes will be removed via a String#substring() call, and the remainder of the connector property key will be used as the client configuration key. The value is fed directly to the client as the configuration value.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)