[jira] [Comment Edited] (KAFKA-7509) Kafka Connect logs unnecessary warnings about unused configurations

["Randall Hauch (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-7509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673834#comment-16673834 ] 

Randall Hauch edited comment on KAFKA-7509 at 11/27/18 9:12 PM:
----------------------------------------------------------------

Since changing the unused property log messages to debug was not the correct change, I closed that PR and took yet another approach that just filters out worker properties before passing them to the producer, consumer, and admin clients. Previously, Connect would pass many of its worker configuration properties (including the `{{producer.}}` and `{{consumer.}}` properties for connectors) into the producer, consumer, and admin client used for *internal* topics, resulting in lots of log warnings about unused config properties.

With [this new approach|https://github.com/apache/kafka/pull/5876], Connect now attempts to filter out the worker’s configuration properties that are not applicable to the producer, consumer, or admin client used for internal topics. (Connect is already including only producer and consumer properties when creating clients for connectors, since those properties are prefixed in the worker config.)

For the most part, this is relatively straightforward, since there are some top-level worker-specific properties that can be easily removed, and most extension-specific properties have Connect-specific properties. Unfortunately, the recently added REST extension is the only type of connect extension that uses unprefixed properties from the worker config, so it is not possible to remove those from the properties passed to the producer, consumer, and admin clients since we can't distinguish between a REST extension property and a property for a client interceptor, metric reporter, etc. Hopefully, REST extensions are not prevalent yet and most users may not see any warnings about unused properties in the producer, consumer, and admin client; but if a user does use a REST extension with extension-specific properties, the user may also see WARN log messages about how they're not known to the clients.

We also want to ensure that we don't pass any top-level properties for the consumer or admin client to the producer used for internal topics; producer and admin client properties to the consumer for internal topics; and producer and consumer properties to the admin client. This is a bit trickier, since there are a lot of common configs between these different types of clients, and we don't want to be remove a property if it's actually used by that client. Note that any property that is unknown (e.g., properties for REST extension, interceptors, metric reporters, serdes, partitioners, etc.) must be passed to the producer, consumer, and admin client. All of these — except for the REST extension properties — should indeed be used by the producer and consumer. But, since the admin client only supports metric reporters, any properties for interceptors, serdes, partitioners and REST extension will also be logged as unused. This is about the best we can do at this point.

All of this filtering logic was added to the {{ConnectUtils}} class, allowing the logic to be encapsulated and easily unit tested. Also, all changes are limited to Kafka Connect, and will work with all client and Connect extensions (passing them to the clients if they are unknown).

This still does not go as far as I'd like. It'd be ideal if Connect could use a RecordingMap to know which properties were used across _all_ of the clients and the worker itself, and to then log _only those_ properties that were never used at all. This is beyond the scope of this issue, however, and can be addressed in a follow-on issue that will require changes to AbstractConfig, Producer, Consumer, and AdminClient so that these components don't record unused components and instead we allow Connect to record all unused properties once.


was (Author: rhauch):
Since changing the unused property log messages to debug was not the correct change, I closed that PR and took yet another approach that just filters out worker properties before passing them to the producer, consumer, and admin clients. Previously, Connect would pass many of its worker configuration properties into the producer, consumer, and admin client used for internal topics, resulting in lots of log warnings about unused config properties.

With [this change|https://github.com/apache/kafka/pull/5876], Connect now attempts to filter out the worker’s configuration properties that are not applicable to the producer, consumer, or admin client used for internal topics. (Connect is already including only producer and consumer properties when creating those clients for connectors, since those properties are prefixed in the worker config.)

For the most part, this is relatively straightforward, since there are some top-level worker-specific properties that can be removed, and most extension-specific properties have Connect-specific properties. Unfortunately, the REST extension is the only type of connect extension that uses unprefixed properties from the worker config, so any it is not possible to remove those from the properties passed to the producer, consumer, and admin clients. Hopefully, REST extensions are prevalant yet, and this will mean most users may not see any warnings about unused properties in the producer, consumer, and admin client.

Removing the Connect worker configs is one step. The other is for the producer to remove the any properties that are specific to the consumer and admin client. Likewise, for the consumer we have to remove any properties that are specific to the producer and admin client, and for the admin client remove any properties that are specific to the producer and consumer. Note that any property that is unknown (e.g., properties for REST extension, interceptors, metric reporters, serdes, partitioners, etc.) must be passed to the producer, consumer, and admin client. All of these — except for the REST extension properties — should indeed be used by the producer and consumer. But, since the admin client only supports metric reporters, any properties for interceptors, serdes, partitioners and REST extension will also be logged as unused. This is about the best we can do at this point.

All of this filtering logic was added to the ConnectUtils class, allowing the logic to be easily unit tested. All changes are limited to Kafka Connect, and will work with all client and Connect extensions (passing them to the clients if they are unknown).

This still does not go as far as I'd like. It'd be ideal if Connect could use a RecordingMap to know which properties were used across _all_ of the clients and the worker itself, and to then log _only those_ properties that were never used at all. This is beyond the scope of this issue, however, and can be addressed in a follow-on issue that will require changes to AbstractConfig.

> Kafka Connect logs unnecessary warnings about unused configurations
> -------------------------------------------------------------------
>
>                 Key: KAFKA-7509
>                 URL: https://issues.apache.org/jira/browse/KAFKA-7509
>             Project: Kafka
>          Issue Type: Improvement
>          Components: clients, KafkaConnect
>    Affects Versions: 0.10.2.0
>            Reporter: Randall Hauch
>            Assignee: Randall Hauch
>            Priority: Major
>
> When running Connect, the logs contain quite a few warnings about "The configuration '{}' was supplied but isn't a known config." This occurs when Connect creates producers, consumers, and admin clients, because the AbstractConfig is logging unused configuration properties upon construction. It's complicated by the fact that the Producer, Consumer, and AdminClient all create their own AbstractConfig instances within the constructor, so we can't even call its {{ignore(String key)}} method.
> See also KAFKA-6793 for a similar issue with Streams.
> There are no arguments in the Producer, Consumer, or AdminClient constructors to control  whether the configs log these warnings, so a simpler workaround is to only pass those configuration properties to the Producer, Consumer, and AdminClient that the ProducerConfig, ConsumerConfig, and AdminClientConfig configdefs know about.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)