[jira] [Commented] (KAFKA-9216) Enforce connect internal topic configuration at startup

["Randall Hauch (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-9216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17127789#comment-17127789 ] 

Randall Hauch commented on KAFKA-9216:
--------------------------------------

[~kkonstantine], thanks for reviewing and merging [~EeveeB]'s PR to check the partition count of the connector configs topic. I've created a followup PR (https://github.com/apache/kafka/pull/8828) that verifies that each internal topic has the `cleanup.policy=compact`; the worker fails to start if the cleanup policy is any other value (e.g., `delete`, `delete,compact` or `compact,delete`). The logic will avoid the check if the worker created the topic, since the topic is always created with `cleanup.policy=compact`.

Note that I chose to avoid checking retention, since that only applies if the `delete` cleanup policy is used. Therefore, the check described in the preceding paragraph should be sufficient. 

> Enforce connect internal topic configuration at startup
> -------------------------------------------------------
>
>                 Key: KAFKA-9216
>                 URL: https://issues.apache.org/jira/browse/KAFKA-9216
>             Project: Kafka
>          Issue Type: Improvement
>          Components: KafkaConnect
>    Affects Versions: 0.11.0.0
>            Reporter: Randall Hauch
>            Assignee: Evelyn Bayes
>            Priority: Major
>             Fix For: 2.3.2, 2.6.0, 2.4.2, 2.5.1
>
>
> Users sometimes configure Connect's internal topic for configurations with more than one partition. One partition is expected, however, and using more than one leads to weird behavior that is sometimes not easy to spot.
> Here's one example of a log message:
> {noformat}
> "textPayload": "[2019-11-20 11:12:14,049] INFO [Worker clientId=connect-1, groupId=td-connect-server] Current config state offset 284 does not match group assignment 274. Forcing rebalance. (org.apache.kafka.connect.runtime.distributed.DistributedHerder:942)\n"
> {noformat}
> Would it be possible to add a check in the KafkaConfigBackingStore and prevent the worker from starting if connect config partition count !=1 ?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)