[jira] [Commented] (KAFKA-9216) Enforce connect internal topic configuration at startup

["Greg Harris (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-9216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111394#comment-17111394 ] 

Greg Harris commented on KAFKA-9216:
------------------------------------

I believe that KAFKA-5087 has an intent similar to this issue, but was never implemented, and erroneously closed as a duplicate.

It does call out the need to enforce that the topics are log-compacted rather than time- or size-deleted in order to prevent loss of state, which I think would be a valuable addition to the validation suggested here.

One additional aspect that isn't covered yet is that the retention period of the topics should also be infinite, since loss of state due to long-running connectors is undesirable.

I think that any one of these issues (config partition count != 1, cleanup policy != compact, or retention != infinite) should generate an exception during startup, to prevent the worker from operating on top of a lossy state-store.

> Enforce connect internal topic configuration at startup
> -------------------------------------------------------
>
>                 Key: KAFKA-9216
>                 URL: https://issues.apache.org/jira/browse/KAFKA-9216
>             Project: Kafka
>          Issue Type: Improvement
>          Components: KafkaConnect
>    Affects Versions: 0.11.0.0
>            Reporter: Randall Hauch
>            Assignee: Evelyn Bayes
>            Priority: Major
>
> Users sometimes configure Connect's internal topic for configurations with more than one partition. One partition is expected, however, and using more than one leads to weird behavior that is sometimes not easy to spot.
> Here's one example of a log message:
> {noformat}
> "textPayload": "[2019-11-20 11:12:14,049] INFO [Worker clientId=connect-1, groupId=td-connect-server] Current config state offset 284 does not match group assignment 274. Forcing rebalance. (org.apache.kafka.connect.runtime.distributed.DistributedHerder:942)\n"
> {noformat}
> Would it be possible to add a check in the KafkaConfigBackingStore and prevent the worker from starting if connect config partition count !=1 ?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)