[jira] [Commented] (KAFKA-4667) Connect (and Stream?) should create internal topics with availability or consistency properly set

["Ewen Cheslack-Postava (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15836801#comment-15836801 ] 

Ewen Cheslack-Postava commented on KAFKA-4667:
----------------------------------------------

[~ecesena] Not sure I understand exactly -- Connect doesn't actually create its internal topics today. We'd been waiting for KIP-4 and the Java AdminClient support to avoid pulling in core + all its dependencies into the framework. So today, I'm not sure what state you're talking about getting into. If you have auto topic creation enabled, that's what should be taking effect. Otherwise, today we'd expect you to create those topics with the desired parameters.

One option would be to convert this to a JIRA for creating internal topics automatically, which doesn't seem to have a JIRA yet but is definitely something we want to do. Part of that would include config options to control what replication factor you want. # of partitions might be a setting for some (but for some internal topics, like config, we expect only a single partition).

> Connect (and Stream?) should create internal topics with availability or consistency properly set
> -------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-4667
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4667
>             Project: Kafka
>          Issue Type: Bug
>          Components: KafkaConnect
>            Reporter: Emanuele Cesena
>
> I'm reporting this as an issue but in fact it requires more investigation (which unfortunately I'm not able to perform at this time).
> Repro steps:
> - configure Kafka for consistency, for example:
> default.replication.factor=3
> min.insync.replicas=2
> unclean.leader.election.enable=false
> - run Connect for the first time, which should create its internal topics
> I believe these topics are created with the broker's default, in particular:
> min.insync.replicas=2
> unclean.leader.election.enable=false
> but connect doesn't produce with acks=all, which in turn may cause the cluster to go in a bad state (see, e.g., https://issues.apache.org/jira/browse/KAFKA-4666).
> Solution would be to force availability mode, i.e. force:
> unclean.leader.election.enable=true
> when creating the connect topics, or viceversa detect availability vs consistency mode and turn acks=all if needed.
> I assume the same happens with other kafka-based services such as streams.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)