Managing topic configuration w/ auto.create.topics.enable

[Mathieu Fenniak <ma...@replicon.com>]

Hey Kafka Users,

When using a Kafka broker w/ auto.create.topics.enable set to true, how do
Kafka users generally manage configuration of those topics?  In
particular, cleanup.policy={compact/delete} can be a crucial configuration
value to get correct.

In my application, I have a couple Kafka Connect sources that generate all
the input to my Kafka brokers, across about 30 different topics.  All of
these specific topics are used in a KStreams app as tables, so they should
be configured w/ cleanup.policy=compact so that state-store restoration can
take place on them without losing data.  As far as I know, there's no way
for a Kafka producer (or more specifically a KC source) to indicate topic
configuration if/when the topic is auto-created, so it needs to be done
afterwards.

Other topics, such as KStreams repartition topics, should be
cleanup.policy=delete, and others such as KStream changelog topics should
be cleanup.policy=compact... but KStreams handles all this for me for
internal topics.

I'm envisioning the ideal solution as something like a scheduled task that
runs against my cluster, with a configuration, that identifies and corrects
misconfigurations.  This could then roll out through different staging and
testing areas just like a general software change.  But developing such a
tool doesn't seem straight-forward since there currently isn't an API for
managing topic configuration either...

Mathieu