You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Anton Liauchuk <an...@gmail.com> on 2025/09/02 08:44:27 UTC

Re: [DISCUSS] KIP-1209: Add configuration to control internal topic creation in Kafka Connect

Hi,

I finalized the implementation https://github.com/apache/kafka/pull/20384:
if auto-creation of topics is disabled, an exception is thrown when a
required topic does not exist. If the topic exists, there is an additional
validation to ensure that it has the correct cleanup policy.

Please review the KIP
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1209%3A+Add+configuration+to+control+internal+topic+creation+in+Kafka+Connect
and PR. I’ll start the voting if there are no concerns

On Tue, Aug 26, 2025 at 4:58 PM Anton Liauchuk <an...@gmail.com> wrote:

> Hello everyone
>
> I'd like to discuss a KIP
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1209%3A+Add+configuration+to+control+internal+topic+creation+in+Kafka+Connect
>
> small POC with discussion: https://github.com/apache/kafka/pull/20384
>
> Motivation:
>
> Kafka Connect requires several internal topics (config, offsets, and
> status) for storing connector metadata and state. By default, these topics
> are automatically created by the worker if they are missing.
> However, in some scenarios this behavior can be problematic:
> - internal topics may already contain information about a large number of
> connectors.
> - if one or more of the original topics are accidentally removed or
> renamed, Kafka Connect will create them silently.
> - this can lead to unexpected states — e.g., if the config topic exists
> but the offsets topic was removed, all connectors may start from scratch.
> We are going to introduce safer mode where Kafka Connect fails fast
> instead of silently creating new topics, allowing to troubleshoot
> misconfigurations and prevent accidental data loss or resets.
>
> Thank you!
>

Re: [DISCUSS] KIP-1209: Add configuration to control internal topic creation in Kafka Connect

Posted by Anton Liauchuk <an...@gmail.com>.
Hi,
Thanks for the feedback.

AS1: Yes, the same worker.properties will be used for topic creation as for
the worker itself.
AS2: I will implement it. Thank you!


On Fri, Feb 27, 2026 at 5:15 PM Andrew Schofield <sc...@apache.org>
wrote:

> Hi Anton,
> Thanks for the KIP. Makes sense to me. Just two comments before I vote.
>
> AS1: KIP-1147 aligned the command-line properties of all of the
> command-line tools. The Connect tools are a little different than all of
> the other command-line tools, so I just want to make sure that I understand
> the way the new tools is supposed to be used. Consistency is good.
>
> I think what you have intended for the new tool is:
> bin/connect-internal-topics.sh create --worker-config
> <worker_properties_file>
>
> Here are the existing ones:
> bin/connect-standalone.sh <worker_properties_file>
> <connector.properties......>
> bin/connect-distributed.sh <worker_properties_file>
> bin/connect-plugin-path.sh list --worker-config <worker_properties_file>
>
> Is it the case that the same worker.properties will be used for the topic
> creation as for the worker itself? I think the answer is yes, but wanted to
> check.
>
> AS2: Please make sure that you implement -h and --help for the new tool,
> in a similar manner as bin/connect-plugin-path.sh.
>
>
> Thanks,
> Andrew
>
> On 2025/09/02 10:50:12 Anton Liauchuk wrote:
> > Thanks for your question.
> >
> > The *auto.create.topics.enable* configuration will remain a broker-level
> > setting. The new proposed configuration,* internal.topics.creation*, will
> > be a Kafka Connect–level setting.
> >
> > For Kafka Connect internal topics, there is a requirement that these
> topics
> > must have *cleanup.policy=compact*. In my proposal, the new
> > *internal.topics.creation* configuration takes precedence. Let’s review
> > some examples:
> >
> >
> >    1.
> >
> >    *Internal topics do not exist, and internal.topics.creation=false
> >    (regardless of auto.create.topics.enable=true/false)*
> >
> >    The Kafka Connect worker will not start, and an exception will be
> thrown.
> >    2.
> >
> >    *Internal topics do not exist, and internal.topics.creation=true
> >    (regardless of auto.create.topics.enable=true/false)*
> >
> >    Kafka Connect will attempt to create the required topics. If topic
> >    creation fails, the worker will not start - this is the current
> behavior in
> >    Kafka Connect.
> >    3.
> >
> >    *Internal topics already exist, and
> internal.topics.creation=true/false
> >    (regardless of auto.create.topics.enable=true/false)*
> >
> >    No difference in behavior: Kafka Connect will validate the cleanup
> >    policy of these topics, and if valid, the worker will start.
> >
> > So as you can see, the broker-level configuration does not affect the
> > behavior of the new configuration.
> >
> > On Tue, Sep 2, 2025 at 11:57 AM Kuan Po Tseng <br...@apache.org>
> wrote:
> >
> > > Hi Anton,
> > >
> > > I have a quick question: how does this setting interact with
> > > auto.create.topics.enable?
> > > If I set internal.topics.creation=false while
> auto.create.topics.enable is
> > > true,
> > > will the Connect internal topic still be created automatically?
> > >
> > > Best,
> > > Kuan-Po Tseng
> > >
> > > On 2025/09/02 08:44:27 Anton Liauchuk wrote:
> > > > Hi,
> > > >
> > > > I finalized the implementation
> > > https://github.com/apache/kafka/pull/20384:
> > > > if auto-creation of topics is disabled, an exception is thrown when a
> > > > required topic does not exist. If the topic exists, there is an
> > > additional
> > > > validation to ensure that it has the correct cleanup policy.
> > > >
> > > > Please review the KIP
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1209%3A+Add+configuration+to+control+internal+topic+creation+in+Kafka+Connect
> > > > and PR. I’ll start the voting if there are no concerns
> > > >
> > > > On Tue, Aug 26, 2025 at 4:58 PM Anton Liauchuk <anton93lev@gmail.com
> >
> > > wrote:
> > > >
> > > > > Hello everyone
> > > > >
> > > > > I'd like to discuss a KIP
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1209%3A+Add+configuration+to+control+internal+topic+creation+in+Kafka+Connect
> > > > >
> > > > > small POC with discussion:
> https://github.com/apache/kafka/pull/20384
> > > > >
> > > > > Motivation:
> > > > >
> > > > > Kafka Connect requires several internal topics (config, offsets,
> and
> > > > > status) for storing connector metadata and state. By default, these
> > > topics
> > > > > are automatically created by the worker if they are missing.
> > > > > However, in some scenarios this behavior can be problematic:
> > > > > - internal topics may already contain information about a large
> number
> > > of
> > > > > connectors.
> > > > > - if one or more of the original topics are accidentally removed or
> > > > > renamed, Kafka Connect will create them silently.
> > > > > - this can lead to unexpected states — e.g., if the config topic
> exists
> > > > > but the offsets topic was removed, all connectors may start from
> > > scratch.
> > > > > We are going to introduce safer mode where Kafka Connect fails fast
> > > > > instead of silently creating new topics, allowing to troubleshoot
> > > > > misconfigurations and prevent accidental data loss or resets.
> > > > >
> > > > > Thank you!
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1209: Add configuration to control internal topic creation in Kafka Connect

Posted by Andrew Schofield <sc...@apache.org>.
Hi Anton,
Thanks for the KIP. Makes sense to me. Just two comments before I vote.

AS1: KIP-1147 aligned the command-line properties of all of the command-line tools. The Connect tools are a little different than all of the other command-line tools, so I just want to make sure that I understand the way the new tools is supposed to be used. Consistency is good.

I think what you have intended for the new tool is:
bin/connect-internal-topics.sh create --worker-config <worker_properties_file>

Here are the existing ones:
bin/connect-standalone.sh <worker_properties_file> <connector.properties......>
bin/connect-distributed.sh <worker_properties_file>
bin/connect-plugin-path.sh list --worker-config <worker_properties_file>

Is it the case that the same worker.properties will be used for the topic creation as for the worker itself? I think the answer is yes, but wanted to check.

AS2: Please make sure that you implement -h and --help for the new tool, in a similar manner as bin/connect-plugin-path.sh.


Thanks,
Andrew

On 2025/09/02 10:50:12 Anton Liauchuk wrote:
> Thanks for your question.
> 
> The *auto.create.topics.enable* configuration will remain a broker-level
> setting. The new proposed configuration,* internal.topics.creation*, will
> be a Kafka Connect–level setting.
> 
> For Kafka Connect internal topics, there is a requirement that these topics
> must have *cleanup.policy=compact*. In my proposal, the new
> *internal.topics.creation* configuration takes precedence. Let’s review
> some examples:
> 
> 
>    1.
> 
>    *Internal topics do not exist, and internal.topics.creation=false
>    (regardless of auto.create.topics.enable=true/false)*
> 
>    The Kafka Connect worker will not start, and an exception will be thrown.
>    2.
> 
>    *Internal topics do not exist, and internal.topics.creation=true
>    (regardless of auto.create.topics.enable=true/false)*
> 
>    Kafka Connect will attempt to create the required topics. If topic
>    creation fails, the worker will not start - this is the current behavior in
>    Kafka Connect.
>    3.
> 
>    *Internal topics already exist, and internal.topics.creation=true/false
>    (regardless of auto.create.topics.enable=true/false)*
> 
>    No difference in behavior: Kafka Connect will validate the cleanup
>    policy of these topics, and if valid, the worker will start.
> 
> So as you can see, the broker-level configuration does not affect the
> behavior of the new configuration.
> 
> On Tue, Sep 2, 2025 at 11:57 AM Kuan Po Tseng <br...@apache.org> wrote:
> 
> > Hi Anton,
> >
> > I have a quick question: how does this setting interact with
> > auto.create.topics.enable?
> > If I set internal.topics.creation=false while auto.create.topics.enable is
> > true,
> > will the Connect internal topic still be created automatically?
> >
> > Best,
> > Kuan-Po Tseng
> >
> > On 2025/09/02 08:44:27 Anton Liauchuk wrote:
> > > Hi,
> > >
> > > I finalized the implementation
> > https://github.com/apache/kafka/pull/20384:
> > > if auto-creation of topics is disabled, an exception is thrown when a
> > > required topic does not exist. If the topic exists, there is an
> > additional
> > > validation to ensure that it has the correct cleanup policy.
> > >
> > > Please review the KIP
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1209%3A+Add+configuration+to+control+internal+topic+creation+in+Kafka+Connect
> > > and PR. I’ll start the voting if there are no concerns
> > >
> > > On Tue, Aug 26, 2025 at 4:58 PM Anton Liauchuk <an...@gmail.com>
> > wrote:
> > >
> > > > Hello everyone
> > > >
> > > > I'd like to discuss a KIP
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1209%3A+Add+configuration+to+control+internal+topic+creation+in+Kafka+Connect
> > > >
> > > > small POC with discussion: https://github.com/apache/kafka/pull/20384
> > > >
> > > > Motivation:
> > > >
> > > > Kafka Connect requires several internal topics (config, offsets, and
> > > > status) for storing connector metadata and state. By default, these
> > topics
> > > > are automatically created by the worker if they are missing.
> > > > However, in some scenarios this behavior can be problematic:
> > > > - internal topics may already contain information about a large number
> > of
> > > > connectors.
> > > > - if one or more of the original topics are accidentally removed or
> > > > renamed, Kafka Connect will create them silently.
> > > > - this can lead to unexpected states — e.g., if the config topic exists
> > > > but the offsets topic was removed, all connectors may start from
> > scratch.
> > > > We are going to introduce safer mode where Kafka Connect fails fast
> > > > instead of silently creating new topics, allowing to troubleshoot
> > > > misconfigurations and prevent accidental data loss or resets.
> > > >
> > > > Thank you!
> > > >
> > >
> >
> 

Re: [DISCUSS] KIP-1209: Add configuration to control internal topic creation in Kafka Connect

Posted by Anton Liauchuk <an...@gmail.com>.
Thanks for your question.

The *auto.create.topics.enable* configuration will remain a broker-level
setting. The new proposed configuration,* internal.topics.creation*, will
be a Kafka Connect–level setting.

For Kafka Connect internal topics, there is a requirement that these topics
must have *cleanup.policy=compact*. In my proposal, the new
*internal.topics.creation* configuration takes precedence. Let’s review
some examples:


   1.

   *Internal topics do not exist, and internal.topics.creation=false
   (regardless of auto.create.topics.enable=true/false)*

   The Kafka Connect worker will not start, and an exception will be thrown.
   2.

   *Internal topics do not exist, and internal.topics.creation=true
   (regardless of auto.create.topics.enable=true/false)*

   Kafka Connect will attempt to create the required topics. If topic
   creation fails, the worker will not start - this is the current behavior in
   Kafka Connect.
   3.

   *Internal topics already exist, and internal.topics.creation=true/false
   (regardless of auto.create.topics.enable=true/false)*

   No difference in behavior: Kafka Connect will validate the cleanup
   policy of these topics, and if valid, the worker will start.

So as you can see, the broker-level configuration does not affect the
behavior of the new configuration.

On Tue, Sep 2, 2025 at 11:57 AM Kuan Po Tseng <br...@apache.org> wrote:

> Hi Anton,
>
> I have a quick question: how does this setting interact with
> auto.create.topics.enable?
> If I set internal.topics.creation=false while auto.create.topics.enable is
> true,
> will the Connect internal topic still be created automatically?
>
> Best,
> Kuan-Po Tseng
>
> On 2025/09/02 08:44:27 Anton Liauchuk wrote:
> > Hi,
> >
> > I finalized the implementation
> https://github.com/apache/kafka/pull/20384:
> > if auto-creation of topics is disabled, an exception is thrown when a
> > required topic does not exist. If the topic exists, there is an
> additional
> > validation to ensure that it has the correct cleanup policy.
> >
> > Please review the KIP
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1209%3A+Add+configuration+to+control+internal+topic+creation+in+Kafka+Connect
> > and PR. I’ll start the voting if there are no concerns
> >
> > On Tue, Aug 26, 2025 at 4:58 PM Anton Liauchuk <an...@gmail.com>
> wrote:
> >
> > > Hello everyone
> > >
> > > I'd like to discuss a KIP
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1209%3A+Add+configuration+to+control+internal+topic+creation+in+Kafka+Connect
> > >
> > > small POC with discussion: https://github.com/apache/kafka/pull/20384
> > >
> > > Motivation:
> > >
> > > Kafka Connect requires several internal topics (config, offsets, and
> > > status) for storing connector metadata and state. By default, these
> topics
> > > are automatically created by the worker if they are missing.
> > > However, in some scenarios this behavior can be problematic:
> > > - internal topics may already contain information about a large number
> of
> > > connectors.
> > > - if one or more of the original topics are accidentally removed or
> > > renamed, Kafka Connect will create them silently.
> > > - this can lead to unexpected states — e.g., if the config topic exists
> > > but the offsets topic was removed, all connectors may start from
> scratch.
> > > We are going to introduce safer mode where Kafka Connect fails fast
> > > instead of silently creating new topics, allowing to troubleshoot
> > > misconfigurations and prevent accidental data loss or resets.
> > >
> > > Thank you!
> > >
> >
>

Re: [DISCUSS] KIP-1209: Add configuration to control internal topic creation in Kafka Connect

Posted by Kuan Po Tseng <br...@apache.org>.
Hi Anton,

I have a quick question: how does this setting interact with auto.create.topics.enable?
If I set internal.topics.creation=false while auto.create.topics.enable is true,
will the Connect internal topic still be created automatically?

Best,
Kuan-Po Tseng

On 2025/09/02 08:44:27 Anton Liauchuk wrote:
> Hi,
> 
> I finalized the implementation https://github.com/apache/kafka/pull/20384:
> if auto-creation of topics is disabled, an exception is thrown when a
> required topic does not exist. If the topic exists, there is an additional
> validation to ensure that it has the correct cleanup policy.
> 
> Please review the KIP
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1209%3A+Add+configuration+to+control+internal+topic+creation+in+Kafka+Connect
> and PR. I’ll start the voting if there are no concerns
> 
> On Tue, Aug 26, 2025 at 4:58 PM Anton Liauchuk <an...@gmail.com> wrote:
> 
> > Hello everyone
> >
> > I'd like to discuss a KIP
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1209%3A+Add+configuration+to+control+internal+topic+creation+in+Kafka+Connect
> >
> > small POC with discussion: https://github.com/apache/kafka/pull/20384
> >
> > Motivation:
> >
> > Kafka Connect requires several internal topics (config, offsets, and
> > status) for storing connector metadata and state. By default, these topics
> > are automatically created by the worker if they are missing.
> > However, in some scenarios this behavior can be problematic:
> > - internal topics may already contain information about a large number of
> > connectors.
> > - if one or more of the original topics are accidentally removed or
> > renamed, Kafka Connect will create them silently.
> > - this can lead to unexpected states — e.g., if the config topic exists
> > but the offsets topic was removed, all connectors may start from scratch.
> > We are going to introduce safer mode where Kafka Connect fails fast
> > instead of silently creating new topics, allowing to troubleshoot
> > misconfigurations and prevent accidental data loss or resets.
> >
> > Thank you!
> >
>