You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Anton Liauchuk <an...@gmail.com> on 2025/09/02 08:44:27 UTC
Re: [DISCUSS] KIP-1209: Add configuration to control internal topic creation in Kafka Connect
Hi,
I finalized the implementation https://github.com/apache/kafka/pull/20384:
if auto-creation of topics is disabled, an exception is thrown when a
required topic does not exist. If the topic exists, there is an additional
validation to ensure that it has the correct cleanup policy.
Please review the KIP
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1209%3A+Add+configuration+to+control+internal+topic+creation+in+Kafka+Connect
and PR. I’ll start the voting if there are no concerns
On Tue, Aug 26, 2025 at 4:58 PM Anton Liauchuk <an...@gmail.com> wrote:
> Hello everyone
>
> I'd like to discuss a KIP
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1209%3A+Add+configuration+to+control+internal+topic+creation+in+Kafka+Connect
>
> small POC with discussion: https://github.com/apache/kafka/pull/20384
>
> Motivation:
>
> Kafka Connect requires several internal topics (config, offsets, and
> status) for storing connector metadata and state. By default, these topics
> are automatically created by the worker if they are missing.
> However, in some scenarios this behavior can be problematic:
> - internal topics may already contain information about a large number of
> connectors.
> - if one or more of the original topics are accidentally removed or
> renamed, Kafka Connect will create them silently.
> - this can lead to unexpected states — e.g., if the config topic exists
> but the offsets topic was removed, all connectors may start from scratch.
> We are going to introduce safer mode where Kafka Connect fails fast
> instead of silently creating new topics, allowing to troubleshoot
> misconfigurations and prevent accidental data loss or resets.
>
> Thank you!
>
Re: [DISCUSS] KIP-1209: Add configuration to control internal topic creation in Kafka Connect
Posted by Anton Liauchuk <an...@gmail.com>.
Hi,
Thanks for the feedback.
AS1: Yes, the same worker.properties will be used for topic creation as for
the worker itself.
AS2: I will implement it. Thank you!
On Fri, Feb 27, 2026 at 5:15 PM Andrew Schofield <sc...@apache.org>
wrote:
> Hi Anton,
> Thanks for the KIP. Makes sense to me. Just two comments before I vote.
>
> AS1: KIP-1147 aligned the command-line properties of all of the
> command-line tools. The Connect tools are a little different than all of
> the other command-line tools, so I just want to make sure that I understand
> the way the new tools is supposed to be used. Consistency is good.
>
> I think what you have intended for the new tool is:
> bin/connect-internal-topics.sh create --worker-config
> <worker_properties_file>
>
> Here are the existing ones:
> bin/connect-standalone.sh <worker_properties_file>
> <connector.properties......>
> bin/connect-distributed.sh <worker_properties_file>
> bin/connect-plugin-path.sh list --worker-config <worker_properties_file>
>
> Is it the case that the same worker.properties will be used for the topic
> creation as for the worker itself? I think the answer is yes, but wanted to
> check.
>
> AS2: Please make sure that you implement -h and --help for the new tool,
> in a similar manner as bin/connect-plugin-path.sh.
>
>
> Thanks,
> Andrew
>
> On 2025/09/02 10:50:12 Anton Liauchuk wrote:
> > Thanks for your question.
> >
> > The *auto.create.topics.enable* configuration will remain a broker-level
> > setting. The new proposed configuration,* internal.topics.creation*, will
> > be a Kafka Connect–level setting.
> >
> > For Kafka Connect internal topics, there is a requirement that these
> topics
> > must have *cleanup.policy=compact*. In my proposal, the new
> > *internal.topics.creation* configuration takes precedence. Let’s review
> > some examples:
> >
> >
> > 1.
> >
> > *Internal topics do not exist, and internal.topics.creation=false
> > (regardless of auto.create.topics.enable=true/false)*
> >
> > The Kafka Connect worker will not start, and an exception will be
> thrown.
> > 2.
> >
> > *Internal topics do not exist, and internal.topics.creation=true
> > (regardless of auto.create.topics.enable=true/false)*
> >
> > Kafka Connect will attempt to create the required topics. If topic
> > creation fails, the worker will not start - this is the current
> behavior in
> > Kafka Connect.
> > 3.
> >
> > *Internal topics already exist, and
> internal.topics.creation=true/false
> > (regardless of auto.create.topics.enable=true/false)*
> >
> > No difference in behavior: Kafka Connect will validate the cleanup
> > policy of these topics, and if valid, the worker will start.
> >
> > So as you can see, the broker-level configuration does not affect the
> > behavior of the new configuration.
> >
> > On Tue, Sep 2, 2025 at 11:57 AM Kuan Po Tseng <br...@apache.org>
> wrote:
> >
> > > Hi Anton,
> > >
> > > I have a quick question: how does this setting interact with
> > > auto.create.topics.enable?
> > > If I set internal.topics.creation=false while
> auto.create.topics.enable is
> > > true,
> > > will the Connect internal topic still be created automatically?
> > >
> > > Best,
> > > Kuan-Po Tseng
> > >
> > > On 2025/09/02 08:44:27 Anton Liauchuk wrote:
> > > > Hi,
> > > >
> > > > I finalized the implementation
> > > https://github.com/apache/kafka/pull/20384:
> > > > if auto-creation of topics is disabled, an exception is thrown when a
> > > > required topic does not exist. If the topic exists, there is an
> > > additional
> > > > validation to ensure that it has the correct cleanup policy.
> > > >
> > > > Please review the KIP
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1209%3A+Add+configuration+to+control+internal+topic+creation+in+Kafka+Connect
> > > > and PR. I’ll start the voting if there are no concerns
> > > >
> > > > On Tue, Aug 26, 2025 at 4:58 PM Anton Liauchuk <anton93lev@gmail.com
> >
> > > wrote:
> > > >
> > > > > Hello everyone
> > > > >
> > > > > I'd like to discuss a KIP
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1209%3A+Add+configuration+to+control+internal+topic+creation+in+Kafka+Connect
> > > > >
> > > > > small POC with discussion:
> https://github.com/apache/kafka/pull/20384
> > > > >
> > > > > Motivation:
> > > > >
> > > > > Kafka Connect requires several internal topics (config, offsets,
> and
> > > > > status) for storing connector metadata and state. By default, these
> > > topics
> > > > > are automatically created by the worker if they are missing.
> > > > > However, in some scenarios this behavior can be problematic:
> > > > > - internal topics may already contain information about a large
> number
> > > of
> > > > > connectors.
> > > > > - if one or more of the original topics are accidentally removed or
> > > > > renamed, Kafka Connect will create them silently.
> > > > > - this can lead to unexpected states — e.g., if the config topic
> exists
> > > > > but the offsets topic was removed, all connectors may start from
> > > scratch.
> > > > > We are going to introduce safer mode where Kafka Connect fails fast
> > > > > instead of silently creating new topics, allowing to troubleshoot
> > > > > misconfigurations and prevent accidental data loss or resets.
> > > > >
> > > > > Thank you!
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-1209: Add configuration to control internal topic creation in Kafka Connect
Posted by Andrew Schofield <sc...@apache.org>.
Hi Anton,
Thanks for the KIP. Makes sense to me. Just two comments before I vote.
AS1: KIP-1147 aligned the command-line properties of all of the command-line tools. The Connect tools are a little different than all of the other command-line tools, so I just want to make sure that I understand the way the new tools is supposed to be used. Consistency is good.
I think what you have intended for the new tool is:
bin/connect-internal-topics.sh create --worker-config <worker_properties_file>
Here are the existing ones:
bin/connect-standalone.sh <worker_properties_file> <connector.properties......>
bin/connect-distributed.sh <worker_properties_file>
bin/connect-plugin-path.sh list --worker-config <worker_properties_file>
Is it the case that the same worker.properties will be used for the topic creation as for the worker itself? I think the answer is yes, but wanted to check.
AS2: Please make sure that you implement -h and --help for the new tool, in a similar manner as bin/connect-plugin-path.sh.
Thanks,
Andrew
On 2025/09/02 10:50:12 Anton Liauchuk wrote:
> Thanks for your question.
>
> The *auto.create.topics.enable* configuration will remain a broker-level
> setting. The new proposed configuration,* internal.topics.creation*, will
> be a Kafka Connect–level setting.
>
> For Kafka Connect internal topics, there is a requirement that these topics
> must have *cleanup.policy=compact*. In my proposal, the new
> *internal.topics.creation* configuration takes precedence. Let’s review
> some examples:
>
>
> 1.
>
> *Internal topics do not exist, and internal.topics.creation=false
> (regardless of auto.create.topics.enable=true/false)*
>
> The Kafka Connect worker will not start, and an exception will be thrown.
> 2.
>
> *Internal topics do not exist, and internal.topics.creation=true
> (regardless of auto.create.topics.enable=true/false)*
>
> Kafka Connect will attempt to create the required topics. If topic
> creation fails, the worker will not start - this is the current behavior in
> Kafka Connect.
> 3.
>
> *Internal topics already exist, and internal.topics.creation=true/false
> (regardless of auto.create.topics.enable=true/false)*
>
> No difference in behavior: Kafka Connect will validate the cleanup
> policy of these topics, and if valid, the worker will start.
>
> So as you can see, the broker-level configuration does not affect the
> behavior of the new configuration.
>
> On Tue, Sep 2, 2025 at 11:57 AM Kuan Po Tseng <br...@apache.org> wrote:
>
> > Hi Anton,
> >
> > I have a quick question: how does this setting interact with
> > auto.create.topics.enable?
> > If I set internal.topics.creation=false while auto.create.topics.enable is
> > true,
> > will the Connect internal topic still be created automatically?
> >
> > Best,
> > Kuan-Po Tseng
> >
> > On 2025/09/02 08:44:27 Anton Liauchuk wrote:
> > > Hi,
> > >
> > > I finalized the implementation
> > https://github.com/apache/kafka/pull/20384:
> > > if auto-creation of topics is disabled, an exception is thrown when a
> > > required topic does not exist. If the topic exists, there is an
> > additional
> > > validation to ensure that it has the correct cleanup policy.
> > >
> > > Please review the KIP
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1209%3A+Add+configuration+to+control+internal+topic+creation+in+Kafka+Connect
> > > and PR. I’ll start the voting if there are no concerns
> > >
> > > On Tue, Aug 26, 2025 at 4:58 PM Anton Liauchuk <an...@gmail.com>
> > wrote:
> > >
> > > > Hello everyone
> > > >
> > > > I'd like to discuss a KIP
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1209%3A+Add+configuration+to+control+internal+topic+creation+in+Kafka+Connect
> > > >
> > > > small POC with discussion: https://github.com/apache/kafka/pull/20384
> > > >
> > > > Motivation:
> > > >
> > > > Kafka Connect requires several internal topics (config, offsets, and
> > > > status) for storing connector metadata and state. By default, these
> > topics
> > > > are automatically created by the worker if they are missing.
> > > > However, in some scenarios this behavior can be problematic:
> > > > - internal topics may already contain information about a large number
> > of
> > > > connectors.
> > > > - if one or more of the original topics are accidentally removed or
> > > > renamed, Kafka Connect will create them silently.
> > > > - this can lead to unexpected states — e.g., if the config topic exists
> > > > but the offsets topic was removed, all connectors may start from
> > scratch.
> > > > We are going to introduce safer mode where Kafka Connect fails fast
> > > > instead of silently creating new topics, allowing to troubleshoot
> > > > misconfigurations and prevent accidental data loss or resets.
> > > >
> > > > Thank you!
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-1209: Add configuration to control internal topic creation in Kafka Connect
Posted by Anton Liauchuk <an...@gmail.com>.
Thanks for your question.
The *auto.create.topics.enable* configuration will remain a broker-level
setting. The new proposed configuration,* internal.topics.creation*, will
be a Kafka Connect–level setting.
For Kafka Connect internal topics, there is a requirement that these topics
must have *cleanup.policy=compact*. In my proposal, the new
*internal.topics.creation* configuration takes precedence. Let’s review
some examples:
1.
*Internal topics do not exist, and internal.topics.creation=false
(regardless of auto.create.topics.enable=true/false)*
The Kafka Connect worker will not start, and an exception will be thrown.
2.
*Internal topics do not exist, and internal.topics.creation=true
(regardless of auto.create.topics.enable=true/false)*
Kafka Connect will attempt to create the required topics. If topic
creation fails, the worker will not start - this is the current behavior in
Kafka Connect.
3.
*Internal topics already exist, and internal.topics.creation=true/false
(regardless of auto.create.topics.enable=true/false)*
No difference in behavior: Kafka Connect will validate the cleanup
policy of these topics, and if valid, the worker will start.
So as you can see, the broker-level configuration does not affect the
behavior of the new configuration.
On Tue, Sep 2, 2025 at 11:57 AM Kuan Po Tseng <br...@apache.org> wrote:
> Hi Anton,
>
> I have a quick question: how does this setting interact with
> auto.create.topics.enable?
> If I set internal.topics.creation=false while auto.create.topics.enable is
> true,
> will the Connect internal topic still be created automatically?
>
> Best,
> Kuan-Po Tseng
>
> On 2025/09/02 08:44:27 Anton Liauchuk wrote:
> > Hi,
> >
> > I finalized the implementation
> https://github.com/apache/kafka/pull/20384:
> > if auto-creation of topics is disabled, an exception is thrown when a
> > required topic does not exist. If the topic exists, there is an
> additional
> > validation to ensure that it has the correct cleanup policy.
> >
> > Please review the KIP
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1209%3A+Add+configuration+to+control+internal+topic+creation+in+Kafka+Connect
> > and PR. I’ll start the voting if there are no concerns
> >
> > On Tue, Aug 26, 2025 at 4:58 PM Anton Liauchuk <an...@gmail.com>
> wrote:
> >
> > > Hello everyone
> > >
> > > I'd like to discuss a KIP
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1209%3A+Add+configuration+to+control+internal+topic+creation+in+Kafka+Connect
> > >
> > > small POC with discussion: https://github.com/apache/kafka/pull/20384
> > >
> > > Motivation:
> > >
> > > Kafka Connect requires several internal topics (config, offsets, and
> > > status) for storing connector metadata and state. By default, these
> topics
> > > are automatically created by the worker if they are missing.
> > > However, in some scenarios this behavior can be problematic:
> > > - internal topics may already contain information about a large number
> of
> > > connectors.
> > > - if one or more of the original topics are accidentally removed or
> > > renamed, Kafka Connect will create them silently.
> > > - this can lead to unexpected states — e.g., if the config topic exists
> > > but the offsets topic was removed, all connectors may start from
> scratch.
> > > We are going to introduce safer mode where Kafka Connect fails fast
> > > instead of silently creating new topics, allowing to troubleshoot
> > > misconfigurations and prevent accidental data loss or resets.
> > >
> > > Thank you!
> > >
> >
>
Re: [DISCUSS] KIP-1209: Add configuration to control internal topic creation in Kafka Connect
Posted by Kuan Po Tseng <br...@apache.org>.
Hi Anton,
I have a quick question: how does this setting interact with auto.create.topics.enable?
If I set internal.topics.creation=false while auto.create.topics.enable is true,
will the Connect internal topic still be created automatically?
Best,
Kuan-Po Tseng
On 2025/09/02 08:44:27 Anton Liauchuk wrote:
> Hi,
>
> I finalized the implementation https://github.com/apache/kafka/pull/20384:
> if auto-creation of topics is disabled, an exception is thrown when a
> required topic does not exist. If the topic exists, there is an additional
> validation to ensure that it has the correct cleanup policy.
>
> Please review the KIP
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1209%3A+Add+configuration+to+control+internal+topic+creation+in+Kafka+Connect
> and PR. I’ll start the voting if there are no concerns
>
> On Tue, Aug 26, 2025 at 4:58 PM Anton Liauchuk <an...@gmail.com> wrote:
>
> > Hello everyone
> >
> > I'd like to discuss a KIP
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1209%3A+Add+configuration+to+control+internal+topic+creation+in+Kafka+Connect
> >
> > small POC with discussion: https://github.com/apache/kafka/pull/20384
> >
> > Motivation:
> >
> > Kafka Connect requires several internal topics (config, offsets, and
> > status) for storing connector metadata and state. By default, these topics
> > are automatically created by the worker if they are missing.
> > However, in some scenarios this behavior can be problematic:
> > - internal topics may already contain information about a large number of
> > connectors.
> > - if one or more of the original topics are accidentally removed or
> > renamed, Kafka Connect will create them silently.
> > - this can lead to unexpected states — e.g., if the config topic exists
> > but the offsets topic was removed, all connectors may start from scratch.
> > We are going to introduce safer mode where Kafka Connect fails fast
> > instead of silently creating new topics, allowing to troubleshoot
> > misconfigurations and prevent accidental data loss or resets.
> >
> > Thank you!
> >
>