You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Joe San <co...@gmail.com> on 2016/01/20 13:19:10 UTC

Create Kafka Topic Programatically

Kafka Users,

How can I create a kafka topic programatically?

I would like to create the topics when I initialize my application. It
should also be in such a way that if the topic already exists, the
initialization code should do nothing!

Thanks and Regards,
Joe

Re: Create Kafka Topic Programatically

Posted by Tommy Becker <to...@tivo.com>.

Awesome, glad to see that's being worked on!

On 01/20/2016 12:39 PM, Joel Koshy wrote:

https://issues.apache.org/jira/browse/KAFKA-2945 and related jiras is
probably what you are looking for. That is planned for the next release.

On Wed, Jan 20, 2016 at 8:41 AM, Tommy Becker <to...@tivo.com> wrote:

This works, but it's clumsy and has limitations. Unfortunately, I'm not
aware of any alternatives. We do some programmatic topic creation when we
want to create a topic with a non-default configuration. But specifying the
configuration is an all or nothing affair. For example, there's no way to
say create this topic using all the defaults except enable log compaction.
Worse, I don't think there's a way to get what the default topic
configuration is from the client. I'd like to see some improvements here.

On 01/20/2016 08:24 AM, Timo Ahokas wrote:

Hi Joe,

We're doing a similar thing that you're looking for with some of our app
nodes. We use the kafka.admin.AdminTools and its
topicExists()/createTopic() methods. Some additional code (e.g. Kafka
context check/creation) might be needed for dev/test environments if you're
using a specific Kafka context instead of the root.

-Timo

On 20 January 2016 at 23:14, Joe San <co...@gmail.com><mailto:
codeintheopen@gmail.com><ma...@gmail.com> wrote:

I doubt that might be enough. Could you tell me if the
auto.create.topics.enable satisfies the following requirement?

1. I want to create a topic with a specific name
2. If I restart the producer client, if the topic with that name already
exists, it should do nothing and use the topic as is
3. Upon producer restart, if the topic already exists, and if the topic has
un consumed messages, it should not be deleted

Thanks and Regards,
Joe

On Wed, Jan 20, 2016 at 2:11 PM, Franco Giacosa <fg...@gmail.com>

<ma...@gmail.com>

wrote:

Hi Joe,

There is an option in the producer called auto.create.topics.enable, so

the

producer can just start sending data to a topic and the topic will be
created with the default values.

2016-01-20 13:19 GMT+01:00 Joe San <co...@gmail.com><mailto:
codeintheopen@gmail.com><ma...@gmail.com>:

Kafka Users,

How can I create a kafka topic programatically?

I would like to create the topics when I initialize my application. It
should also be in such a way that if the topic already exists, the
initialization code should do nothing!

Thanks and Regards,
Joe

--
Tommy Becker
Senior Software Engineer

Digitalsmiths
A TiVo Company

www.digitalsmiths.com<http://www.digitalsmiths.com><http://www.digitalsmiths.com><http://www.digitalsmiths.com>
tobecker@tivo.com<ma...@tivo.com>

________________________________

This email and any attachments may contain confidential and privileged
material for the sole use of the intended recipient. Any review, copying,
or distribution of this email (or any attachments) by others is prohibited.
If you are not the intended recipient, please contact the sender
immediately and permanently delete this email and any attachments. No
employee or agent of TiVo Inc. is authorized to conclude any binding
agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
Inc. may only be made by a signed written agreement.

--
Tommy Becker
Senior Software Engineer

Digitalsmiths
A TiVo Company

www.digitalsmiths.com<http://www.digitalsmiths.com>
tobecker@tivo.com<ma...@tivo.com>

________________________________

This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of TiVo Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo Inc. may only be made by a signed written agreement.

Re: Create Kafka Topic Programatically

Posted by Joel Koshy <jj...@gmail.com>.

https://issues.apache.org/jira/browse/KAFKA-2945 and related jiras is
probably what you are looking for. That is planned for the next release.

On Wed, Jan 20, 2016 at 8:41 AM, Tommy Becker <to...@tivo.com> wrote:

> This works, but it's clumsy and has limitations. Unfortunately, I'm not
> aware of any alternatives. We do some programmatic topic creation when we
> want to create a topic with a non-default configuration. But specifying the
> configuration is an all or nothing affair. For example, there's no way to
> say create this topic using all the defaults except enable log compaction.
> Worse, I don't think there's a way to get what the default topic
> configuration is from the client. I'd like to see some improvements here.
>
> On 01/20/2016 08:24 AM, Timo Ahokas wrote:
>
> Hi Joe,
>
> We're doing a similar thing that you're looking for with some of our app
> nodes. We use the kafka.admin.AdminTools and its
> topicExists()/createTopic() methods. Some additional code (e.g. Kafka
> context check/creation) might be needed for dev/test environments if you're
> using a specific Kafka context instead of the root.
>
> -Timo
>
> On 20 January 2016 at 23:14, Joe San <co...@gmail.com><mailto:
> codeintheopen@gmail.com> wrote:
>
>
>
> I doubt that might be enough. Could you tell me if the
> auto.create.topics.enable satisfies the following requirement?
>
> 1. I want to create a topic with a specific name
> 2. If I restart the producer client, if the topic with that name already
> exists, it should do nothing and use the topic as is
> 3. Upon producer restart, if the topic already exists, and if the topic has
> un consumed messages, it should not be deleted
>
> Thanks and Regards,
> Joe
>
> On Wed, Jan 20, 2016 at 2:11 PM, Franco Giacosa <fgiacosa@gmail.com
> ><ma...@gmail.com>
> wrote:
>
>
>
> Hi Joe,
>
> There is an option in the producer called auto.create.topics.enable, so
>
>
> the
>
>
> producer can just start sending data to a topic and the topic will be
> created with the default values.
>
>
>
> 2016-01-20 13:19 GMT+01:00 Joe San <co...@gmail.com><mailto:
> codeintheopen@gmail.com>:
>
>
>
> Kafka Users,
>
> How can I create a kafka topic programatically?
>
> I would like to create the topics when I initialize my application. It
> should also be in such a way that if the topic already exists, the
> initialization code should do nothing!
>
> Thanks and Regards,
> Joe
>
>
>
>
>
>
>
>
>
>
>
>
> --
> Tommy Becker
> Senior Software Engineer
>
> Digitalsmiths
> A TiVo Company
>
> www.digitalsmiths.com<http://www.digitalsmiths.com>
> tobecker@tivo.com<ma...@tivo.com>
>
> ________________________________
>
> This email and any attachments may contain confidential and privileged
> material for the sole use of the intended recipient. Any review, copying,
> or distribution of this email (or any attachments) by others is prohibited.
> If you are not the intended recipient, please contact the sender
> immediately and permanently delete this email and any attachments. No
> employee or agent of TiVo Inc. is authorized to conclude any binding
> agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> Inc. may only be made by a signed written agreement.
>

Re: Create Kafka Topic Programatically

Posted by Tommy Becker <to...@tivo.com>.

This works, but it's clumsy and has limitations. Unfortunately, I'm not aware of any alternatives. We do some programmatic topic creation when we want to create a topic with a non-default configuration. But specifying the configuration is an all or nothing affair. For example, there's no way to say create this topic using all the defaults except enable log compaction. Worse, I don't think there's a way to get what the default topic configuration is from the client. I'd like to see some improvements here.

On 01/20/2016 08:24 AM, Timo Ahokas wrote:

Hi Joe,

-Timo

On 20 January 2016 at 23:14, Joe San <co...@gmail.com> wrote:

I doubt that might be enough. Could you tell me if the
auto.create.topics.enable satisfies the following requirement?

Thanks and Regards,
Joe

On Wed, Jan 20, 2016 at 2:11 PM, Franco Giacosa <fg...@gmail.com>
wrote:

Hi Joe,

There is an option in the producer called auto.create.topics.enable, so

the

producer can just start sending data to a topic and the topic will be
created with the default values.

2016-01-20 13:19 GMT+01:00 Joe San <co...@gmail.com>:

Kafka Users,

How can I create a kafka topic programatically?

I would like to create the topics when I initialize my application. It
should also be in such a way that if the topic already exists, the
initialization code should do nothing!

Thanks and Regards,
Joe

--
Tommy Becker
Senior Software Engineer

Digitalsmiths
A TiVo Company

www.digitalsmiths.com<http://www.digitalsmiths.com>
tobecker@tivo.com<ma...@tivo.com>

________________________________

This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of TiVo Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo Inc. may only be made by a signed written agreement.

Re: Create Kafka Topic Programatically

Posted by Timo Ahokas <ti...@gmail.com>.

Hi Joe,

We're doing a similar thing that you're looking for with some of our app
nodes. We use the kafka.admin.AdminTools and its
topicExists()/createTopic() methods. Some additional code (e.g. Kafka
context check/creation) might be needed for dev/test environments if you're
using a specific Kafka context instead of the root.

-Timo

On 20 January 2016 at 23:14, Joe San <co...@gmail.com> wrote:

> I doubt that might be enough. Could you tell me if the
> auto.create.topics.enable satisfies the following requirement?
>
> 1. I want to create a topic with a specific name
> 2. If I restart the producer client, if the topic with that name already
> exists, it should do nothing and use the topic as is
> 3. Upon producer restart, if the topic already exists, and if the topic has
> un consumed messages, it should not be deleted
>
> Thanks and Regards,
> Joe
>
> On Wed, Jan 20, 2016 at 2:11 PM, Franco Giacosa <fg...@gmail.com>
> wrote:
>
> > Hi Joe,
> >
> > There is an option in the producer called auto.create.topics.enable, so
> the
> > producer can just start sending data to a topic and the topic will be
> > created with the default values.
> >
> >
> >
> > 2016-01-20 13:19 GMT+01:00 Joe San <co...@gmail.com>:
> >
> > > Kafka Users,
> > >
> > > How can I create a kafka topic programatically?
> > >
> > > I would like to create the topics when I initialize my application. It
> > > should also be in such a way that if the topic already exists, the
> > > initialization code should do nothing!
> > >
> > > Thanks and Regards,
> > > Joe
> > >
> >
>

Re: Create Kafka Topic Programatically

Posted by Damian Guy <da...@gmail.com>.

Hi,

I believe it is a broker property.

It will create the topic with the name you provide.
The topic will not get deleted unless you manually delete it.
It wont get re-created on subsequent calls (unless you've deleted it)

HTH,
Damian

On 20 January 2016 at 13:14, Joe San <co...@gmail.com> wrote:

> I doubt that might be enough. Could you tell me if the
> auto.create.topics.enable satisfies the following requirement?
>
> 1. I want to create a topic with a specific name
> 2. If I restart the producer client, if the topic with that name already
> exists, it should do nothing and use the topic as is
> 3. Upon producer restart, if the topic already exists, and if the topic has
> un consumed messages, it should not be deleted
>
> Thanks and Regards,
> Joe
>
> On Wed, Jan 20, 2016 at 2:11 PM, Franco Giacosa <fg...@gmail.com>
> wrote:
>
> > Hi Joe,
> >
> > There is an option in the producer called auto.create.topics.enable, so
> the
> > producer can just start sending data to a topic and the topic will be
> > created with the default values.
> >
> >
> >
> > 2016-01-20 13:19 GMT+01:00 Joe San <co...@gmail.com>:
> >
> > > Kafka Users,
> > >
> > > How can I create a kafka topic programatically?
> > >
> > > I would like to create the topics when I initialize my application. It
> > > should also be in such a way that if the topic already exists, the
> > > initialization code should do nothing!
> > >
> > > Thanks and Regards,
> > > Joe
> > >
> >
>

Re: Create Kafka Topic Programatically

Posted by Joe San <co...@gmail.com>.

I doubt that might be enough. Could you tell me if the
auto.create.topics.enable satisfies the following requirement?

1. I want to create a topic with a specific name
2. If I restart the producer client, if the topic with that name already
exists, it should do nothing and use the topic as is
3. Upon producer restart, if the topic already exists, and if the topic has
un consumed messages, it should not be deleted

Thanks and Regards,
Joe

On Wed, Jan 20, 2016 at 2:11 PM, Franco Giacosa <fg...@gmail.com> wrote:

> Hi Joe,
>
> There is an option in the producer called auto.create.topics.enable, so the
> producer can just start sending data to a topic and the topic will be
> created with the default values.
>
>
>
> 2016-01-20 13:19 GMT+01:00 Joe San <co...@gmail.com>:
>
> > Kafka Users,
> >
> > How can I create a kafka topic programatically?
> >
> > I would like to create the topics when I initialize my application. It
> > should also be in such a way that if the topic already exists, the
> > initialization code should do nothing!
> >
> > Thanks and Regards,
> > Joe
> >
>

Re: Create Kafka Topic Programatically

Posted by Franco Giacosa <fg...@gmail.com>.

Hi Joe,

There is an option in the producer called auto.create.topics.enable, so the
producer can just start sending data to a topic and the topic will be
created with the default values.



2016-01-20 13:19 GMT+01:00 Joe San <co...@gmail.com>:

> Kafka Users,
>
> How can I create a kafka topic programatically?
>
> I would like to create the topics when I initialize my application. It
> should also be in such a way that if the topic already exists, the
> initialization code should do nothing!
>
> Thanks and Regards,
> Joe
>