Dynamic topic subscription without using pattern

[chintan mavawala <ch...@gmail.com>]

Hi,

We have a use case where we need to start kafka consumer with a fixed list
of kafka topics and add more topics on the fly. Since there is no pattern
in names of topics, using pattern for dynamic subscriptions of topics is
not feasible.

Is it a good practice to subscribe to topics on given kafka consumer on fly
using after starting consumer?

I understand that KafkaConsumer is not thread safe. Any ill effects due to
multi threaded env?

Regards,
Chintan

Re: Dynamic topic subscription without using pattern

[Skip Montanaro <sk...@gmail.com>]

I don't think you'll know for sure until you try. You will have to properly
protect your consumer, as it's not thread safe. (I asked recently in a
situation where all I wanted to do from a separate thread was commit().)

Seems like an easy experiment to run. You have a producer which posts to
two topics, A and B, say, once every ten seconds. Your consumer listens to
both, and prints out the partition and offset for each message as well as
the current time. Another "new" topic is subscribed to, but is initially
quiescent.

After a minute or so, have the producer start publishing to topic C, and
post a "C" message to the "new" topic. Your consumer picks that up, and
subscribes to A, B and C. What happens? Do you miss any messages on A or B?
It there any apparent delay? How much? Do you get the first message on C?
Now repeat the experiment with shorter intervals, say 100ms. Does the
behavior change? My guess is that this will all work. Now try with your
proposed multithreaded setup.

This is about a "ten line" producer and consumer in Python. I assume not
much more in other supported languages.

Skip

On Jan 7, 2018 7:30 AM, "chintan mavawala" <ch...@gmail.com> wrote:

> Exactly. This is what I want to achieve.
>
> Is it good practice to keep on updating subscription list of same kafka
> consumer?
>
> Any concerns in multi threaded environment?
>
> Regards,
> Chintan
>
> On 07-Jan-2018 6:32 PM, "Skip Montanaro" <sk...@gmail.com> wrote:
>
> Got it. Does it work to maintain your currently subscribed topic list, then
> spend to it and subscribe to the extended list when you receive an update?
>
> On Jan 6, 2018 9:37 PM, "chintan mavawala" <ch...@gmail.com> wrote:
>
> > Discovery of new topic is not a concern. Clients who creates a new topic
> > sends my process the topic name. In simpler words, my process performs
> some
> > operations which any client might be wanting. So they create a Kafka
> topic
> > and send me the name of the topic for me to subscribe.
> >
> > My process needs start consuming from this topic too in addition to
> already
> > subscribed topics.
> >
> > I hope it makes sense.
> >
> > Regards,
> > Chintan
> >
> > On 06-Jan-2018 11:30 PM, "Skip Montanaro" <sk...@gmail.com>
> > wrote:
> >
> > I'm a bit confused. How do you discover that new topics have been
> created?
> > If you discover a new topic somehow, how do you know it's one you are
> > interested in?
> >
> > I suppose you could have a consumer which subscribes to all topics using
> > pattern=".", then have it just publish new topic names to a fixed "new
> > topics" topic. You could subscribe to that. Still, with no pattern to the
> > name you'd need some external information (database query?) to decide if
> > you wanted to subscribe to it.
> >
> > Skip
> >
> > On Jan 6, 2018 10:43 AM, "chintan mavawala" <ch...@gmail.com>
> > wrote:
> >
> > > In theory pattern "." would work. But that would mean subscribing on
> all
> > > topics which I don't want.
> > >
> > > I share same pain in using pattern which is why I would like to know if
> > it
> > > is good practice to subscribe to topics after kafka consumer has
> started.
> > >
> > > Regards,
> > > Chintan
> > >
> > > On 06-Jan-2018 5:47 PM, "Skip Montanaro" <sk...@gmail.com>
> > wrote:
> > >
> > > In theory, wouldn't
> > >
> > > consumer.subscribe(pattern=".")
> > >
> > > work? I say "in theory" because my experience with subscribing by
> pattern
> > > hasn't been great. I suspect my mental model of how it's implemented
> > isn't
> > > a close approximation of reality.
> > >
> > > Skip
> > >
> > > On Jan 6, 2018 4:07 AM, "chintan mavawala" <ch...@gmail.com>
> > wrote:
> > >
> > > I want to subscribe to all topics as they are created but topic names
> > does
> > > not have any pattern. Consumer starts with subscription to let's say 3
> > > topics and add more topics to exisiting kafka consumer as they are
> > created.
> > >
> > > Regards,
> > > Chintan
> > >
> > > On 04-Jan-2018 11:02 PM, "Jordan Pilat" <jr...@gmail.com> wrote:
> > >
> > > > Did you want to subscribe to all topics as they are created, or to
> > > exclude
> > > > some of them?
> > > > If the latter, is there a pattern in the names of the topics you wish
> > to
> > > > exclude?
> > > >
> > > > - Jordan Pilat
> > > >
> > > > On 2018-01-03 23:39, chintan mavawala <ch...@gmail.com>
> wrote:
> > > > > Hi,
> > > > >
> > > > > We have a use case where we need to start kafka consumer with a
> fixed
> > > > list
> > > > > of kafka topics and add more topics on the fly. Since there is no
> > > pattern
> > > > > in names of topics, using pattern for dynamic subscriptions of
> topics
> > > is
> > > > > not feasible.
> > > > >
> > > > > Is it a good practice to subscribe to topics on given kafka
> consumer
> > on
> > > > fly
> > > > > using after starting consumer?
> > > > >
> > > > > I understand that KafkaConsumer is not thread safe. Any ill effects
> > due
> > > > to
> > > > > multi threaded env?
> > > > >
> > > > > Regards,
> > > > > Chintan
> > > > >
> > > >
> > >
> >
>

Re: Dynamic topic subscription without using pattern

[chintan mavawala <ch...@gmail.com>]

Exactly. This is what I want to achieve.

Is it good practice to keep on updating subscription list of same kafka
consumer?

Any concerns in multi threaded environment?

Regards,
Chintan

On 07-Jan-2018 6:32 PM, "Skip Montanaro" <sk...@gmail.com> wrote:

Got it. Does it work to maintain your currently subscribed topic list, then
spend to it and subscribe to the extended list when you receive an update?

On Jan 6, 2018 9:37 PM, "chintan mavawala" <ch...@gmail.com> wrote:

> Discovery of new topic is not a concern. Clients who creates a new topic
> sends my process the topic name. In simpler words, my process performs
some
> operations which any client might be wanting. So they create a Kafka topic
> and send me the name of the topic for me to subscribe.
>
> My process needs start consuming from this topic too in addition to
already
> subscribed topics.
>
> I hope it makes sense.
>
> Regards,
> Chintan
>
> On 06-Jan-2018 11:30 PM, "Skip Montanaro" <sk...@gmail.com>
> wrote:
>
> I'm a bit confused. How do you discover that new topics have been created?
> If you discover a new topic somehow, how do you know it's one you are
> interested in?
>
> I suppose you could have a consumer which subscribes to all topics using
> pattern=".", then have it just publish new topic names to a fixed "new
> topics" topic. You could subscribe to that. Still, with no pattern to the
> name you'd need some external information (database query?) to decide if
> you wanted to subscribe to it.
>
> Skip
>
> On Jan 6, 2018 10:43 AM, "chintan mavawala" <ch...@gmail.com>
> wrote:
>
> > In theory pattern "." would work. But that would mean subscribing on all
> > topics which I don't want.
> >
> > I share same pain in using pattern which is why I would like to know if
> it
> > is good practice to subscribe to topics after kafka consumer has
started.
> >
> > Regards,
> > Chintan
> >
> > On 06-Jan-2018 5:47 PM, "Skip Montanaro" <sk...@gmail.com>
> wrote:
> >
> > In theory, wouldn't
> >
> > consumer.subscribe(pattern=".")
> >
> > work? I say "in theory" because my experience with subscribing by
pattern
> > hasn't been great. I suspect my mental model of how it's implemented
> isn't
> > a close approximation of reality.
> >
> > Skip
> >
> > On Jan 6, 2018 4:07 AM, "chintan mavawala" <ch...@gmail.com>
> wrote:
> >
> > I want to subscribe to all topics as they are created but topic names
> does
> > not have any pattern. Consumer starts with subscription to let's say 3
> > topics and add more topics to exisiting kafka consumer as they are
> created.
> >
> > Regards,
> > Chintan
> >
> > On 04-Jan-2018 11:02 PM, "Jordan Pilat" <jr...@gmail.com> wrote:
> >
> > > Did you want to subscribe to all topics as they are created, or to
> > exclude
> > > some of them?
> > > If the latter, is there a pattern in the names of the topics you wish
> to
> > > exclude?
> > >
> > > - Jordan Pilat
> > >
> > > On 2018-01-03 23:39, chintan mavawala <ch...@gmail.com> wrote:
> > > > Hi,
> > > >
> > > > We have a use case where we need to start kafka consumer with a
fixed
> > > list
> > > > of kafka topics and add more topics on the fly. Since there is no
> > pattern
> > > > in names of topics, using pattern for dynamic subscriptions of
topics
> > is
> > > > not feasible.
> > > >
> > > > Is it a good practice to subscribe to topics on given kafka consumer
> on
> > > fly
> > > > using after starting consumer?
> > > >
> > > > I understand that KafkaConsumer is not thread safe. Any ill effects
> due
> > > to
> > > > multi threaded env?
> > > >
> > > > Regards,
> > > > Chintan
> > > >
> > >
> >
>

Re: Dynamic topic subscription without using pattern

[Skip Montanaro <sk...@gmail.com>]

Got it. Does it work to maintain your currently subscribed topic list, then
spend to it and subscribe to the extended list when you receive an update?

On Jan 6, 2018 9:37 PM, "chintan mavawala" <ch...@gmail.com> wrote:

> Discovery of new topic is not a concern. Clients who creates a new topic
> sends my process the topic name. In simpler words, my process performs some
> operations which any client might be wanting. So they create a Kafka topic
> and send me the name of the topic for me to subscribe.
>
> My process needs start consuming from this topic too in addition to already
> subscribed topics.
>
> I hope it makes sense.
>
> Regards,
> Chintan
>
> On 06-Jan-2018 11:30 PM, "Skip Montanaro" <sk...@gmail.com>
> wrote:
>
> I'm a bit confused. How do you discover that new topics have been created?
> If you discover a new topic somehow, how do you know it's one you are
> interested in?
>
> I suppose you could have a consumer which subscribes to all topics using
> pattern=".", then have it just publish new topic names to a fixed "new
> topics" topic. You could subscribe to that. Still, with no pattern to the
> name you'd need some external information (database query?) to decide if
> you wanted to subscribe to it.
>
> Skip
>
> On Jan 6, 2018 10:43 AM, "chintan mavawala" <ch...@gmail.com>
> wrote:
>
> > In theory pattern "." would work. But that would mean subscribing on all
> > topics which I don't want.
> >
> > I share same pain in using pattern which is why I would like to know if
> it
> > is good practice to subscribe to topics after kafka consumer has started.
> >
> > Regards,
> > Chintan
> >
> > On 06-Jan-2018 5:47 PM, "Skip Montanaro" <sk...@gmail.com>
> wrote:
> >
> > In theory, wouldn't
> >
> > consumer.subscribe(pattern=".")
> >
> > work? I say "in theory" because my experience with subscribing by pattern
> > hasn't been great. I suspect my mental model of how it's implemented
> isn't
> > a close approximation of reality.
> >
> > Skip
> >
> > On Jan 6, 2018 4:07 AM, "chintan mavawala" <ch...@gmail.com>
> wrote:
> >
> > I want to subscribe to all topics as they are created but topic names
> does
> > not have any pattern. Consumer starts with subscription to let's say 3
> > topics and add more topics to exisiting kafka consumer as they are
> created.
> >
> > Regards,
> > Chintan
> >
> > On 04-Jan-2018 11:02 PM, "Jordan Pilat" <jr...@gmail.com> wrote:
> >
> > > Did you want to subscribe to all topics as they are created, or to
> > exclude
> > > some of them?
> > > If the latter, is there a pattern in the names of the topics you wish
> to
> > > exclude?
> > >
> > > - Jordan Pilat
> > >
> > > On 2018-01-03 23:39, chintan mavawala <ch...@gmail.com> wrote:
> > > > Hi,
> > > >
> > > > We have a use case where we need to start kafka consumer with a fixed
> > > list
> > > > of kafka topics and add more topics on the fly. Since there is no
> > pattern
> > > > in names of topics, using pattern for dynamic subscriptions of topics
> > is
> > > > not feasible.
> > > >
> > > > Is it a good practice to subscribe to topics on given kafka consumer
> on
> > > fly
> > > > using after starting consumer?
> > > >
> > > > I understand that KafkaConsumer is not thread safe. Any ill effects
> due
> > > to
> > > > multi threaded env?
> > > >
> > > > Regards,
> > > > Chintan
> > > >
> > >
> >
>

Re: Dynamic topic subscription without using pattern

[chintan mavawala <ch...@gmail.com>]

Discovery of new topic is not a concern. Clients who creates a new topic
sends my process the topic name. In simpler words, my process performs some
operations which any client might be wanting. So they create a Kafka topic
and send me the name of the topic for me to subscribe.

My process needs start consuming from this topic too in addition to already
subscribed topics.

I hope it makes sense.

Regards,
Chintan

On 06-Jan-2018 11:30 PM, "Skip Montanaro" <sk...@gmail.com> wrote:

I'm a bit confused. How do you discover that new topics have been created?
If you discover a new topic somehow, how do you know it's one you are
interested in?

I suppose you could have a consumer which subscribes to all topics using
pattern=".", then have it just publish new topic names to a fixed "new
topics" topic. You could subscribe to that. Still, with no pattern to the
name you'd need some external information (database query?) to decide if
you wanted to subscribe to it.

Skip

On Jan 6, 2018 10:43 AM, "chintan mavawala" <ch...@gmail.com> wrote:

> In theory pattern "." would work. But that would mean subscribing on all
> topics which I don't want.
>
> I share same pain in using pattern which is why I would like to know if it
> is good practice to subscribe to topics after kafka consumer has started.
>
> Regards,
> Chintan
>
> On 06-Jan-2018 5:47 PM, "Skip Montanaro" <sk...@gmail.com> wrote:
>
> In theory, wouldn't
>
> consumer.subscribe(pattern=".")
>
> work? I say "in theory" because my experience with subscribing by pattern
> hasn't been great. I suspect my mental model of how it's implemented isn't
> a close approximation of reality.
>
> Skip
>
> On Jan 6, 2018 4:07 AM, "chintan mavawala" <ch...@gmail.com> wrote:
>
> I want to subscribe to all topics as they are created but topic names does
> not have any pattern. Consumer starts with subscription to let's say 3
> topics and add more topics to exisiting kafka consumer as they are
created.
>
> Regards,
> Chintan
>
> On 04-Jan-2018 11:02 PM, "Jordan Pilat" <jr...@gmail.com> wrote:
>
> > Did you want to subscribe to all topics as they are created, or to
> exclude
> > some of them?
> > If the latter, is there a pattern in the names of the topics you wish to
> > exclude?
> >
> > - Jordan Pilat
> >
> > On 2018-01-03 23:39, chintan mavawala <ch...@gmail.com> wrote:
> > > Hi,
> > >
> > > We have a use case where we need to start kafka consumer with a fixed
> > list
> > > of kafka topics and add more topics on the fly. Since there is no
> pattern
> > > in names of topics, using pattern for dynamic subscriptions of topics
> is
> > > not feasible.
> > >
> > > Is it a good practice to subscribe to topics on given kafka consumer
on
> > fly
> > > using after starting consumer?
> > >
> > > I understand that KafkaConsumer is not thread safe. Any ill effects
due
> > to
> > > multi threaded env?
> > >
> > > Regards,
> > > Chintan
> > >
> >
>

Re: Dynamic topic subscription without using pattern

[Skip Montanaro <sk...@gmail.com>]

I'm a bit confused. How do you discover that new topics have been created?
If you discover a new topic somehow, how do you know it's one you are
interested in?

I suppose you could have a consumer which subscribes to all topics using
pattern=".", then have it just publish new topic names to a fixed "new
topics" topic. You could subscribe to that. Still, with no pattern to the
name you'd need some external information (database query?) to decide if
you wanted to subscribe to it.

Skip

On Jan 6, 2018 10:43 AM, "chintan mavawala" <ch...@gmail.com> wrote:

> In theory pattern "." would work. But that would mean subscribing on all
> topics which I don't want.
>
> I share same pain in using pattern which is why I would like to know if it
> is good practice to subscribe to topics after kafka consumer has started.
>
> Regards,
> Chintan
>
> On 06-Jan-2018 5:47 PM, "Skip Montanaro" <sk...@gmail.com> wrote:
>
> In theory, wouldn't
>
> consumer.subscribe(pattern=".")
>
> work? I say "in theory" because my experience with subscribing by pattern
> hasn't been great. I suspect my mental model of how it's implemented isn't
> a close approximation of reality.
>
> Skip
>
> On Jan 6, 2018 4:07 AM, "chintan mavawala" <ch...@gmail.com> wrote:
>
> I want to subscribe to all topics as they are created but topic names does
> not have any pattern. Consumer starts with subscription to let's say 3
> topics and add more topics to exisiting kafka consumer as they are created.
>
> Regards,
> Chintan
>
> On 04-Jan-2018 11:02 PM, "Jordan Pilat" <jr...@gmail.com> wrote:
>
> > Did you want to subscribe to all topics as they are created, or to
> exclude
> > some of them?
> > If the latter, is there a pattern in the names of the topics you wish to
> > exclude?
> >
> > - Jordan Pilat
> >
> > On 2018-01-03 23:39, chintan mavawala <ch...@gmail.com> wrote:
> > > Hi,
> > >
> > > We have a use case where we need to start kafka consumer with a fixed
> > list
> > > of kafka topics and add more topics on the fly. Since there is no
> pattern
> > > in names of topics, using pattern for dynamic subscriptions of topics
> is
> > > not feasible.
> > >
> > > Is it a good practice to subscribe to topics on given kafka consumer on
> > fly
> > > using after starting consumer?
> > >
> > > I understand that KafkaConsumer is not thread safe. Any ill effects due
> > to
> > > multi threaded env?
> > >
> > > Regards,
> > > Chintan
> > >
> >
>

Re: Dynamic topic subscription without using pattern

[chintan mavawala <ch...@gmail.com>]

In theory pattern "." would work. But that would mean subscribing on all
topics which I don't want.

I share same pain in using pattern which is why I would like to know if it
is good practice to subscribe to topics after kafka consumer has started.

Regards,
Chintan

On 06-Jan-2018 5:47 PM, "Skip Montanaro" <sk...@gmail.com> wrote:

In theory, wouldn't

consumer.subscribe(pattern=".")

work? I say "in theory" because my experience with subscribing by pattern
hasn't been great. I suspect my mental model of how it's implemented isn't
a close approximation of reality.

Skip

On Jan 6, 2018 4:07 AM, "chintan mavawala" <ch...@gmail.com> wrote:

I want to subscribe to all topics as they are created but topic names does
not have any pattern. Consumer starts with subscription to let's say 3
topics and add more topics to exisiting kafka consumer as they are created.

Regards,
Chintan

On 04-Jan-2018 11:02 PM, "Jordan Pilat" <jr...@gmail.com> wrote:

> Did you want to subscribe to all topics as they are created, or to exclude
> some of them?
> If the latter, is there a pattern in the names of the topics you wish to
> exclude?
>
> - Jordan Pilat
>
> On 2018-01-03 23:39, chintan mavawala <ch...@gmail.com> wrote:
> > Hi,
> >
> > We have a use case where we need to start kafka consumer with a fixed
> list
> > of kafka topics and add more topics on the fly. Since there is no
pattern
> > in names of topics, using pattern for dynamic subscriptions of topics is
> > not feasible.
> >
> > Is it a good practice to subscribe to topics on given kafka consumer on
> fly
> > using after starting consumer?
> >
> > I understand that KafkaConsumer is not thread safe. Any ill effects due
> to
> > multi threaded env?
> >
> > Regards,
> > Chintan
> >
>

Re: Dynamic topic subscription without using pattern

[Skip Montanaro <sk...@gmail.com>]

In theory, wouldn't

consumer.subscribe(pattern=".")

work? I say "in theory" because my experience with subscribing by pattern
hasn't been great. I suspect my mental model of how it's implemented isn't
a close approximation of reality.

Skip

On Jan 6, 2018 4:07 AM, "chintan mavawala" <ch...@gmail.com> wrote:

I want to subscribe to all topics as they are created but topic names does
not have any pattern. Consumer starts with subscription to let's say 3
topics and add more topics to exisiting kafka consumer as they are created.

Regards,
Chintan

On 04-Jan-2018 11:02 PM, "Jordan Pilat" <jr...@gmail.com> wrote:

> Did you want to subscribe to all topics as they are created, or to exclude
> some of them?
> If the latter, is there a pattern in the names of the topics you wish to
> exclude?
>
> - Jordan Pilat
>
> On 2018-01-03 23:39, chintan mavawala <ch...@gmail.com> wrote:
> > Hi,
> >
> > We have a use case where we need to start kafka consumer with a fixed
> list
> > of kafka topics and add more topics on the fly. Since there is no
pattern
> > in names of topics, using pattern for dynamic subscriptions of topics is
> > not feasible.
> >
> > Is it a good practice to subscribe to topics on given kafka consumer on
> fly
> > using after starting consumer?
> >
> > I understand that KafkaConsumer is not thread safe. Any ill effects due
> to
> > multi threaded env?
> >
> > Regards,
> > Chintan
> >
>

Re: Dynamic topic subscription without using pattern

[chintan mavawala <ch...@gmail.com>]

I want to subscribe to all topics as they are created but topic names does
not have any pattern. Consumer starts with subscription to let's say 3
topics and add more topics to exisiting kafka consumer as they are created.

Regards,
Chintan

On 04-Jan-2018 11:02 PM, "Jordan Pilat" <jr...@gmail.com> wrote:

> Did you want to subscribe to all topics as they are created, or to exclude
> some of them?
> If the latter, is there a pattern in the names of the topics you wish to
> exclude?
>
> - Jordan Pilat
>
> On 2018-01-03 23:39, chintan mavawala <ch...@gmail.com> wrote:
> > Hi,
> >
> > We have a use case where we need to start kafka consumer with a fixed
> list
> > of kafka topics and add more topics on the fly. Since there is no pattern
> > in names of topics, using pattern for dynamic subscriptions of topics is
> > not feasible.
> >
> > Is it a good practice to subscribe to topics on given kafka consumer on
> fly
> > using after starting consumer?
> >
> > I understand that KafkaConsumer is not thread safe. Any ill effects due
> to
> > multi threaded env?
> >
> > Regards,
> > Chintan
> >
>

Re: Dynamic topic subscription without using pattern

[Jordan Pilat <jr...@gmail.com>]

Did you want to subscribe to all topics as they are created, or to exclude some of them?
If the latter, is there a pattern in the names of the topics you wish to exclude?

- Jordan Pilat

On 2018-01-03 23:39, chintan mavawala <ch...@gmail.com> wrote: 
> Hi,
> 
> We have a use case where we need to start kafka consumer with a fixed list
> of kafka topics and add more topics on the fly. Since there is no pattern
> in names of topics, using pattern for dynamic subscriptions of topics is
> not feasible.
> 
> Is it a good practice to subscribe to topics on given kafka consumer on fly
> using after starting consumer?
> 
> I understand that KafkaConsumer is not thread safe. Any ill effects due to
> multi threaded env?
> 
> Regards,
> Chintan
>