You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by AJ Chen <ca...@gmail.com> on 2020/06/12 16:01:04 UTC
dynamic produce and consume new topics
I'm working on a data streaming system where users will publish to
different new topics and subscribe to different new topics without
restarting the system. A large number of topics will be created over time.
Uer will choose different new topics to publish and subscribe dynamically
at runtime. Can someone share good design or practice to implement such
dynamic topic handling? Links to sample code will be very helpful.
Thanks,
aj
Re: dynamic produce and consume new topics
Posted by Boyang Chen <re...@gmail.com>.
Good question, TBH I'm not an expert on Spring, maybe cross post this
question in the Spring community?
On Sat, Jun 13, 2020 at 7:43 AM AJ Chen <aj...@web2express.org> wrote:
> Another strategy to handle dynamic topics:
> SInce Spring Kafka MessageListenerContainer holds a list of topics, can we
> simply add a new topic into the existing container at runtime?
> Does the existing listener container object require a manual stop/start
> after its topic list changes?
>
> thanks,
> -aj
>
>
>
> On Fri, Jun 12, 2020 at 3:25 PM AJ Chen <aj...@web2express.org> wrote:
>
> > I figured out this approach to consume new topics that become available
> at
> > runtime:
> > 1. create ConcurrentMessageListenerContainer for each new topic, call
> > start() to start listening.
> > 2. track all containers in a map, which listen and consume messages from
> > the assigned topics.
> > 3. call container.stop() when shutdown.
> >
> > Is there anything important I should add to make it more robust?
> > Appreciate any suggestions.
> >
> > -aj
> >
> > On Fri, Jun 12, 2020 at 12:55 PM AJ Chen <aj...@web2express.org> wrote:
> >
> >> Thanks, Boyang.
> >>
> >> Spring''s KafkaTemplate can easily create new topic on the flight
> >> already.
> >>
> >> On consumer side, is there java sample code to replace this annotation?
> >> That will create a new listener for a new topic and group-id on the
> >> fliight.
> >> @KafkaListener(topics = "test", groupId = "grp0")
> >>
> >> -aj
> >>
> >> On Fri, Jun 12, 2020 at 9:30 AM Boyang Chen <reluctanthero104@gmail.com
> >
> >> wrote:
> >>
> >>> Hey AJ,
> >>>
> >>> you should be able to make the subscribed topics and output topics
> >>> dynamic
> >>> configs for Consumer and Producer. If you need to create new topics on
> >>> runtime, there is some broker side setting you could use to allow topic
> >>> creation based of produced records:
> >>>
> >>>
> https://stackoverflow.com/questions/36441768/how-to-create-topics-in-apache-kafka
> >>>
> >>> On Fri, Jun 12, 2020 at 9:01 AM AJ Chen <ca...@gmail.com> wrote:
> >>>
> >>> > I'm working on a data streaming system where users will publish to
> >>> > different new topics and subscribe to different new topics without
> >>> > restarting the system. A large number of topics will be created over
> >>> time.
> >>> > Uer will choose different new topics to publish and subscribe
> >>> dynamically
> >>> > at runtime. Can someone share good design or practice to implement
> such
> >>> > dynamic topic handling? Links to sample code will be very helpful.
> >>> >
> >>> > Thanks,
> >>> > aj
> >>> >
> >>>
> >>
>
Re: dynamic produce and consume new topics
Posted by AJ Chen <aj...@web2express.org>.
Another strategy to handle dynamic topics:
SInce Spring Kafka MessageListenerContainer holds a list of topics, can we
simply add a new topic into the existing container at runtime?
Does the existing listener container object require a manual stop/start
after its topic list changes?
thanks,
-aj
On Fri, Jun 12, 2020 at 3:25 PM AJ Chen <aj...@web2express.org> wrote:
> I figured out this approach to consume new topics that become available at
> runtime:
> 1. create ConcurrentMessageListenerContainer for each new topic, call
> start() to start listening.
> 2. track all containers in a map, which listen and consume messages from
> the assigned topics.
> 3. call container.stop() when shutdown.
>
> Is there anything important I should add to make it more robust?
> Appreciate any suggestions.
>
> -aj
>
> On Fri, Jun 12, 2020 at 12:55 PM AJ Chen <aj...@web2express.org> wrote:
>
>> Thanks, Boyang.
>>
>> Spring''s KafkaTemplate can easily create new topic on the flight
>> already.
>>
>> On consumer side, is there java sample code to replace this annotation?
>> That will create a new listener for a new topic and group-id on the
>> fliight.
>> @KafkaListener(topics = "test", groupId = "grp0")
>>
>> -aj
>>
>> On Fri, Jun 12, 2020 at 9:30 AM Boyang Chen <re...@gmail.com>
>> wrote:
>>
>>> Hey AJ,
>>>
>>> you should be able to make the subscribed topics and output topics
>>> dynamic
>>> configs for Consumer and Producer. If you need to create new topics on
>>> runtime, there is some broker side setting you could use to allow topic
>>> creation based of produced records:
>>>
>>> https://stackoverflow.com/questions/36441768/how-to-create-topics-in-apache-kafka
>>>
>>> On Fri, Jun 12, 2020 at 9:01 AM AJ Chen <ca...@gmail.com> wrote:
>>>
>>> > I'm working on a data streaming system where users will publish to
>>> > different new topics and subscribe to different new topics without
>>> > restarting the system. A large number of topics will be created over
>>> time.
>>> > Uer will choose different new topics to publish and subscribe
>>> dynamically
>>> > at runtime. Can someone share good design or practice to implement such
>>> > dynamic topic handling? Links to sample code will be very helpful.
>>> >
>>> > Thanks,
>>> > aj
>>> >
>>>
>>
Re: dynamic produce and consume new topics
Posted by AJ Chen <aj...@web2express.org>.
I figured out this approach to consume new topics that become available at
runtime:
1. create ConcurrentMessageListenerContainer for each new topic, call
start() to start listening.
2. track all containers in a map, which listen and consume messages from
the assigned topics.
3. call container.stop() when shutdown.
Is there anything important I should add to make it more robust? Appreciate
any suggestions.
-aj
On Fri, Jun 12, 2020 at 12:55 PM AJ Chen <aj...@web2express.org> wrote:
> Thanks, Boyang.
>
> Spring''s KafkaTemplate can easily create new topic on the flight
> already.
>
> On consumer side, is there java sample code to replace this annotation?
> That will create a new listener for a new topic and group-id on the
> fliight.
> @KafkaListener(topics = "test", groupId = "grp0")
>
> -aj
>
> On Fri, Jun 12, 2020 at 9:30 AM Boyang Chen <re...@gmail.com>
> wrote:
>
>> Hey AJ,
>>
>> you should be able to make the subscribed topics and output topics dynamic
>> configs for Consumer and Producer. If you need to create new topics on
>> runtime, there is some broker side setting you could use to allow topic
>> creation based of produced records:
>>
>> https://stackoverflow.com/questions/36441768/how-to-create-topics-in-apache-kafka
>>
>> On Fri, Jun 12, 2020 at 9:01 AM AJ Chen <ca...@gmail.com> wrote:
>>
>> > I'm working on a data streaming system where users will publish to
>> > different new topics and subscribe to different new topics without
>> > restarting the system. A large number of topics will be created over
>> time.
>> > Uer will choose different new topics to publish and subscribe
>> dynamically
>> > at runtime. Can someone share good design or practice to implement such
>> > dynamic topic handling? Links to sample code will be very helpful.
>> >
>> > Thanks,
>> > aj
>> >
>>
>
Re: dynamic produce and consume new topics
Posted by AJ Chen <aj...@web2express.org>.
Thanks, Boyang.
Spring''s KafkaTemplate can easily create new topic on the flight already.
On consumer side, is there java sample code to replace this annotation?
That will create a new listener for a new topic and group-id on the
fliight.
@KafkaListener(topics = "test", groupId = "grp0")
-aj
On Fri, Jun 12, 2020 at 9:30 AM Boyang Chen <re...@gmail.com>
wrote:
> Hey AJ,
>
> you should be able to make the subscribed topics and output topics dynamic
> configs for Consumer and Producer. If you need to create new topics on
> runtime, there is some broker side setting you could use to allow topic
> creation based of produced records:
>
> https://stackoverflow.com/questions/36441768/how-to-create-topics-in-apache-kafka
>
> On Fri, Jun 12, 2020 at 9:01 AM AJ Chen <ca...@gmail.com> wrote:
>
> > I'm working on a data streaming system where users will publish to
> > different new topics and subscribe to different new topics without
> > restarting the system. A large number of topics will be created over
> time.
> > Uer will choose different new topics to publish and subscribe dynamically
> > at runtime. Can someone share good design or practice to implement such
> > dynamic topic handling? Links to sample code will be very helpful.
> >
> > Thanks,
> > aj
> >
>
Re: dynamic produce and consume new topics
Posted by Boyang Chen <re...@gmail.com>.
Hey AJ,
you should be able to make the subscribed topics and output topics dynamic
configs for Consumer and Producer. If you need to create new topics on
runtime, there is some broker side setting you could use to allow topic
creation based of produced records:
https://stackoverflow.com/questions/36441768/how-to-create-topics-in-apache-kafka
On Fri, Jun 12, 2020 at 9:01 AM AJ Chen <ca...@gmail.com> wrote:
> I'm working on a data streaming system where users will publish to
> different new topics and subscribe to different new topics without
> restarting the system. A large number of topics will be created over time.
> Uer will choose different new topics to publish and subscribe dynamically
> at runtime. Can someone share good design or practice to implement such
> dynamic topic handling? Links to sample code will be very helpful.
>
> Thanks,
> aj
>