You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by marimuthu eee <ma...@gmail.com> on 2019/01/08 09:56:52 UTC
Dynamic Partitioning
Hi,
My requirement is to process notification messages in parallel.So i have an
idea to pick kafka as a messaging system.But i need dynamic partitioning to
handle individual user notification messages.Is it possible to create
dynamic partition creation for a particular topic.
Re: Dynamic Partitioning
Posted by Dimitry Lvovsky <dl...@gmail.com>.
One approach you can take is to set an upper bound number of partitions a
priori. Imagine your key was the username. If you had 2 partitions in your
topic and 4 users writing messages, then Kafka would split these messages
between two partitions of the topic (assuming the usernames are unique).
For your design I would ask the questions:
- how many simultaneous users will I have
- how long does it take to process a message
- how many messages per second will I have
Answering the above questions will allow you to define the expectations and
limits of your applications.
So you if know, for example, that users will send 2 messages per second,
and say it takes 250ms to process a message, a user could wait potentially
wait for 1 ( assuming the messages are equally distributed -- which they
might not be) second to receive a message. That might be fine or you might
want that to go faster. If it's the latter, then build it from the
beginning with more partitions in mind. There is nothing stopping you from
say declaring a topic with 64 partitions from day one (or more, just
don't go crazy as to-many are not good either), but use only 2 consumers
from the start. In this case, each consumer is reading from 32
partitions. Now let's say more users sign on, and you have some kind of
KPI you want to meet, e.g. on average users shouldn't wait more than 500ms
to receive a message. In this case, you can simply spin up more
consumers. In the example above, say 8 users are now online, so you spin
up 2 more consumers -- now each consumer will have 16 partitions each. You
can scale up or down by the number of consumers you have.
There can come a point when the demand on your system will be greater than
the original limits you set when doing your initial design. This is the
point I would consider adding more partitions.
Hope this helps.
Dimitry
On Tue, Jan 8, 2019 at 6:29 PM Ryanne Dolan <ry...@gmail.com> wrote:
> Just key your messages by user ID and they'll be processed in sequence.
> Every message with the same key will go to the same partition, regardless
> of how many partitions there are.
>
> Ryanne
>
> On Tue, Jan 8, 2019, 8:57 AM marimuthu eee <marimuthueee76@gmail.com
> wrote:
>
> > Hi Ryanne Dolan,
> >
> > My requirement is to process notification messages in parallel.So i have
> > an idea to pick kafka as a messaging system.But i need dynamic
> partitioning*
> > to handle individual user notification messages in serial manner*.Is it
> > possible to create dynamic partition creation for a particular topic.
> >
> >
> >
> > On Tue, Jan 8, 2019 at 8:20 PM Ryanne Dolan <ry...@gmail.com>
> wrote:
> >
> >> Can you explain why you think you need dynamic partitioning? Maybe we
> can
> >> help you design without it.
> >>
> >> Ryanne
> >>
> >> On Tue, Jan 8, 2019, 4:04 AM marimuthu eee <marimuthueee76@gmail.com
> >> wrote:
> >>
> >> > Hi,
> >> >
> >> > My requirement is to process notification messages in parallel.So i
> >> have an
> >> > idea to pick kafka as a messaging system.But i need dynamic
> >> partitioning to
> >> > handle individual user notification messages.Is it possible to create
> >> > dynamic partition creation for a particular topic.
> >> >
> >>
> >
>
Re: Dynamic Partitioning
Posted by Ryanne Dolan <ry...@gmail.com>.
Just key your messages by user ID and they'll be processed in sequence.
Every message with the same key will go to the same partition, regardless
of how many partitions there are.
Ryanne
On Tue, Jan 8, 2019, 8:57 AM marimuthu eee <marimuthueee76@gmail.com wrote:
> Hi Ryanne Dolan,
>
> My requirement is to process notification messages in parallel.So i have
> an idea to pick kafka as a messaging system.But i need dynamic partitioning*
> to handle individual user notification messages in serial manner*.Is it
> possible to create dynamic partition creation for a particular topic.
>
>
>
> On Tue, Jan 8, 2019 at 8:20 PM Ryanne Dolan <ry...@gmail.com> wrote:
>
>> Can you explain why you think you need dynamic partitioning? Maybe we can
>> help you design without it.
>>
>> Ryanne
>>
>> On Tue, Jan 8, 2019, 4:04 AM marimuthu eee <marimuthueee76@gmail.com
>> wrote:
>>
>> > Hi,
>> >
>> > My requirement is to process notification messages in parallel.So i
>> have an
>> > idea to pick kafka as a messaging system.But i need dynamic
>> partitioning to
>> > handle individual user notification messages.Is it possible to create
>> > dynamic partition creation for a particular topic.
>> >
>>
>
Re: Dynamic Partitioning
Posted by marimuthu eee <ma...@gmail.com>.
Hi Ryanne Dolan,
My requirement is to process notification messages in parallel.So i have an
idea to pick kafka as a messaging system.But i need dynamic partitioning*
to handle individual user notification messages in serial manner*.Is it
possible to create dynamic partition creation for a particular topic.
On Tue, Jan 8, 2019 at 8:20 PM Ryanne Dolan <ry...@gmail.com> wrote:
> Can you explain why you think you need dynamic partitioning? Maybe we can
> help you design without it.
>
> Ryanne
>
> On Tue, Jan 8, 2019, 4:04 AM marimuthu eee <marimuthueee76@gmail.com
> wrote:
>
> > Hi,
> >
> > My requirement is to process notification messages in parallel.So i have
> an
> > idea to pick kafka as a messaging system.But i need dynamic partitioning
> to
> > handle individual user notification messages.Is it possible to create
> > dynamic partition creation for a particular topic.
> >
>
Re: Dynamic Partitioning
Posted by Ryanne Dolan <ry...@gmail.com>.
Can you explain why you think you need dynamic partitioning? Maybe we can
help you design without it.
Ryanne
On Tue, Jan 8, 2019, 4:04 AM marimuthu eee <marimuthueee76@gmail.com wrote:
> Hi,
>
> My requirement is to process notification messages in parallel.So i have an
> idea to pick kafka as a messaging system.But i need dynamic partitioning to
> handle individual user notification messages.Is it possible to create
> dynamic partition creation for a particular topic.
>
Re: Dynamic Partitioning
Posted by Jonathan Santilli <jo...@gmail.com>.
Hello Marimuthu,
Partitions are created at the moment the topic is created, however, you can
also add more new partitions to a topic once have been created. There is
not a possibility to create partitions dynamically (out-of-the-box) into a
topic unless you create your own business logic and/or scripts to do that.
Kafka offers some scripts to alter the topics, take a look to
https://kafka.apache.org/documentation/#basic_ops_modify_topic. Maybe you
can consider using internal Kafka API Admin to do that as well (
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/admin/AdminClient.java
)
Beware that once the new partitions are created, maybe you will need to
distribute them accordingly, take a look to
https://kafka.apache.org/documentation/#basic_ops_partitionassignment
Hope this helps,
--
Jonathan
On Tue, Jan 8, 2019 at 10:04 AM marimuthu eee <ma...@gmail.com>
wrote:
> Hi,
>
> My requirement is to process notification messages in parallel.So i have an
> idea to pick kafka as a messaging system.But i need dynamic partitioning to
> handle individual user notification messages.Is it possible to create
> dynamic partition creation for a particular topic.
>
--
Santilli Jonathan