You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Mike Gould <mi...@gmail.com> on 2017/05/23 07:47:00 UTC
Partition assignment with multiple topics
Hi
We have a couple of hundred topics - each carrying a similar but distinct
message type but to keep the total partition count down each only has 3
partitions.
If I start Kafka-streams consuming all topics only 3 threads ever get
assigned any partitions.
I think the first thread to start gets the first partition of each topic,
and so on until the 3rd thread, after that all the partitions are assigned
- any further threads are just left idle.
Is there any way to make the partition assignment smarter and either add a
random element that moves partitions when further consumers start or
considers all the partitions of subscribed topics together when assigning
them?
Our only alternative is creating many more partitions for each topic - and
we're worried about how far this will scale.
Thank you
Mike G
--
- MikeG
http://en.wikipedia.org/wiki/Common_misconceptions
<http://en.wikipedia.org/wiki/Special:Random>
Re: Partition assignment with multiple topics
Posted by Mike Gould <mi...@gmail.com>.
Hi
No joins - they're all separate data flows.
Having separate stream instances for subsets of topics would probably work.
Is doesn't seem as clean. It's also slightly more tricky to distribute the
load across separate processes. We'd have to have only one stream thread
allocated to each topic in each process or the first to start grabs all the
load for the whole system.
Are there any details about the number of topics & partitions we can scale
to on example hardware?
Thanks
On Tue, 23 May 2017 at 09:00, Michal Borowiecki <
michal.borowiecki@openbet.com> wrote:
> Hi Mike,
>
> Are you performing any operations (e.g. joins) across all topics?
>
> If so, I'd think increasing the number of partitions is indeed the way to
> go. Partition is the unit of parallelism per topic and all topics are bound
> together in your app in that case.
>
> If not, your other option is to break up your application into a number of
> KafkaStreams instances, each dealing with a subset of topics.
> Hope that helps.
> Michał
>
>
> On 23/05/17 08:47, Mike Gould wrote:
>
> Hi
> We have a couple of hundred topics - each carrying a similar but distinct
> message type but to keep the total partition count down each only has 3
> partitions.
>
> If I start Kafka-streams consuming all topics only 3 threads ever get
> assigned any partitions.
>
> I think the first thread to start gets the first partition of each topic,
> and so on until the 3rd thread, after that all the partitions are assigned
> - any further threads are just left idle.
>
> Is there any way to make the partition assignment smarter and either add a
> random element that moves partitions when further consumers start or
> considers all the partitions of subscribed topics together when assigning
> them?
>
> Our only alternative is creating many more partitions for each topic - and
> we're worried about how far this will scale.
>
> Thank you
> Mike G
>
>
>
>
> --
> <http://www.openbet.com/> Michal Borowiecki
> Senior Software Engineer L4
> T: +44 208 742 1600
>
>
> +44 203 249 8448
>
>
>
> E: michal.borowiecki@openbet.com
> W: www.openbet.com
> OpenBet Ltd
>
> Chiswick Park Building 9
>
> 566 Chiswick High Rd
>
> London
>
> W4 5XT
>
> UK
> <https://www.openbet.com/email_promo>
> This message is confidential and intended only for the addressee. If you
> have received this message in error, please immediately notify the
> postmaster@openbet.com and delete it from your system as well as any
> copies. The content of e-mails as well as traffic data may be monitored by
> OpenBet for employment and security purposes. To protect the environment
> please do not print this e-mail unless necessary. OpenBet Ltd. Registered
> Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4 5XT,
> United Kingdom. A company registered in England and Wales. Registered no.
> 3134634. VAT no. GB927523612
>
--
- MikeG
http://en.wikipedia.org/wiki/Common_misconceptions
<http://en.wikipedia.org/wiki/Special:Random>
Re: Partition assignment with multiple topics
Posted by Michal Borowiecki <mi...@openbet.com>.
Hi Mike,
Are you performing any operations (e.g. joins) across all topics?
If so, I'd think increasing the number of partitions is indeed the way
to go. Partition is the unit of parallelism per topic and all topics are
bound together in your app in that case.
If not, your other option is to break up your application into a number
of KafkaStreams instances, each dealing with a subset of topics.
Hope that helps.
Michał
On 23/05/17 08:47, Mike Gould wrote:
> Hi
> We have a couple of hundred topics - each carrying a similar but distinct
> message type but to keep the total partition count down each only has 3
> partitions.
>
> If I start Kafka-streams consuming all topics only 3 threads ever get
> assigned any partitions.
>
> I think the first thread to start gets the first partition of each topic,
> and so on until the 3rd thread, after that all the partitions are assigned
> - any further threads are just left idle.
>
> Is there any way to make the partition assignment smarter and either add a
> random element that moves partitions when further consumers start or
> considers all the partitions of subscribed topics together when assigning
> them?
>
> Our only alternative is creating many more partitions for each topic - and
> we're worried about how far this will scale.
>
> Thank you
> Mike G
>
>
--
Signature
<http://www.openbet.com/> Michal Borowiecki
Senior Software Engineer L4
T: +44 208 742 1600
+44 203 249 8448
E: michal.borowiecki@openbet.com
W: www.openbet.com <http://www.openbet.com/>
OpenBet Ltd
Chiswick Park Building 9
566 Chiswick High Rd
London
W4 5XT
UK
<https://www.openbet.com/email_promo>
This message is confidential and intended only for the addressee. If you
have received this message in error, please immediately notify the
postmaster@openbet.com <ma...@openbet.com> and delete it
from your system as well as any copies. The content of e-mails as well
as traffic data may be monitored by OpenBet for employment and security
purposes. To protect the environment please do not print this e-mail
unless necessary. OpenBet Ltd. Registered Office: Chiswick Park Building
9, 566 Chiswick High Road, London, W4 5XT, United Kingdom. A company
registered in England and Wales. Registered no. 3134634. VAT no.
GB927523612