Re: Partition Count Dilemma

[Antony Stubbs <an...@confluent.io.INVALID>]

Hi there, sorry just checking the user list for "parallel consumer" - you
can use Confluent's Parallel Consumer client library to increase your
concurrency far beyond the number of partitions you have available.

For example in your case - you can process 1,000's of messages in parallel
without touching the partition count.

Have a look here:
https://www.confluent.io/confluent-accelerators/#parallel-consumer

Latest talk here:
https://www.confluent.io/events/kafka-summit-europe-2021/introducing-confluent-labs-parallel-consumer-client/

Github here: https://github.com/confluentinc/parallel-consumer

On Thu, Mar 21, 2019 at 9:18 AM Jérémy Thulliez <je...@zenika.com>
wrote:

> In your case,
>
> Increase the number of partitions on an existing topic will have an impact
> on how the message will be dispatched into partitions.
>
> I mean if you use the default partitionner,  and your message have key, the
> algorithm is 'hash(key) % number of partitions'
>
> That means that a message with the same key, won't go into the same
> partition, before and after the change.
>
> IMO, It was important to notice, in case you need ordering by key.
>
> Jeremy
>


-- 

<https://www.confluent.io>

Antony Stubbs


Follow us:  Blog
<https://confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog>
• Slack <https://slackpass.io/confluentcommunity> • Twitter
<https://twitter.com/ConfluentInc> • YouTube <https://youtube.com/confluent>