the problems of parititons assigned to consumers

[揣立武 <ch...@gmail.com>]

Hi, all. There are two problems when we use kafka about partitions assigned
to consumers.

Problem 1: Partitions will be reassigned to consumers when consumer online
or offline, then the messages latency become higher.

Problem 2: If we have two partitions, only two consumers can consume
messages。How let more consumers to consume, but expand partitions.

Thanks!

Re: the problems of parititons assigned to consumers

[Robert Quinlivan <rq...@signal.co>]

For problem 1, you could implement your own PartitionAssignor class to do
static assignments. The downside of that is that if one consumer goes
offline (due to hardware failure or a bug) then you aren't consuming from
the respective partition.

For problem 2, if you are finding that the throughput with two partitions
is not sufficient, I would first look and see if you have a bottleneck in
your consuming application. If you have already ruled out that possibility,
you may want to increase the partition count and add more consumers.
Consumers acquire a lock on a partition, so if you want more consumers you
must add more partitions.

On Tue, Apr 25, 2017 at 12:25 PM, David Garcia <da...@spiceworks.com>
wrote:

> For Problem 1, you will probably have to either use the low-level API,
> and/or do manual partition assignment.
>
> For problem 2, you can simply re-publish the messages to a new topic with
> more partitions…or, as in the first problem, just use the low level API.
> You can also create more consumer groups (i.e. if you need to add a new
> consumer of your topics, but all consumer groups are “full”, just make a
> new one)
>
> -David
>
> On 4/24/17, 2:01 AM, "揣立武" <ch...@gmail.com> wrote:
>
>     Hi, all. There are two problems when we use kafka about partitions
> assigned
>     to consumers.
>
>     Problem 1: Partitions will be reassigned to consumers when consumer
> online
>     or offline, then the messages latency become higher.
>
>     Problem 2: If we have two partitions, only two consumers can consume
>     messages。How let more consumers to consume, but expand partitions.
>
>     Thanks!
>
>
>


-- 
Robert Quinlivan
Software Engineer, Signal

Re: the problems of parititons assigned to consumers

[David Garcia <da...@spiceworks.com>]

For Problem 1, you will probably have to either use the low-level API, and/or do manual partition assignment.

For problem 2, you can simply re-publish the messages to a new topic with more partitions…or, as in the first problem, just use the low level API.  You can also create more consumer groups (i.e. if you need to add a new consumer of your topics, but all consumer groups are “full”, just make a new one)

-David

On 4/24/17, 2:01 AM, "揣立武" <ch...@gmail.com> wrote:

    Hi, all. There are two problems when we use kafka about partitions assigned
    to consumers.
    
    Problem 1: Partitions will be reassigned to consumers when consumer online
    or offline, then the messages latency become higher.
    
    Problem 2: If we have two partitions, only two consumers can consume
    messages。How let more consumers to consume, but expand partitions.
    
    Thanks!