kafka high level consumer - threads guaranteed to read a single partition?

[Josh J <jo...@gmail.com>]

Hi,

For the kafka high level consumer, if I create exactly the number of
threads as the number of partitions, is there a guarantee that each thread
will be the only thread that reads from a particular partition? I'm
following this example
<https://github.com/bingoohuang/java-sandbox/blob/92318c6d3f2533bbadb253c59a201e4e70f72ad2/src/main/java/org/n3r/sandbox/kafka/ConsumerGroupExample.java>
.

Thanks,
Josh

Re: kafka high level consumer - threads guaranteed to read a single partition?

["Mattijs Ugen (DT)" <ma...@holmes.nl>]

> For the kafka high level consumer, if I create exactly the number of
> threads as the number of partitions, is there a guarantee that each
> thread will be the only thread that reads from a particular partition?
> I'm following this example
> <https://github.com/bingoohuang/java-sandbox/blob/92318c6d3f2533bbadb253c59a201e4e70f72ad2/src/main/java/org/n3r/sandbox/kafka/ConsumerGroupExample.java>.

The documentation for that particular piece of code
(https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example)
states no such guarantee. The main point for the high level consumer is
automatic balancing through zookeeper and the kafka brokers. Should one
of your threads encounter an error, an external source add a partition
to your topic a set of packets be magically dropped, the consumer group
will rebalance the partition assignments and your exactly one thread per
partition system will be lost.

That said, the kafka people will be better able to shed light on this,
you might want to consider posting this to the kafka mailing list
(http://kafka.apache.org/contact.html) rather than the storm one :)

Kind regards,

Mattijs