Kafka Consumer - Java

[vivek shankar <vi...@gmail.com>]

Hello All,

Can you please help with the below :

I was reading up on Kafka 0.9 API version and came across the below :

The following is a draft design that uses a high-available consumer
coordinator at the broker side to handle consumer rebalance. By migrating
the rebalance logic from the consumer to the coordinator we can resolve the
consumer split brain problem and help thinner the consumer client.

Please can you help with the below questions  :

I have a single topic with single consumer group id. There are 20
partitions. The consumer client is running on a single node which has 20
threads. We are using zookeeper to fetch messages from Kafka.

1) When using Kafka 0.8.2 API , how can re balancing be done? Can the same
consumer project be deployed into multiple nodes and will zookeeper
automatically do the rebalancing?

2) What are the changes that we need to take care of when upgrading to 0.9
API version? Should underlying Kafka installation be changed  by itself or
is it suffice to just use 0.9 consumer api?

3) Can zookeeper be completely eliminated when upgrading to 0.9 version?

Thank You!

Regards

Re: Kafka Consumer - Java

[Gwen Shapira <gw...@confluent.io>]

See below

On Tue, Feb 23, 2016 at 11:45 AM, vivek shankar <vi...@gmail.com>
wrote:

> Hello All,
>
> Can you please help with the below :
>
> I was reading up on Kafka 0.9 API version and came across the below :
>
> The following is a draft design that uses a high-available consumer
> coordinator at the broker side to handle consumer rebalance. By migrating
> the rebalance logic from the consumer to the coordinator we can resolve the
> consumer split brain problem and help thinner the consumer client.
>
> Please can you help with the below questions  :
>
> I have a single topic with single consumer group id. There are 20
> partitions. The consumer client is running on a single node which has 20
> threads. We are using zookeeper to fetch messages from Kafka.
>
> 1) When using Kafka 0.8.2 API , how can re balancing be done? Can the same
> consumer project be deployed into multiple nodes and will zookeeper
> automatically do the rebalancing?
>

0.8.2 has two consumer APIs. The high level consumer (Also known as
ZookeeperConsumerConnector) has the concept of Consumer Groups. If the
"same consumer" is deployed on different nodes and they all have the same
group.id, the consumers will discover each other via zookeeper and balance
consumption of partitions between them.


>
> 2) What are the changes that we need to take care of when upgrading to 0.9
> API version? Should underlying Kafka installation be changed  by itself or
> is it suffice to just use 0.9 consumer api?
>

Apache Kafka upgrades are always broker first, so you much upgrade the
Kafka brokers before using the 0.9 API on the clients.
The 0.9 API is completely different, so you will also need to rewrite quite
a bit of code when upgrading.


>
> 3) Can zookeeper be completely eliminated when upgrading to 0.9 version?
>

In 0.9. clients no longer need to connect to Zookeeper. The brokers still
use Zookeeper for things like topic configuration, controller election, etc.


>
> Thank You!
>
> Regards
>

Hope this helps,

Gwen