Questions about new consumer API

["hsy541@gmail.com" <hs...@gmail.com>]

Hi guys,

I'm interested in the new Consumer API.
http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/

I have couple of question.
1. In this doc it says kafka consumer will automatically do load balance.
Is it based on throughtput or same as what we have now balance the
cardinality among all consumers in same ConsumerGroup? In a real case
different partitions could have different peak time.
2. In the API, threre is subscribe(partition...) method saying not using
group management, does it mean the group.id property will be discarded and
developer has full control of distributing partitions to consumers?
3. Is new API compatible with old broker?
4. Will simple consumer api and high-level consumer api still be supported?

Thanks!

Best,
Siyuan

Re: Questions about new consumer API

[Neha Narkhede <ne...@gmail.com>]

The offsets are keyed on <group, topic, partition> so if you have more than
one owner per partition, they will rewrite each other's offsets and lead to
incorrect state.

On Tue, Dec 2, 2014 at 2:32 PM, hsy541@gmail.com <hs...@gmail.com> wrote:

> Thanks Neha, another question, so if offsets are stored under group.id,
> dose it mean in one group, there should be at most one subscriber for each
> topic partition?
>
> Best,
> Siyuan
>
> On Tue, Dec 2, 2014 at 12:55 PM, Neha Narkhede <ne...@gmail.com>
> wrote:
>
> > 1. In this doc it says kafka consumer will automatically do load balance.
> > Is it based on throughtput or same as what we have now balance the
> > cardinality among all consumers in same ConsumerGroup? In a real case
> > different partitions could have different peak time.
> >
> > Load balancing is still based on # of partitions for the subscribed
> topics
> > and
> > ensuring that each partition has exactly one consumer as the owner.
> >
> > 2. In the API, threre is subscribe(partition...) method saying not using
> > group management, does it mean the group.id property will be discarded
> and
> > developer has full control of distributing partitions to consumers?
> >
> > group.id is also required for offset management, if the user chooses to
> > use
> > Kafka based offset management. The user will have full control over
> > distribution
> > of partitions to consumers.
> >
> > 3. Is new API compatible with old broker?
> >
> > Yes, it will.
> >
> > 4. Will simple consumer api and high-level consumer api still be
> supported?
> >
> > Over time, we will phase out the current high-level and simple consumer
> > since the
> > 0.9 API supports both.
> >
> > Thanks,
> > Neha
> >
> > On Tue, Dec 2, 2014 at 12:07 PM, hsy541@gmail.com <hs...@gmail.com>
> > wrote:
> >
> > > Hi guys,
> > >
> > > I'm interested in the new Consumer API.
> > > http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/
> > >
> > > I have couple of question.
> > > 1. In this doc it says kafka consumer will automatically do load
> balance.
> > > Is it based on throughtput or same as what we have now balance the
> > > cardinality among all consumers in same ConsumerGroup? In a real case
> > > different partitions could have different peak time.
> > > 2. In the API, threre is subscribe(partition...) method saying not
> using
> > > group management, does it mean the group.id property will be discarded
> > and
> > > developer has full control of distributing partitions to consumers?
> > > 3. Is new API compatible with old broker?
> > > 4. Will simple consumer api and high-level consumer api still be
> > supported?
> > >
> > > Thanks!
> > >
> > > Best,
> > > Siyuan
> > >
> >
>

Re: Questions about new consumer API

["hsy541@gmail.com" <hs...@gmail.com>]

Thanks Neha, another question, so if offsets are stored under group.id,
dose it mean in one group, there should be at most one subscriber for each
topic partition?

Best,
Siyuan

On Tue, Dec 2, 2014 at 12:55 PM, Neha Narkhede <ne...@gmail.com>
wrote:

> 1. In this doc it says kafka consumer will automatically do load balance.
> Is it based on throughtput or same as what we have now balance the
> cardinality among all consumers in same ConsumerGroup? In a real case
> different partitions could have different peak time.
>
> Load balancing is still based on # of partitions for the subscribed topics
> and
> ensuring that each partition has exactly one consumer as the owner.
>
> 2. In the API, threre is subscribe(partition...) method saying not using
> group management, does it mean the group.id property will be discarded and
> developer has full control of distributing partitions to consumers?
>
> group.id is also required for offset management, if the user chooses to
> use
> Kafka based offset management. The user will have full control over
> distribution
> of partitions to consumers.
>
> 3. Is new API compatible with old broker?
>
> Yes, it will.
>
> 4. Will simple consumer api and high-level consumer api still be supported?
>
> Over time, we will phase out the current high-level and simple consumer
> since the
> 0.9 API supports both.
>
> Thanks,
> Neha
>
> On Tue, Dec 2, 2014 at 12:07 PM, hsy541@gmail.com <hs...@gmail.com>
> wrote:
>
> > Hi guys,
> >
> > I'm interested in the new Consumer API.
> > http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/
> >
> > I have couple of question.
> > 1. In this doc it says kafka consumer will automatically do load balance.
> > Is it based on throughtput or same as what we have now balance the
> > cardinality among all consumers in same ConsumerGroup? In a real case
> > different partitions could have different peak time.
> > 2. In the API, threre is subscribe(partition...) method saying not using
> > group management, does it mean the group.id property will be discarded
> and
> > developer has full control of distributing partitions to consumers?
> > 3. Is new API compatible with old broker?
> > 4. Will simple consumer api and high-level consumer api still be
> supported?
> >
> > Thanks!
> >
> > Best,
> > Siyuan
> >
>

Re: Questions about new consumer API

[Neha Narkhede <ne...@gmail.com>]

1. In this doc it says kafka consumer will automatically do load balance.
Is it based on throughtput or same as what we have now balance the
cardinality among all consumers in same ConsumerGroup? In a real case
different partitions could have different peak time.

Load balancing is still based on # of partitions for the subscribed topics
and
ensuring that each partition has exactly one consumer as the owner.

2. In the API, threre is subscribe(partition...) method saying not using
group management, does it mean the group.id property will be discarded and
developer has full control of distributing partitions to consumers?

group.id is also required for offset management, if the user chooses to use
Kafka based offset management. The user will have full control over
distribution
of partitions to consumers.

3. Is new API compatible with old broker?

Yes, it will.

4. Will simple consumer api and high-level consumer api still be supported?

Over time, we will phase out the current high-level and simple consumer
since the
0.9 API supports both.

Thanks,
Neha

On Tue, Dec 2, 2014 at 12:07 PM, hsy541@gmail.com <hs...@gmail.com> wrote:

> Hi guys,
>
> I'm interested in the new Consumer API.
> http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/
>
> I have couple of question.
> 1. In this doc it says kafka consumer will automatically do load balance.
> Is it based on throughtput or same as what we have now balance the
> cardinality among all consumers in same ConsumerGroup? In a real case
> different partitions could have different peak time.
> 2. In the API, threre is subscribe(partition...) method saying not using
> group management, does it mean the group.id property will be discarded and
> developer has full control of distributing partitions to consumers?
> 3. Is new API compatible with old broker?
> 4. Will simple consumer api and high-level consumer api still be supported?
>
> Thanks!
>
> Best,
> Siyuan
>