You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Mohit Anchlia <mo...@gmail.com> on 2015/10/19 19:01:36 UTC
Client consumer question
I see most of the consumer examples create a while/for loop and then fetch
messages iteratively. Is that the only way by which clients can consumer
messages? If this is the preferred way then how do you deal with failures,
exceptions such that messages are not lost.
Also, please point me to examples that one would consider as a robust way
of coding consumers.
Re: Client consumer question
Posted by Mohit Anchlia <mo...@gmail.com>.
I read through the documentation however when I try to access Java API
through the link posted on the design page I get "no page found"
http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/kafka/clients/consumer/KafkaConsumer.html
On Wed, Oct 21, 2015 at 9:59 AM, Mohit Anchlia <mo...@gmail.com>
wrote:
> never mind, I found the documentation
>
> On Wed, Oct 21, 2015 at 9:50 AM, Mohit Anchlia <mo...@gmail.com>
> wrote:
>
>> Thanks. Where can I find new Java consumer API documentation with
>> examples?
>>
>> On Tue, Oct 20, 2015 at 6:37 PM, Guozhang Wang <wa...@gmail.com>
>> wrote:
>>
>>> There are a bunch of new features added in 0.9 plus quite a lot of bug
>>> fixes as well, a complete ticket list can be found here:
>>>
>>>
>>> https://issues.apache.org/jira/browse/KAFKA-1686?jql=project%20%3D%20KAFKA%20AND%20fixVersion%20%3D%200.9.0.0%20ORDER%20BY%20updated%20DESC
>>>
>>> In a short summary of the new features, 0.9 introduces:
>>>
>>> 1) security and quota management on the brokers.
>>> 2) new Java consumer.
>>> 3) copycat framework for ingress / egress of Kafka.
>>>
>>> Guozhang
>>>
>>> On Tue, Oct 20, 2015 at 4:32 PM, Mohit Anchlia <mo...@gmail.com>
>>> wrote:
>>>
>>> > Thanks. Are there any other major changes in .9 release other than the
>>> > Consumer changes. Should I wait for .9 or go ahead and performance test
>>> > with .8?
>>> >
>>> > On Tue, Oct 20, 2015 at 3:54 PM, Guozhang Wang <wa...@gmail.com>
>>> wrote:
>>> >
>>> > > We will have a release document for that on the release date, it is
>>> not
>>> > > complete yet.
>>> > >
>>> > > Guozhang
>>> > >
>>> > > On Tue, Oct 20, 2015 at 3:18 PM, Mohit Anchlia <
>>> mohitanchlia@gmail.com>
>>> > > wrote:
>>> > >
>>> > > > Is there a wiki page where I can find all the major design changes
>>> in
>>> > > > 0.9.0?
>>> > > >
>>> > > > On Mon, Oct 19, 2015 at 4:24 PM, Guozhang Wang <wangguoz@gmail.com
>>> >
>>> > > wrote:
>>> > > >
>>> > > > > It is not released yet, we are shooting for Nov. for 0.9.0.
>>> > > > >
>>> > > > > Guozhang
>>> > > > >
>>> > > > > On Mon, Oct 19, 2015 at 4:08 PM, Mohit Anchlia <
>>> > mohitanchlia@gmail.com
>>> > > >
>>> > > > > wrote:
>>> > > > >
>>> > > > > > Is 0.9.0 still under development? I don't see it here:
>>> > > > > > http://kafka.apache.org/downloads.html
>>> > > > > >
>>> > > > > > On Mon, Oct 19, 2015 at 4:05 PM, Guozhang Wang <
>>> wangguoz@gmail.com
>>> > >
>>> > > > > wrote:
>>> > > > > >
>>> > > > > > > The links you are referring are for the old consumer.
>>> > > > > > >
>>> > > > > > > If you are using the ZooKeeper based high-level version of
>>> the
>>> > old
>>> > > > > > consumer
>>> > > > > > > which is described in the second link, then failures are
>>> handled
>>> > > and
>>> > > > > > > abstracted from you so that if there is a failure in the
>>> current
>>> > > > > process,
>>> > > > > > > its fetching partitions will be re-assigned to other
>>> consumers
>>> > > within
>>> > > > > the
>>> > > > > > > same group starting at the last checkpointed offset. And
>>> offsets
>>> > > can
>>> > > > be
>>> > > > > > > either checkpointed periodically or manually throw
>>> > > consumer.commit()
>>> > > > > > calls.
>>> > > > > > >
>>> > > > > > > BTW, in the coming 0.9.0 release there is a new consumer
>>> written
>>> > in
>>> > > > > Java
>>> > > > > > > which uses a poll() based API instead of a stream iterating
>>> API.
>>> > > More
>>> > > > > > > details can be found here in case you are interested in
>>> trying it
>>> > > > out:
>>> > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
>>> > > > > > >
>>> > > > > > > Guozhang
>>> > > > > > >
>>> > > > > > > On Mon, Oct 19, 2015 at 2:54 PM, Mohit Anchlia <
>>> > > > mohitanchlia@gmail.com
>>> > > > > >
>>> > > > > > > wrote:
>>> > > > > > >
>>> > > > > > > > By old consumer you mean version < .8?
>>> > > > > > > >
>>> > > > > > > > Here are the links:
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
>>> > > > > > > >
>>> > > > > >
>>> > > >
>>> >
>>> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
>>> > > > > > > >
>>> > > > > > > > On Mon, Oct 19, 2015 at 12:52 PM, Guozhang Wang <
>>> > > > wangguoz@gmail.com>
>>> > > > > > > > wrote:
>>> > > > > > > >
>>> > > > > > > > > Hi Mohit,
>>> > > > > > > > >
>>> > > > > > > > > Are you referring to the new Java consumer or the old
>>> > consumer?
>>> > > > Or
>>> > > > > > more
>>> > > > > > > > > specifically what examples doc are you referring to?
>>> > > > > > > > >
>>> > > > > > > > > Guozhang
>>> > > > > > > > >
>>> > > > > > > > > On Mon, Oct 19, 2015 at 10:01 AM, Mohit Anchlia <
>>> > > > > > > mohitanchlia@gmail.com>
>>> > > > > > > > > wrote:
>>> > > > > > > > >
>>> > > > > > > > > > I see most of the consumer examples create a while/for
>>> loop
>>> > > and
>>> > > > > > then
>>> > > > > > > > > fetch
>>> > > > > > > > > > messages iteratively. Is that the only way by which
>>> clients
>>> > > can
>>> > > > > > > > consumer
>>> > > > > > > > > > messages? If this is the preferred way then how do you
>>> deal
>>> > > > with
>>> > > > > > > > > failures,
>>> > > > > > > > > > exceptions such that messages are not lost.
>>> > > > > > > > > >
>>> > > > > > > > > > Also, please point me to examples that one would
>>> consider
>>> > as
>>> > > a
>>> > > > > > robust
>>> > > > > > > > way
>>> > > > > > > > > > of coding consumers.
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > > --
>>> > > > > > > > > -- Guozhang
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > --
>>> > > > > > > -- Guozhang
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > --
>>> > > > > -- Guozhang
>>> > > > >
>>> > > >
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > -- Guozhang
>>> > >
>>> >
>>>
>>>
>>>
>>> --
>>> -- Guozhang
>>>
>>
>>
>
Re: Client consumer question
Posted by Mohit Anchlia <mo...@gmail.com>.
never mind, I found the documentation
On Wed, Oct 21, 2015 at 9:50 AM, Mohit Anchlia <mo...@gmail.com>
wrote:
> Thanks. Where can I find new Java consumer API documentation with
> examples?
>
> On Tue, Oct 20, 2015 at 6:37 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
>> There are a bunch of new features added in 0.9 plus quite a lot of bug
>> fixes as well, a complete ticket list can be found here:
>>
>>
>> https://issues.apache.org/jira/browse/KAFKA-1686?jql=project%20%3D%20KAFKA%20AND%20fixVersion%20%3D%200.9.0.0%20ORDER%20BY%20updated%20DESC
>>
>> In a short summary of the new features, 0.9 introduces:
>>
>> 1) security and quota management on the brokers.
>> 2) new Java consumer.
>> 3) copycat framework for ingress / egress of Kafka.
>>
>> Guozhang
>>
>> On Tue, Oct 20, 2015 at 4:32 PM, Mohit Anchlia <mo...@gmail.com>
>> wrote:
>>
>> > Thanks. Are there any other major changes in .9 release other than the
>> > Consumer changes. Should I wait for .9 or go ahead and performance test
>> > with .8?
>> >
>> > On Tue, Oct 20, 2015 at 3:54 PM, Guozhang Wang <wa...@gmail.com>
>> wrote:
>> >
>> > > We will have a release document for that on the release date, it is
>> not
>> > > complete yet.
>> > >
>> > > Guozhang
>> > >
>> > > On Tue, Oct 20, 2015 at 3:18 PM, Mohit Anchlia <
>> mohitanchlia@gmail.com>
>> > > wrote:
>> > >
>> > > > Is there a wiki page where I can find all the major design changes
>> in
>> > > > 0.9.0?
>> > > >
>> > > > On Mon, Oct 19, 2015 at 4:24 PM, Guozhang Wang <wa...@gmail.com>
>> > > wrote:
>> > > >
>> > > > > It is not released yet, we are shooting for Nov. for 0.9.0.
>> > > > >
>> > > > > Guozhang
>> > > > >
>> > > > > On Mon, Oct 19, 2015 at 4:08 PM, Mohit Anchlia <
>> > mohitanchlia@gmail.com
>> > > >
>> > > > > wrote:
>> > > > >
>> > > > > > Is 0.9.0 still under development? I don't see it here:
>> > > > > > http://kafka.apache.org/downloads.html
>> > > > > >
>> > > > > > On Mon, Oct 19, 2015 at 4:05 PM, Guozhang Wang <
>> wangguoz@gmail.com
>> > >
>> > > > > wrote:
>> > > > > >
>> > > > > > > The links you are referring are for the old consumer.
>> > > > > > >
>> > > > > > > If you are using the ZooKeeper based high-level version of the
>> > old
>> > > > > > consumer
>> > > > > > > which is described in the second link, then failures are
>> handled
>> > > and
>> > > > > > > abstracted from you so that if there is a failure in the
>> current
>> > > > > process,
>> > > > > > > its fetching partitions will be re-assigned to other consumers
>> > > within
>> > > > > the
>> > > > > > > same group starting at the last checkpointed offset. And
>> offsets
>> > > can
>> > > > be
>> > > > > > > either checkpointed periodically or manually throw
>> > > consumer.commit()
>> > > > > > calls.
>> > > > > > >
>> > > > > > > BTW, in the coming 0.9.0 release there is a new consumer
>> written
>> > in
>> > > > > Java
>> > > > > > > which uses a poll() based API instead of a stream iterating
>> API.
>> > > More
>> > > > > > > details can be found here in case you are interested in
>> trying it
>> > > > out:
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
>> > > > > > >
>> > > > > > > Guozhang
>> > > > > > >
>> > > > > > > On Mon, Oct 19, 2015 at 2:54 PM, Mohit Anchlia <
>> > > > mohitanchlia@gmail.com
>> > > > > >
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > By old consumer you mean version < .8?
>> > > > > > > >
>> > > > > > > > Here are the links:
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
>> > > > > > > >
>> > > > > >
>> > > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
>> > > > > > > >
>> > > > > > > > On Mon, Oct 19, 2015 at 12:52 PM, Guozhang Wang <
>> > > > wangguoz@gmail.com>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Hi Mohit,
>> > > > > > > > >
>> > > > > > > > > Are you referring to the new Java consumer or the old
>> > consumer?
>> > > > Or
>> > > > > > more
>> > > > > > > > > specifically what examples doc are you referring to?
>> > > > > > > > >
>> > > > > > > > > Guozhang
>> > > > > > > > >
>> > > > > > > > > On Mon, Oct 19, 2015 at 10:01 AM, Mohit Anchlia <
>> > > > > > > mohitanchlia@gmail.com>
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > I see most of the consumer examples create a while/for
>> loop
>> > > and
>> > > > > > then
>> > > > > > > > > fetch
>> > > > > > > > > > messages iteratively. Is that the only way by which
>> clients
>> > > can
>> > > > > > > > consumer
>> > > > > > > > > > messages? If this is the preferred way then how do you
>> deal
>> > > > with
>> > > > > > > > > failures,
>> > > > > > > > > > exceptions such that messages are not lost.
>> > > > > > > > > >
>> > > > > > > > > > Also, please point me to examples that one would
>> consider
>> > as
>> > > a
>> > > > > > robust
>> > > > > > > > way
>> > > > > > > > > > of coding consumers.
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > --
>> > > > > > > > > -- Guozhang
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > --
>> > > > > > > -- Guozhang
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > -- Guozhang
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > -- Guozhang
>> > >
>> >
>>
>>
>>
>> --
>> -- Guozhang
>>
>
>
Re: Client consumer question
Posted by Mohit Anchlia <mo...@gmail.com>.
Thanks. Where can I find new Java consumer API documentation with examples?
On Tue, Oct 20, 2015 at 6:37 PM, Guozhang Wang <wa...@gmail.com> wrote:
> There are a bunch of new features added in 0.9 plus quite a lot of bug
> fixes as well, a complete ticket list can be found here:
>
>
> https://issues.apache.org/jira/browse/KAFKA-1686?jql=project%20%3D%20KAFKA%20AND%20fixVersion%20%3D%200.9.0.0%20ORDER%20BY%20updated%20DESC
>
> In a short summary of the new features, 0.9 introduces:
>
> 1) security and quota management on the brokers.
> 2) new Java consumer.
> 3) copycat framework for ingress / egress of Kafka.
>
> Guozhang
>
> On Tue, Oct 20, 2015 at 4:32 PM, Mohit Anchlia <mo...@gmail.com>
> wrote:
>
> > Thanks. Are there any other major changes in .9 release other than the
> > Consumer changes. Should I wait for .9 or go ahead and performance test
> > with .8?
> >
> > On Tue, Oct 20, 2015 at 3:54 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >
> > > We will have a release document for that on the release date, it is not
> > > complete yet.
> > >
> > > Guozhang
> > >
> > > On Tue, Oct 20, 2015 at 3:18 PM, Mohit Anchlia <mohitanchlia@gmail.com
> >
> > > wrote:
> > >
> > > > Is there a wiki page where I can find all the major design changes in
> > > > 0.9.0?
> > > >
> > > > On Mon, Oct 19, 2015 at 4:24 PM, Guozhang Wang <wa...@gmail.com>
> > > wrote:
> > > >
> > > > > It is not released yet, we are shooting for Nov. for 0.9.0.
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Mon, Oct 19, 2015 at 4:08 PM, Mohit Anchlia <
> > mohitanchlia@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Is 0.9.0 still under development? I don't see it here:
> > > > > > http://kafka.apache.org/downloads.html
> > > > > >
> > > > > > On Mon, Oct 19, 2015 at 4:05 PM, Guozhang Wang <
> wangguoz@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > The links you are referring are for the old consumer.
> > > > > > >
> > > > > > > If you are using the ZooKeeper based high-level version of the
> > old
> > > > > > consumer
> > > > > > > which is described in the second link, then failures are
> handled
> > > and
> > > > > > > abstracted from you so that if there is a failure in the
> current
> > > > > process,
> > > > > > > its fetching partitions will be re-assigned to other consumers
> > > within
> > > > > the
> > > > > > > same group starting at the last checkpointed offset. And
> offsets
> > > can
> > > > be
> > > > > > > either checkpointed periodically or manually throw
> > > consumer.commit()
> > > > > > calls.
> > > > > > >
> > > > > > > BTW, in the coming 0.9.0 release there is a new consumer
> written
> > in
> > > > > Java
> > > > > > > which uses a poll() based API instead of a stream iterating
> API.
> > > More
> > > > > > > details can be found here in case you are interested in trying
> it
> > > > out:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
> > > > > > >
> > > > > > > Guozhang
> > > > > > >
> > > > > > > On Mon, Oct 19, 2015 at 2:54 PM, Mohit Anchlia <
> > > > mohitanchlia@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > By old consumer you mean version < .8?
> > > > > > > >
> > > > > > > > Here are the links:
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
> > > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
> > > > > > > >
> > > > > > > > On Mon, Oct 19, 2015 at 12:52 PM, Guozhang Wang <
> > > > wangguoz@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Mohit,
> > > > > > > > >
> > > > > > > > > Are you referring to the new Java consumer or the old
> > consumer?
> > > > Or
> > > > > > more
> > > > > > > > > specifically what examples doc are you referring to?
> > > > > > > > >
> > > > > > > > > Guozhang
> > > > > > > > >
> > > > > > > > > On Mon, Oct 19, 2015 at 10:01 AM, Mohit Anchlia <
> > > > > > > mohitanchlia@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I see most of the consumer examples create a while/for
> loop
> > > and
> > > > > > then
> > > > > > > > > fetch
> > > > > > > > > > messages iteratively. Is that the only way by which
> clients
> > > can
> > > > > > > > consumer
> > > > > > > > > > messages? If this is the preferred way then how do you
> deal
> > > > with
> > > > > > > > > failures,
> > > > > > > > > > exceptions such that messages are not lost.
> > > > > > > > > >
> > > > > > > > > > Also, please point me to examples that one would consider
> > as
> > > a
> > > > > > robust
> > > > > > > > way
> > > > > > > > > > of coding consumers.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > -- Guozhang
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > -- Guozhang
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
>
>
> --
> -- Guozhang
>
Re: Client consumer question
Posted by Guozhang Wang <wa...@gmail.com>.
There are a bunch of new features added in 0.9 plus quite a lot of bug
fixes as well, a complete ticket list can be found here:
https://issues.apache.org/jira/browse/KAFKA-1686?jql=project%20%3D%20KAFKA%20AND%20fixVersion%20%3D%200.9.0.0%20ORDER%20BY%20updated%20DESC
In a short summary of the new features, 0.9 introduces:
1) security and quota management on the brokers.
2) new Java consumer.
3) copycat framework for ingress / egress of Kafka.
Guozhang
On Tue, Oct 20, 2015 at 4:32 PM, Mohit Anchlia <mo...@gmail.com>
wrote:
> Thanks. Are there any other major changes in .9 release other than the
> Consumer changes. Should I wait for .9 or go ahead and performance test
> with .8?
>
> On Tue, Oct 20, 2015 at 3:54 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > We will have a release document for that on the release date, it is not
> > complete yet.
> >
> > Guozhang
> >
> > On Tue, Oct 20, 2015 at 3:18 PM, Mohit Anchlia <mo...@gmail.com>
> > wrote:
> >
> > > Is there a wiki page where I can find all the major design changes in
> > > 0.9.0?
> > >
> > > On Mon, Oct 19, 2015 at 4:24 PM, Guozhang Wang <wa...@gmail.com>
> > wrote:
> > >
> > > > It is not released yet, we are shooting for Nov. for 0.9.0.
> > > >
> > > > Guozhang
> > > >
> > > > On Mon, Oct 19, 2015 at 4:08 PM, Mohit Anchlia <
> mohitanchlia@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Is 0.9.0 still under development? I don't see it here:
> > > > > http://kafka.apache.org/downloads.html
> > > > >
> > > > > On Mon, Oct 19, 2015 at 4:05 PM, Guozhang Wang <wangguoz@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > The links you are referring are for the old consumer.
> > > > > >
> > > > > > If you are using the ZooKeeper based high-level version of the
> old
> > > > > consumer
> > > > > > which is described in the second link, then failures are handled
> > and
> > > > > > abstracted from you so that if there is a failure in the current
> > > > process,
> > > > > > its fetching partitions will be re-assigned to other consumers
> > within
> > > > the
> > > > > > same group starting at the last checkpointed offset. And offsets
> > can
> > > be
> > > > > > either checkpointed periodically or manually throw
> > consumer.commit()
> > > > > calls.
> > > > > >
> > > > > > BTW, in the coming 0.9.0 release there is a new consumer written
> in
> > > > Java
> > > > > > which uses a poll() based API instead of a stream iterating API.
> > More
> > > > > > details can be found here in case you are interested in trying it
> > > out:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > > On Mon, Oct 19, 2015 at 2:54 PM, Mohit Anchlia <
> > > mohitanchlia@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > By old consumer you mean version < .8?
> > > > > > >
> > > > > > > Here are the links:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
> > > > > > >
> > > > > > > On Mon, Oct 19, 2015 at 12:52 PM, Guozhang Wang <
> > > wangguoz@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Mohit,
> > > > > > > >
> > > > > > > > Are you referring to the new Java consumer or the old
> consumer?
> > > Or
> > > > > more
> > > > > > > > specifically what examples doc are you referring to?
> > > > > > > >
> > > > > > > > Guozhang
> > > > > > > >
> > > > > > > > On Mon, Oct 19, 2015 at 10:01 AM, Mohit Anchlia <
> > > > > > mohitanchlia@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I see most of the consumer examples create a while/for loop
> > and
> > > > > then
> > > > > > > > fetch
> > > > > > > > > messages iteratively. Is that the only way by which clients
> > can
> > > > > > > consumer
> > > > > > > > > messages? If this is the preferred way then how do you deal
> > > with
> > > > > > > > failures,
> > > > > > > > > exceptions such that messages are not lost.
> > > > > > > > >
> > > > > > > > > Also, please point me to examples that one would consider
> as
> > a
> > > > > robust
> > > > > > > way
> > > > > > > > > of coding consumers.
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > -- Guozhang
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -- Guozhang
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
--
-- Guozhang
Re: Client consumer question
Posted by Mohit Anchlia <mo...@gmail.com>.
Thanks. Are there any other major changes in .9 release other than the
Consumer changes. Should I wait for .9 or go ahead and performance test
with .8?
On Tue, Oct 20, 2015 at 3:54 PM, Guozhang Wang <wa...@gmail.com> wrote:
> We will have a release document for that on the release date, it is not
> complete yet.
>
> Guozhang
>
> On Tue, Oct 20, 2015 at 3:18 PM, Mohit Anchlia <mo...@gmail.com>
> wrote:
>
> > Is there a wiki page where I can find all the major design changes in
> > 0.9.0?
> >
> > On Mon, Oct 19, 2015 at 4:24 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >
> > > It is not released yet, we are shooting for Nov. for 0.9.0.
> > >
> > > Guozhang
> > >
> > > On Mon, Oct 19, 2015 at 4:08 PM, Mohit Anchlia <mohitanchlia@gmail.com
> >
> > > wrote:
> > >
> > > > Is 0.9.0 still under development? I don't see it here:
> > > > http://kafka.apache.org/downloads.html
> > > >
> > > > On Mon, Oct 19, 2015 at 4:05 PM, Guozhang Wang <wa...@gmail.com>
> > > wrote:
> > > >
> > > > > The links you are referring are for the old consumer.
> > > > >
> > > > > If you are using the ZooKeeper based high-level version of the old
> > > > consumer
> > > > > which is described in the second link, then failures are handled
> and
> > > > > abstracted from you so that if there is a failure in the current
> > > process,
> > > > > its fetching partitions will be re-assigned to other consumers
> within
> > > the
> > > > > same group starting at the last checkpointed offset. And offsets
> can
> > be
> > > > > either checkpointed periodically or manually throw
> consumer.commit()
> > > > calls.
> > > > >
> > > > > BTW, in the coming 0.9.0 release there is a new consumer written in
> > > Java
> > > > > which uses a poll() based API instead of a stream iterating API.
> More
> > > > > details can be found here in case you are interested in trying it
> > out:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Mon, Oct 19, 2015 at 2:54 PM, Mohit Anchlia <
> > mohitanchlia@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > By old consumer you mean version < .8?
> > > > > >
> > > > > > Here are the links:
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
> > > > > >
> > > > > > On Mon, Oct 19, 2015 at 12:52 PM, Guozhang Wang <
> > wangguoz@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Mohit,
> > > > > > >
> > > > > > > Are you referring to the new Java consumer or the old consumer?
> > Or
> > > > more
> > > > > > > specifically what examples doc are you referring to?
> > > > > > >
> > > > > > > Guozhang
> > > > > > >
> > > > > > > On Mon, Oct 19, 2015 at 10:01 AM, Mohit Anchlia <
> > > > > mohitanchlia@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I see most of the consumer examples create a while/for loop
> and
> > > > then
> > > > > > > fetch
> > > > > > > > messages iteratively. Is that the only way by which clients
> can
> > > > > > consumer
> > > > > > > > messages? If this is the preferred way then how do you deal
> > with
> > > > > > > failures,
> > > > > > > > exceptions such that messages are not lost.
> > > > > > > >
> > > > > > > > Also, please point me to examples that one would consider as
> a
> > > > robust
> > > > > > way
> > > > > > > > of coding consumers.
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > -- Guozhang
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
>
>
> --
> -- Guozhang
>
Re: Client consumer question
Posted by Guozhang Wang <wa...@gmail.com>.
We will have a release document for that on the release date, it is not
complete yet.
Guozhang
On Tue, Oct 20, 2015 at 3:18 PM, Mohit Anchlia <mo...@gmail.com>
wrote:
> Is there a wiki page where I can find all the major design changes in
> 0.9.0?
>
> On Mon, Oct 19, 2015 at 4:24 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > It is not released yet, we are shooting for Nov. for 0.9.0.
> >
> > Guozhang
> >
> > On Mon, Oct 19, 2015 at 4:08 PM, Mohit Anchlia <mo...@gmail.com>
> > wrote:
> >
> > > Is 0.9.0 still under development? I don't see it here:
> > > http://kafka.apache.org/downloads.html
> > >
> > > On Mon, Oct 19, 2015 at 4:05 PM, Guozhang Wang <wa...@gmail.com>
> > wrote:
> > >
> > > > The links you are referring are for the old consumer.
> > > >
> > > > If you are using the ZooKeeper based high-level version of the old
> > > consumer
> > > > which is described in the second link, then failures are handled and
> > > > abstracted from you so that if there is a failure in the current
> > process,
> > > > its fetching partitions will be re-assigned to other consumers within
> > the
> > > > same group starting at the last checkpointed offset. And offsets can
> be
> > > > either checkpointed periodically or manually throw consumer.commit()
> > > calls.
> > > >
> > > > BTW, in the coming 0.9.0 release there is a new consumer written in
> > Java
> > > > which uses a poll() based API instead of a stream iterating API. More
> > > > details can be found here in case you are interested in trying it
> out:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
> > > >
> > > > Guozhang
> > > >
> > > > On Mon, Oct 19, 2015 at 2:54 PM, Mohit Anchlia <
> mohitanchlia@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > By old consumer you mean version < .8?
> > > > >
> > > > > Here are the links:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
> > > > >
> > > > > On Mon, Oct 19, 2015 at 12:52 PM, Guozhang Wang <
> wangguoz@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Mohit,
> > > > > >
> > > > > > Are you referring to the new Java consumer or the old consumer?
> Or
> > > more
> > > > > > specifically what examples doc are you referring to?
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > > On Mon, Oct 19, 2015 at 10:01 AM, Mohit Anchlia <
> > > > mohitanchlia@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I see most of the consumer examples create a while/for loop and
> > > then
> > > > > > fetch
> > > > > > > messages iteratively. Is that the only way by which clients can
> > > > > consumer
> > > > > > > messages? If this is the preferred way then how do you deal
> with
> > > > > > failures,
> > > > > > > exceptions such that messages are not lost.
> > > > > > >
> > > > > > > Also, please point me to examples that one would consider as a
> > > robust
> > > > > way
> > > > > > > of coding consumers.
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -- Guozhang
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
--
-- Guozhang
Re: Client consumer question
Posted by Mohit Anchlia <mo...@gmail.com>.
Is there a wiki page where I can find all the major design changes in 0.9.0?
On Mon, Oct 19, 2015 at 4:24 PM, Guozhang Wang <wa...@gmail.com> wrote:
> It is not released yet, we are shooting for Nov. for 0.9.0.
>
> Guozhang
>
> On Mon, Oct 19, 2015 at 4:08 PM, Mohit Anchlia <mo...@gmail.com>
> wrote:
>
> > Is 0.9.0 still under development? I don't see it here:
> > http://kafka.apache.org/downloads.html
> >
> > On Mon, Oct 19, 2015 at 4:05 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >
> > > The links you are referring are for the old consumer.
> > >
> > > If you are using the ZooKeeper based high-level version of the old
> > consumer
> > > which is described in the second link, then failures are handled and
> > > abstracted from you so that if there is a failure in the current
> process,
> > > its fetching partitions will be re-assigned to other consumers within
> the
> > > same group starting at the last checkpointed offset. And offsets can be
> > > either checkpointed periodically or manually throw consumer.commit()
> > calls.
> > >
> > > BTW, in the coming 0.9.0 release there is a new consumer written in
> Java
> > > which uses a poll() based API instead of a stream iterating API. More
> > > details can be found here in case you are interested in trying it out:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
> > >
> > > Guozhang
> > >
> > > On Mon, Oct 19, 2015 at 2:54 PM, Mohit Anchlia <mohitanchlia@gmail.com
> >
> > > wrote:
> > >
> > > > By old consumer you mean version < .8?
> > > >
> > > > Here are the links:
> > > >
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
> > > >
> > > > On Mon, Oct 19, 2015 at 12:52 PM, Guozhang Wang <wa...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Mohit,
> > > > >
> > > > > Are you referring to the new Java consumer or the old consumer? Or
> > more
> > > > > specifically what examples doc are you referring to?
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Mon, Oct 19, 2015 at 10:01 AM, Mohit Anchlia <
> > > mohitanchlia@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > I see most of the consumer examples create a while/for loop and
> > then
> > > > > fetch
> > > > > > messages iteratively. Is that the only way by which clients can
> > > > consumer
> > > > > > messages? If this is the preferred way then how do you deal with
> > > > > failures,
> > > > > > exceptions such that messages are not lost.
> > > > > >
> > > > > > Also, please point me to examples that one would consider as a
> > robust
> > > > way
> > > > > > of coding consumers.
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
>
>
> --
> -- Guozhang
>
Re: Client consumer question
Posted by Guozhang Wang <wa...@gmail.com>.
It is not released yet, we are shooting for Nov. for 0.9.0.
Guozhang
On Mon, Oct 19, 2015 at 4:08 PM, Mohit Anchlia <mo...@gmail.com>
wrote:
> Is 0.9.0 still under development? I don't see it here:
> http://kafka.apache.org/downloads.html
>
> On Mon, Oct 19, 2015 at 4:05 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > The links you are referring are for the old consumer.
> >
> > If you are using the ZooKeeper based high-level version of the old
> consumer
> > which is described in the second link, then failures are handled and
> > abstracted from you so that if there is a failure in the current process,
> > its fetching partitions will be re-assigned to other consumers within the
> > same group starting at the last checkpointed offset. And offsets can be
> > either checkpointed periodically or manually throw consumer.commit()
> calls.
> >
> > BTW, in the coming 0.9.0 release there is a new consumer written in Java
> > which uses a poll() based API instead of a stream iterating API. More
> > details can be found here in case you are interested in trying it out:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
> >
> > Guozhang
> >
> > On Mon, Oct 19, 2015 at 2:54 PM, Mohit Anchlia <mo...@gmail.com>
> > wrote:
> >
> > > By old consumer you mean version < .8?
> > >
> > > Here are the links:
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
> > >
> > > On Mon, Oct 19, 2015 at 12:52 PM, Guozhang Wang <wa...@gmail.com>
> > > wrote:
> > >
> > > > Hi Mohit,
> > > >
> > > > Are you referring to the new Java consumer or the old consumer? Or
> more
> > > > specifically what examples doc are you referring to?
> > > >
> > > > Guozhang
> > > >
> > > > On Mon, Oct 19, 2015 at 10:01 AM, Mohit Anchlia <
> > mohitanchlia@gmail.com>
> > > > wrote:
> > > >
> > > > > I see most of the consumer examples create a while/for loop and
> then
> > > > fetch
> > > > > messages iteratively. Is that the only way by which clients can
> > > consumer
> > > > > messages? If this is the preferred way then how do you deal with
> > > > failures,
> > > > > exceptions such that messages are not lost.
> > > > >
> > > > > Also, please point me to examples that one would consider as a
> robust
> > > way
> > > > > of coding consumers.
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
--
-- Guozhang
Re: Client consumer question
Posted by Mohit Anchlia <mo...@gmail.com>.
Is 0.9.0 still under development? I don't see it here:
http://kafka.apache.org/downloads.html
On Mon, Oct 19, 2015 at 4:05 PM, Guozhang Wang <wa...@gmail.com> wrote:
> The links you are referring are for the old consumer.
>
> If you are using the ZooKeeper based high-level version of the old consumer
> which is described in the second link, then failures are handled and
> abstracted from you so that if there is a failure in the current process,
> its fetching partitions will be re-assigned to other consumers within the
> same group starting at the last checkpointed offset. And offsets can be
> either checkpointed periodically or manually throw consumer.commit() calls.
>
> BTW, in the coming 0.9.0 release there is a new consumer written in Java
> which uses a poll() based API instead of a stream iterating API. More
> details can be found here in case you are interested in trying it out:
>
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
>
> Guozhang
>
> On Mon, Oct 19, 2015 at 2:54 PM, Mohit Anchlia <mo...@gmail.com>
> wrote:
>
> > By old consumer you mean version < .8?
> >
> > Here are the links:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
> > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
> >
> > On Mon, Oct 19, 2015 at 12:52 PM, Guozhang Wang <wa...@gmail.com>
> > wrote:
> >
> > > Hi Mohit,
> > >
> > > Are you referring to the new Java consumer or the old consumer? Or more
> > > specifically what examples doc are you referring to?
> > >
> > > Guozhang
> > >
> > > On Mon, Oct 19, 2015 at 10:01 AM, Mohit Anchlia <
> mohitanchlia@gmail.com>
> > > wrote:
> > >
> > > > I see most of the consumer examples create a while/for loop and then
> > > fetch
> > > > messages iteratively. Is that the only way by which clients can
> > consumer
> > > > messages? If this is the preferred way then how do you deal with
> > > failures,
> > > > exceptions such that messages are not lost.
> > > >
> > > > Also, please point me to examples that one would consider as a robust
> > way
> > > > of coding consumers.
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
>
>
> --
> -- Guozhang
>
Re: Client consumer question
Posted by Guozhang Wang <wa...@gmail.com>.
The links you are referring are for the old consumer.
If you are using the ZooKeeper based high-level version of the old consumer
which is described in the second link, then failures are handled and
abstracted from you so that if there is a failure in the current process,
its fetching partitions will be re-assigned to other consumers within the
same group starting at the last checkpointed offset. And offsets can be
either checkpointed periodically or manually throw consumer.commit() calls.
BTW, in the coming 0.9.0 release there is a new consumer written in Java
which uses a poll() based API instead of a stream iterating API. More
details can be found here in case you are interested in trying it out:
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
Guozhang
On Mon, Oct 19, 2015 at 2:54 PM, Mohit Anchlia <mo...@gmail.com>
wrote:
> By old consumer you mean version < .8?
>
> Here are the links:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
>
> On Mon, Oct 19, 2015 at 12:52 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
>
> > Hi Mohit,
> >
> > Are you referring to the new Java consumer or the old consumer? Or more
> > specifically what examples doc are you referring to?
> >
> > Guozhang
> >
> > On Mon, Oct 19, 2015 at 10:01 AM, Mohit Anchlia <mo...@gmail.com>
> > wrote:
> >
> > > I see most of the consumer examples create a while/for loop and then
> > fetch
> > > messages iteratively. Is that the only way by which clients can
> consumer
> > > messages? If this is the preferred way then how do you deal with
> > failures,
> > > exceptions such that messages are not lost.
> > >
> > > Also, please point me to examples that one would consider as a robust
> way
> > > of coding consumers.
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
--
-- Guozhang
Re: Client consumer question
Posted by Mohit Anchlia <mo...@gmail.com>.
By old consumer you mean version < .8?
Here are the links:
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
On Mon, Oct 19, 2015 at 12:52 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Hi Mohit,
>
> Are you referring to the new Java consumer or the old consumer? Or more
> specifically what examples doc are you referring to?
>
> Guozhang
>
> On Mon, Oct 19, 2015 at 10:01 AM, Mohit Anchlia <mo...@gmail.com>
> wrote:
>
> > I see most of the consumer examples create a while/for loop and then
> fetch
> > messages iteratively. Is that the only way by which clients can consumer
> > messages? If this is the preferred way then how do you deal with
> failures,
> > exceptions such that messages are not lost.
> >
> > Also, please point me to examples that one would consider as a robust way
> > of coding consumers.
> >
>
>
>
> --
> -- Guozhang
>
Re: Client consumer question
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Mohit,
Are you referring to the new Java consumer or the old consumer? Or more
specifically what examples doc are you referring to?
Guozhang
On Mon, Oct 19, 2015 at 10:01 AM, Mohit Anchlia <mo...@gmail.com>
wrote:
> I see most of the consumer examples create a while/for loop and then fetch
> messages iteratively. Is that the only way by which clients can consumer
> messages? If this is the preferred way then how do you deal with failures,
> exceptions such that messages are not lost.
>
> Also, please point me to examples that one would consider as a robust way
> of coding consumers.
>
--
-- Guozhang