You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Binita Bharati <bi...@gmail.com> on 2014/02/25 07:27:43 UTC
Consumer group ID for high level consumer
Hi all,
I am referring to this e.g:
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example.
What is the consumer group ID being referred here ?
Thanks
Binita
Re: Consumer group ID for high level consumer
Posted by Binita Bharati <bi...@gmail.com>.
Hi Neha,
I had to set the "auto.offset.reset" to "smallest" in the particular case
where consumer threads weren't running when the data was produced.
Then on, its working as you said, ie , as long as the group.id is same, the
consumer threads continue to receive new data.
Thanks
On Mon, Mar 3, 2014 at 10:55 PM, Neha Narkhede <ne...@gmail.com>wrote:
> Question : How to make the consumer threads receive the older data that
> was produced when the consumer threads aren't running ?
>
> As long as you make sure the group.id doesn't change, the consumer will
> always
> consume data from where it left off during its last run. In other words,
> the consumer
> can consume data that was produced when it wasn't running, if it is always
> started
> using the same group.id
>
> Thanks,
> Neha
>
>
> On Mon, Mar 3, 2014 at 8:02 AM, Binita Bharati <binita.bharati@gmail.com
> >wrote:
>
> > Hi all,
> >
> > The HighLevelConsumer is working fine. Ref :
> > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
> .
> >
> > I was expecting the consumer threads to be able to receive older data (
> ie
> > the data put into the topic when the consumer threads weren't running )
> > That was the root of the problem, and once, I started my consumer
> > application first, and then, generated fresh data into the topic, the
> > consumer threads started receiving the new data.
> >
> > Question : How to make the consumer threads receive the older data that
> > was produced when the consumer threads aren't running ?
> >
> > Thanks
> >
> >
> > On Wed, Feb 26, 2014 at 6:30 PM, Martin Kleppmann
> > <mk...@linkedin.com>wrote:
> >
> > > Hi Binita,
> > >
> > > The consumer group (group.id) is a mechanism for sharing the load of
> > > consuming a high-volume topic between multiple consumers. If you don't
> > set
> > > a group ID, each consumer consumes all the partitions of a topic. If
> you
> > > set several consumers to the same group ID, the partitions of the topic
> > > you're consuming will be shared amongst the consumers in that group, so
> > > that each message is delivered to only one of the consumers in the
> group.
> > >
> > > You can set the group ID to be whatever you like.
> > >
> > > Hope that helps,
> > > Martin
> > >
> > > On 25 Feb 2014, at 06:27, Binita Bharati <bi...@gmail.com>
> > wrote:
> > > > Hi all,
> > > >
> > > > I am referring to this e.g:
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
> > > .
> > > >
> > > > What is the consumer group ID being referred here ?
> > > >
> > > > Thanks
> > > > Binita
> > >
> > >
> >
>
Re: Consumer group ID for high level consumer
Posted by Neha Narkhede <ne...@gmail.com>.
Question : How to make the consumer threads receive the older data that
was produced when the consumer threads aren't running ?
As long as you make sure the group.id doesn't change, the consumer will
always
consume data from where it left off during its last run. In other words,
the consumer
can consume data that was produced when it wasn't running, if it is always
started
using the same group.id
Thanks,
Neha
On Mon, Mar 3, 2014 at 8:02 AM, Binita Bharati <bi...@gmail.com>wrote:
> Hi all,
>
> The HighLevelConsumer is working fine. Ref :
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example.
>
> I was expecting the consumer threads to be able to receive older data ( ie
> the data put into the topic when the consumer threads weren't running )
> That was the root of the problem, and once, I started my consumer
> application first, and then, generated fresh data into the topic, the
> consumer threads started receiving the new data.
>
> Question : How to make the consumer threads receive the older data that
> was produced when the consumer threads aren't running ?
>
> Thanks
>
>
> On Wed, Feb 26, 2014 at 6:30 PM, Martin Kleppmann
> <mk...@linkedin.com>wrote:
>
> > Hi Binita,
> >
> > The consumer group (group.id) is a mechanism for sharing the load of
> > consuming a high-volume topic between multiple consumers. If you don't
> set
> > a group ID, each consumer consumes all the partitions of a topic. If you
> > set several consumers to the same group ID, the partitions of the topic
> > you're consuming will be shared amongst the consumers in that group, so
> > that each message is delivered to only one of the consumers in the group.
> >
> > You can set the group ID to be whatever you like.
> >
> > Hope that helps,
> > Martin
> >
> > On 25 Feb 2014, at 06:27, Binita Bharati <bi...@gmail.com>
> wrote:
> > > Hi all,
> > >
> > > I am referring to this e.g:
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
> > .
> > >
> > > What is the consumer group ID being referred here ?
> > >
> > > Thanks
> > > Binita
> >
> >
>
Re: Consumer group ID for high level consumer
Posted by Binita Bharati <bi...@gmail.com>.
Hi all,
The HighLevelConsumer is working fine. Ref :
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example.
I was expecting the consumer threads to be able to receive older data ( ie
the data put into the topic when the consumer threads weren't running )
That was the root of the problem, and once, I started my consumer
application first, and then, generated fresh data into the topic, the
consumer threads started receiving the new data.
Question : How to make the consumer threads receive the older data that
was produced when the consumer threads aren't running ?
Thanks
On Wed, Feb 26, 2014 at 6:30 PM, Martin Kleppmann
<mk...@linkedin.com>wrote:
> Hi Binita,
>
> The consumer group (group.id) is a mechanism for sharing the load of
> consuming a high-volume topic between multiple consumers. If you don't set
> a group ID, each consumer consumes all the partitions of a topic. If you
> set several consumers to the same group ID, the partitions of the topic
> you're consuming will be shared amongst the consumers in that group, so
> that each message is delivered to only one of the consumers in the group.
>
> You can set the group ID to be whatever you like.
>
> Hope that helps,
> Martin
>
> On 25 Feb 2014, at 06:27, Binita Bharati <bi...@gmail.com> wrote:
> > Hi all,
> >
> > I am referring to this e.g:
> > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
> .
> >
> > What is the consumer group ID being referred here ?
> >
> > Thanks
> > Binita
>
>
Re: Consumer group ID for high level consumer
Posted by Martin Kleppmann <mk...@linkedin.com>.
Hi Binita,
The consumer group (group.id) is a mechanism for sharing the load of consuming a high-volume topic between multiple consumers. If you don't set a group ID, each consumer consumes all the partitions of a topic. If you set several consumers to the same group ID, the partitions of the topic you're consuming will be shared amongst the consumers in that group, so that each message is delivered to only one of the consumers in the group.
You can set the group ID to be whatever you like.
Hope that helps,
Martin
On 25 Feb 2014, at 06:27, Binita Bharati <bi...@gmail.com> wrote:
> Hi all,
>
> I am referring to this e.g:
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example.
>
> What is the consumer group ID being referred here ?
>
> Thanks
> Binita