You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by svante karlsson <sa...@csi.se> on 2014/12/11 20:59:18 UTC
How do I create a consumer group
We're using 0.82 beta and a homegrown c++ async library based on boost
asio that has support for the offset api.
(apikeys OffsetCommitRequest = 8, OffsetFetchRequest = 9,
ConsumerMetadataRequest = 10)
If we use a java client and commit an offset then the consumer group shows
up in the response from ConsumerMetadataRequest. However I cant figure out
how to create a new one using the Kafka API.
Also, my __consumer_offsets topic shows up with a replication factor of 1.
Is that changeable?
thanks,
svante
Re: How do I create a consumer group
Posted by svante karlsson <sa...@csi.se>.
>Yes - see the offsets.topic.num.partitions and
>offsets.topic.replication.factor broker configs.
Joel, that exactly what I was looking for. I'll look into that and the
source for OffsetsMessageFormatter later today!
thanks
svante
>
Re: How do I create a consumer group
Posted by Joel Koshy <jj...@gmail.com>.
> However it turns out to be difficult to know the existing consumer group
> strings. Is the message format in __consumer_offsets "public"/stable in any
> way or is there a better way of listing the existing group names?
Yes it is "stable" but not described in the user documentation since
it is an internal implementation detail. You can take a look at the
OffsetManager code and specifically the OffsetsMessageFormatter class.
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/OffsetManager.scala
Joel
>
>
> 2014-12-11 20:59 GMT+01:00 svante karlsson <sa...@csi.se>:
> >
> > We're using 0.82 beta and a homegrown c++ async library based on boost
> > asio that has support for the offset api.
> > (apikeys OffsetCommitRequest = 8, OffsetFetchRequest = 9,
> > ConsumerMetadataRequest = 10)
> >
> > If we use a java client and commit an offset then the consumer group shows
> > up in the response from ConsumerMetadataRequest. However I cant figure out
> > how to create a new one using the Kafka API.
> >
> > Also, my __consumer_offsets topic shows up with a replication factor of 1.
> > Is that changeable?
> >
> > thanks,
> > svante
> >
Re: How do I create a consumer group
Posted by svante karlsson <sa...@csi.se>.
What's your use case for knowing all consumer groups? For admin?
Yes, I need to monitor producer and consumer lags from several
interconnected kafka sites.
/svante
2014-12-16 0:34 GMT+01:00 Jun Rao <ju...@confluent.io>:
>
> What's your use case for knowing all consumer groups? For admin?
>
> Thanks,
>
> Jun
>
> On Fri, Dec 12, 2014 at 2:45 AM, svante karlsson <sa...@csi.se> wrote:
>
> > Disregard the creation question - we must have done something wrong
> because
> > now our code is working without obvious changes (on another set of
> > brokers).
> >
> > However it turns out to be difficult to know the existing consumer group
> > strings. Is the message format in __consumer_offsets "public"/stable in
> any
> > way or is there a better way of listing the existing group names?
> >
> > svante
> >
> >
> > 2014-12-11 20:59 GMT+01:00 svante karlsson <sa...@csi.se>:
> > >
> > > We're using 0.82 beta and a homegrown c++ async library based on boost
> > > asio that has support for the offset api.
> > > (apikeys OffsetCommitRequest = 8, OffsetFetchRequest = 9,
> > > ConsumerMetadataRequest = 10)
> > >
> > > If we use a java client and commit an offset then the consumer group
> > shows
> > > up in the response from ConsumerMetadataRequest. However I cant figure
> > out
> > > how to create a new one using the Kafka API.
> > >
> > > Also, my __consumer_offsets topic shows up with a replication factor of
> > 1.
> > > Is that changeable?
> > >
> > > thanks,
> > > svante
> > >
> >
>
Re: How do I create a consumer group
Posted by Jun Rao <ju...@confluent.io>.
What's your use case for knowing all consumer groups? For admin?
Thanks,
Jun
On Fri, Dec 12, 2014 at 2:45 AM, svante karlsson <sa...@csi.se> wrote:
> Disregard the creation question - we must have done something wrong because
> now our code is working without obvious changes (on another set of
> brokers).
>
> However it turns out to be difficult to know the existing consumer group
> strings. Is the message format in __consumer_offsets "public"/stable in any
> way or is there a better way of listing the existing group names?
>
> svante
>
>
> 2014-12-11 20:59 GMT+01:00 svante karlsson <sa...@csi.se>:
> >
> > We're using 0.82 beta and a homegrown c++ async library based on boost
> > asio that has support for the offset api.
> > (apikeys OffsetCommitRequest = 8, OffsetFetchRequest = 9,
> > ConsumerMetadataRequest = 10)
> >
> > If we use a java client and commit an offset then the consumer group
> shows
> > up in the response from ConsumerMetadataRequest. However I cant figure
> out
> > how to create a new one using the Kafka API.
> >
> > Also, my __consumer_offsets topic shows up with a replication factor of
> 1.
> > Is that changeable?
> >
> > thanks,
> > svante
> >
>
Re: How do I create a consumer group
Posted by svante karlsson <sa...@csi.se>.
If I understand KAFKA-1476 it is only a command line tool that gives access
by using ZKUtils not an API to Kafka. We're looking for a Kafka API so I
guess that this functionality is missing.
thanks for the pointer
Svante Karlsson
2014-12-12 19:03 GMT+01:00 Jiangjie Qin <jq...@linkedin.com.invalid>:
>
> KAFKA-1476 is addressing this need, but it¹s not checked in yet. Currently
> maybe you can use zookeeper client to check the zookeeper path.
>
> ‹Jiangjie (Becket) Qin
>
> On 12/12/14, 2:45 AM, "svante karlsson" <sa...@csi.se> wrote:
>
> >Disregard the creation question - we must have done something wrong
> >because
> >now our code is working without obvious changes (on another set of
> >brokers).
> >
> >However it turns out to be difficult to know the existing consumer group
> >strings. Is the message format in __consumer_offsets "public"/stable in
> >any
> >way or is there a better way of listing the existing group names?
> >
> >svante
> >
> >
> >2014-12-11 20:59 GMT+01:00 svante karlsson <sa...@csi.se>:
> >>
> >> We're using 0.82 beta and a homegrown c++ async library based on boost
> >> asio that has support for the offset api.
> >> (apikeys OffsetCommitRequest = 8, OffsetFetchRequest = 9,
> >> ConsumerMetadataRequest = 10)
> >>
> >> If we use a java client and commit an offset then the consumer group
> >>shows
> >> up in the response from ConsumerMetadataRequest. However I cant figure
> >>out
> >> how to create a new one using the Kafka API.
> >>
> >> Also, my __consumer_offsets topic shows up with a replication factor of
> >>1.
> >> Is that changeable?
> >>
> >> thanks,
> >> svante
> >>
>
>
Re: How do I create a consumer group
Posted by Jiangjie Qin <jq...@linkedin.com.INVALID>.
KAFKA-1476 is addressing this need, but it¹s not checked in yet. Currently
maybe you can use zookeeper client to check the zookeeper path.
‹Jiangjie (Becket) Qin
On 12/12/14, 2:45 AM, "svante karlsson" <sa...@csi.se> wrote:
>Disregard the creation question - we must have done something wrong
>because
>now our code is working without obvious changes (on another set of
>brokers).
>
>However it turns out to be difficult to know the existing consumer group
>strings. Is the message format in __consumer_offsets "public"/stable in
>any
>way or is there a better way of listing the existing group names?
>
>svante
>
>
>2014-12-11 20:59 GMT+01:00 svante karlsson <sa...@csi.se>:
>>
>> We're using 0.82 beta and a homegrown c++ async library based on boost
>> asio that has support for the offset api.
>> (apikeys OffsetCommitRequest = 8, OffsetFetchRequest = 9,
>> ConsumerMetadataRequest = 10)
>>
>> If we use a java client and commit an offset then the consumer group
>>shows
>> up in the response from ConsumerMetadataRequest. However I cant figure
>>out
>> how to create a new one using the Kafka API.
>>
>> Also, my __consumer_offsets topic shows up with a replication factor of
>>1.
>> Is that changeable?
>>
>> thanks,
>> svante
>>
Re: How do I create a consumer group
Posted by svante karlsson <sa...@csi.se>.
Disregard the creation question - we must have done something wrong because
now our code is working without obvious changes (on another set of
brokers).
However it turns out to be difficult to know the existing consumer group
strings. Is the message format in __consumer_offsets "public"/stable in any
way or is there a better way of listing the existing group names?
svante
2014-12-11 20:59 GMT+01:00 svante karlsson <sa...@csi.se>:
>
> We're using 0.82 beta and a homegrown c++ async library based on boost
> asio that has support for the offset api.
> (apikeys OffsetCommitRequest = 8, OffsetFetchRequest = 9,
> ConsumerMetadataRequest = 10)
>
> If we use a java client and commit an offset then the consumer group shows
> up in the response from ConsumerMetadataRequest. However I cant figure out
> how to create a new one using the Kafka API.
>
> Also, my __consumer_offsets topic shows up with a replication factor of 1.
> Is that changeable?
>
> thanks,
> svante
>
Re: How do I create a consumer group
Posted by Joel Koshy <jj...@gmail.com>.
>
> Also, my __consumer_offsets topic shows up with a replication factor of 1.
> Is that changeable?
Yes - see the offsets.topic.num.partitions and
offsets.topic.replication.factor broker configs.
--
Joel