You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Jerry George <je...@gmail.com> on 2017/05/26 13:54:44 UTC
Trouble with querying offsets when using new consumer groups API
Hi
I had question about the new consumer APIs.
I am having trouble retrieving the offsets once the consumers are
*disconnected* when using new consumer v2 API. Following is what I am
trying to do,
*bin/kafka-consumer-groups.sh -new-consumer --bootstrap-server kafka:9092
--group group --describe*
If I query this when the consumers are connected, there is no problem.
However, once the consumers are disconnected it says there is no such
group, though the offsets are retained in __consumer_offsets.
The offset retention policy is default; i.e. 1440 minutes, I believe.
Once the consumers are reconnected, I am able to query the offsets once
again.
Could anyone here please help me understand why this is?
Kafka: 0.10.1
Consumer Library: sarama golang library
Regards,
Jerry
Re: Trouble with querying offsets when using new consumer groups API
Posted by Hans Jespersen <ha...@confluent.io>.
It is definitely expected behavior that the new consumer version of kafka-consumer-groups.sh —describe only returns metadata for ‘active’ members. It will print an error message if the consumer group you provide has no active members.
https://github.com/confluentinc/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala#L88 <https://github.com/confluentinc/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala#L88>
As to the reason for this I am not certain. My guess is that since offsets are now stored in a partitioned kafka topic (i.e. _offsets) and uses a key that includes the topic+partition+consumerID then it was deemed too problematic or resource intensive to support queries for inactive consumers because the tool would have to read every message in every partition of the _offset topic in order to find a complete list of the offsets for a given inactive consumer.
-hans
> On May 30, 2017, at 8:09 AM, Jerry George <je...@gmail.com> wrote:
>
> Hi Abhimanyu,
>
> No, actually waiting for someone with operational experience to reply on
> the list. Thank you for bumping the question though :)
>
> If anyone in the list has experience increasing the retention or if this is
> expected behaviour, could kindly suggest an alternative?
>
>
> Regards,
> Jerry
>
> On Sat, May 27, 2017 at 8:18 AM, Abhimanyu Nagrath <
> abhimanyunagrath@gmail.com> wrote:
>
>> Hi Jerry,
>>
>> I am also facing the same issue. Did you found the solution?
>>
>> Regards,
>> Abhimanyu
>>
>> On Fri, May 26, 2017 at 7:24 PM, Jerry George <je...@gmail.com> wrote:
>>
>>> Hi
>>>
>>> I had question about the new consumer APIs.
>>>
>>> I am having trouble retrieving the offsets once the consumers are
>>> *disconnected* when using new consumer v2 API. Following is what I am
>>> trying to do,
>>>
>>> *bin/kafka-consumer-groups.sh -new-consumer --bootstrap-server kafka:9092
>>> --group group --describe*
>>>
>>> If I query this when the consumers are connected, there is no problem.
>>> However, once the consumers are disconnected it says there is no such
>>> group, though the offsets are retained in __consumer_offsets.
>>>
>>> The offset retention policy is default; i.e. 1440 minutes, I believe.
>>>
>>> Once the consumers are reconnected, I am able to query the offsets once
>>> again.
>>>
>>> Could anyone here please help me understand why this is?
>>>
>>> Kafka: 0.10.1
>>> Consumer Library: sarama golang library
>>>
>>> Regards,
>>> Jerry
>>>
>>
Re: Trouble with querying offsets when using new consumer groups API
Posted by Jerry George <je...@gmail.com>.
Hi Abhimanyu,
No, actually waiting for someone with operational experience to reply on
the list. Thank you for bumping the question though :)
If anyone in the list has experience increasing the retention or if this is
expected behaviour, could kindly suggest an alternative?
Regards,
Jerry
On Sat, May 27, 2017 at 8:18 AM, Abhimanyu Nagrath <
abhimanyunagrath@gmail.com> wrote:
> Hi Jerry,
>
> I am also facing the same issue. Did you found the solution?
>
> Regards,
> Abhimanyu
>
> On Fri, May 26, 2017 at 7:24 PM, Jerry George <je...@gmail.com> wrote:
>
> > Hi
> >
> > I had question about the new consumer APIs.
> >
> > I am having trouble retrieving the offsets once the consumers are
> > *disconnected* when using new consumer v2 API. Following is what I am
> > trying to do,
> >
> > *bin/kafka-consumer-groups.sh -new-consumer --bootstrap-server kafka:9092
> > --group group --describe*
> >
> > If I query this when the consumers are connected, there is no problem.
> > However, once the consumers are disconnected it says there is no such
> > group, though the offsets are retained in __consumer_offsets.
> >
> > The offset retention policy is default; i.e. 1440 minutes, I believe.
> >
> > Once the consumers are reconnected, I am able to query the offsets once
> > again.
> >
> > Could anyone here please help me understand why this is?
> >
> > Kafka: 0.10.1
> > Consumer Library: sarama golang library
> >
> > Regards,
> > Jerry
> >
>
Re: Trouble with querying offsets when using new consumer groups API
Posted by Abhimanyu Nagrath <ab...@gmail.com>.
Hi Jerry,
I am also facing the same issue. Did you found the solution?
Regards,
Abhimanyu
On Fri, May 26, 2017 at 7:24 PM, Jerry George <je...@gmail.com> wrote:
> Hi
>
> I had question about the new consumer APIs.
>
> I am having trouble retrieving the offsets once the consumers are
> *disconnected* when using new consumer v2 API. Following is what I am
> trying to do,
>
> *bin/kafka-consumer-groups.sh -new-consumer --bootstrap-server kafka:9092
> --group group --describe*
>
> If I query this when the consumers are connected, there is no problem.
> However, once the consumers are disconnected it says there is no such
> group, though the offsets are retained in __consumer_offsets.
>
> The offset retention policy is default; i.e. 1440 minutes, I believe.
>
> Once the consumers are reconnected, I am able to query the offsets once
> again.
>
> Could anyone here please help me understand why this is?
>
> Kafka: 0.10.1
> Consumer Library: sarama golang library
>
> Regards,
> Jerry
>
Re: Trouble with querying offsets when using new consumer groups API
Posted by Hans Jespersen <ha...@confluent.io>.
I can confirm that in 0.10.2.1 I get offset information for disconnected consumers. The note in the output is a bit misleading because it also works with non-Java clients as long as they implement the new consumer. For example below is what I get when using the blizzard/node-rdkafka client which wraps librdkafka.
[bin] $ ./kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group node-red-rdkafka-groupid
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
mytopic 0 4 4 0 -0b713303-e69e-47d9-baf7-70df49ff48a6 /10.20.0.1
-hans
> On May 30, 2017, at 10:57 AM, Jerry George <je...@gmail.com> wrote:
>
> Thank you Hans and Vahid.
>
> That was definitely of great help. Much appreciated!
>
> Regards,
> Jerry
>
> On Tue, May 30, 2017 at 1:53 PM, Vahid S Hashemian <
> vahidhashemian@us.ibm.com> wrote:
>
>> Hi Jerry,
>>
>> The behavior you are expecting is implemented in 0.10.2 through KIP-88 (
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> 88%3A+OffsetFetch+Protocol+Update
>> ) and KAFKA-3853 (https://issues.apache.org/jira/browse/KAFKA-3853).
>> Starting from this release when you query a consumer group (new consumer
>> only) you'll see the stored offset corresponding to a topic partition even
>> if there is no active consumer consuming from it.
>>
>> I hope this helps.
>> --Vahid
>>
>>
>>
>>
>> From: Jerry George <je...@gmail.com>
>> To: users@kafka.apache.org
>> Date: 05/26/2017 06:55 AM
>> Subject: Trouble with querying offsets when using new consumer
>> groups API
>>
>>
>>
>> Hi
>>
>> I had question about the new consumer APIs.
>>
>> I am having trouble retrieving the offsets once the consumers are
>> *disconnected* when using new consumer v2 API. Following is what I am
>> trying to do,
>>
>> *bin/kafka-consumer-groups.sh -new-consumer --bootstrap-server kafka:9092
>> --group group --describe*
>>
>> If I query this when the consumers are connected, there is no problem.
>> However, once the consumers are disconnected it says there is no such
>> group, though the offsets are retained in __consumer_offsets.
>>
>> The offset retention policy is default; i.e. 1440 minutes, I believe.
>>
>> Once the consumers are reconnected, I am able to query the offsets once
>> again.
>>
>> Could anyone here please help me understand why this is?
>>
>> Kafka: 0.10.1
>> Consumer Library: sarama golang library
>>
>> Regards,
>> Jerry
>>
>>
>>
>>
>>
Re: Trouble with querying offsets when using new consumer groups API
Posted by Jerry George <je...@gmail.com>.
Thank you Hans and Vahid.
That was definitely of great help. Much appreciated!
Regards,
Jerry
On Tue, May 30, 2017 at 1:53 PM, Vahid S Hashemian <
vahidhashemian@us.ibm.com> wrote:
> Hi Jerry,
>
> The behavior you are expecting is implemented in 0.10.2 through KIP-88 (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 88%3A+OffsetFetch+Protocol+Update
> ) and KAFKA-3853 (https://issues.apache.org/jira/browse/KAFKA-3853).
> Starting from this release when you query a consumer group (new consumer
> only) you'll see the stored offset corresponding to a topic partition even
> if there is no active consumer consuming from it.
>
> I hope this helps.
> --Vahid
>
>
>
>
> From: Jerry George <je...@gmail.com>
> To: users@kafka.apache.org
> Date: 05/26/2017 06:55 AM
> Subject: Trouble with querying offsets when using new consumer
> groups API
>
>
>
> Hi
>
> I had question about the new consumer APIs.
>
> I am having trouble retrieving the offsets once the consumers are
> *disconnected* when using new consumer v2 API. Following is what I am
> trying to do,
>
> *bin/kafka-consumer-groups.sh -new-consumer --bootstrap-server kafka:9092
> --group group --describe*
>
> If I query this when the consumers are connected, there is no problem.
> However, once the consumers are disconnected it says there is no such
> group, though the offsets are retained in __consumer_offsets.
>
> The offset retention policy is default; i.e. 1440 minutes, I believe.
>
> Once the consumers are reconnected, I am able to query the offsets once
> again.
>
> Could anyone here please help me understand why this is?
>
> Kafka: 0.10.1
> Consumer Library: sarama golang library
>
> Regards,
> Jerry
>
>
>
>
>
Re: Trouble with querying offsets when using new consumer groups API
Posted by Vahid S Hashemian <va...@us.ibm.com>.
Hi Jerry,
The behavior you are expecting is implemented in 0.10.2 through KIP-88 (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-88%3A+OffsetFetch+Protocol+Update
) and KAFKA-3853 (https://issues.apache.org/jira/browse/KAFKA-3853).
Starting from this release when you query a consumer group (new consumer
only) you'll see the stored offset corresponding to a topic partition even
if there is no active consumer consuming from it.
I hope this helps.
--Vahid
From: Jerry George <je...@gmail.com>
To: users@kafka.apache.org
Date: 05/26/2017 06:55 AM
Subject: Trouble with querying offsets when using new consumer
groups API
Hi
I had question about the new consumer APIs.
I am having trouble retrieving the offsets once the consumers are
*disconnected* when using new consumer v2 API. Following is what I am
trying to do,
*bin/kafka-consumer-groups.sh -new-consumer --bootstrap-server kafka:9092
--group group --describe*
If I query this when the consumers are connected, there is no problem.
However, once the consumers are disconnected it says there is no such
group, though the offsets are retained in __consumer_offsets.
The offset retention policy is default; i.e. 1440 minutes, I believe.
Once the consumers are reconnected, I am able to query the offsets once
again.
Could anyone here please help me understand why this is?
Kafka: 0.10.1
Consumer Library: sarama golang library
Regards,
Jerry