You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Wouter Bancken <wo...@aca-it.be> on 2018/02/04 19:51:26 UTC
Re: Kafka Consumer Offsets unavailable during rebalancing
Can anyone clarify if this is a bug in Kafka or the expected behavior?
Best regards,
Wouter
On 30 January 2018 at 21:04, Wouter Bancken <wo...@aca-it.be>
wrote:
> Hi,
>
> I'm trying to write an external tool to monitor consumer lag on Apache
> Kafka.
>
> For this purpose, I'm using the kafka-consumer-groups tool to fetch the
> consumer offsets.
>
> When using this tool, partition assignments seem to be unavailable
> temporarily during the creation of a new topic even if the consumer group
> has no subscription on this new topic. This seems to match the
> documentation
> <https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Client-side+Assignment+Proposal>
> saying *"Topic metadata changes which have no impact on subscriptions
> cause resync"*.
>
> However, when this occurs I'd expect the state of the consumer to be
> "PreparingRebalance" or "AwaitingSync" but it is simply "Stable".
>
> Is this a bug in the tooling or is there a different way to obtain the
> correct offsets for a consumer group during a rebalance?
>
> I'm using Kafka 10.2.1 but I haven't found any related issues in recent
> changelogs.
> Best regards,
> Wouter
>
Re: Kafka Consumer Offsets unavailable during rebalancing
Posted by Wouter Bancken <wo...@aca-it.be>.
I logged https://issues.apache.org/jira/browse/KAFKA-6557
Best regards,
Wouter
On 13 February 2018 at 12:50, Jan Filipiak <Ja...@trivago.com> wrote:
> I would encourage you todo so.
> I also think its not reasonable behavior
>
> On 13.02.2018 11:28, Wouter Bancken wrote:
>
>> We have upgraded our Kafka version as an attempt to solve this issue.
>> However, the issue is still present in Kafka 1.0.0.
>>
>> Can I log a bug for this in JIRA?
>>
>> Wouter
>>
>> On 5 February 2018 at 09:22, Wouter Bancken <wo...@aca-it.be>
>> wrote:
>>
>> The consumers in consumer group 'X' do not have a regex subscription
>>> matching the newly created topic 'C'. They simply subscribe with
>>> the subscribe(java.util.Collection<java.lang.String> topics) method on
>>> topics 'A' and 'B'.
>>>
>>> Shouldn't the consumer group have a different state from "Stable" during
>>> a
>>> rebalancing regardless of the cause? How else can we determine the
>>> consumer
>>> lag of the group during the rebalancing?
>>>
>>> Best regards,
>>> Wouter
>>>
>>> Have a look at our brand NEW job website: jobs.aca-it.be !
>>>
>>>
>>> *ACA IT-Solutions NV*
>>> *HQ:* Herkenrodesingel 8B 2.01 | 3500 Hasselt
>>> T +32(0)11 26 50 10 | F +32(0)11 26 50 11
>>> www.aca-it.be | Twitter <https://twitter.com/dorien_jorissen> | Facebook
>>> <http://www.facebook.com/pages/ACA-IT-Solutions/278520212202909> |
>>> Linkedin <http://www.linkedin.com/company/6524>
>>>
>>>
>>> On 5 February 2018 at 00:13, Hans Jespersen <ha...@confluent.io> wrote:
>>>
>>> Do the consumers in consumer group ‘X’ have a regex subscription that
>>>> matches the newly created topic ‘C’?
>>>>
>>>> If they do then they will only discover this new topic once their ‘
>>>> metadata.max.age.ms’ metadata refresh interval has passed, which
>>>> defaults to 5 minutes.
>>>>
>>>> metadata.max.age.ms The period of time in milliseconds after which
>>>> we force a refresh of metadata even if we haven't seen any partition
>>>> leadership changes to proactively discover any new brokers or partitions
>>>> -hans
>>>>
>>>>
>>>> On Feb 4, 2018, at 2:16 PM, Wouter Bancken <wo...@aca-it.be>
>>>>>
>>>> wrote:
>>>>
>>>>> Hi Hans,
>>>>>
>>>>> Thanks for the response!
>>>>>
>>>>> However, I get this result for all topics, not just for the newly
>>>>>
>>>> created
>>>>
>>>>> topic.
>>>>>
>>>>> Situation sketch:
>>>>> 1. I have a consumer group 'X' subscribed to topics 'A' and 'B' with
>>>>> partition assignments and lag information. Consumer group 'X' is
>>>>>
>>>> "Stable".
>>>>
>>>>> 2a. Topic 'C' is (being) created.
>>>>> 2b. During this creation, I do not have a partition assignment for
>>>>>
>>>> consumer
>>>>
>>>>> group 'X' for topics 'A' and 'B' but the consumer group is still
>>>>>
>>>> "Stable".
>>>>
>>>>> 3. A second later: I have a partition assignment for consumer group 'X'
>>>>>
>>>> for
>>>>
>>>>> topics 'A' and 'B' again and the consumer group is still "Stable".
>>>>>
>>>>> I expected the state of consumer group 'X' during step 2b to be
>>>>> "PreparingRebalance" or "AwaitingSync".
>>>>>
>>>>> Best regards,
>>>>> Wouter
>>>>>
>>>>> On 4 February 2018 at 21:25, Hans Jespersen <ha...@confluent.io> wrote:
>>>>>>
>>>>>> I believe this is expected behavior.
>>>>>>
>>>>>> If there are no subscriptions to a new topic, and therefor no
>>>>>> partition
>>>>>> assignments, and definitely no committed offsets, then lag is an
>>>>>>
>>>>> undefined
>>>>
>>>>> concept. When the consumers subscribe to this new topic they may chose
>>>>>>
>>>>> to
>>>>
>>>>> start at the beginning or end of the commit log so the lag cannot be
>>>>>> predicted in advance.
>>>>>>
>>>>>> -hans
>>>>>>
>>>>>> On Feb 4, 2018, at 11:51 AM, Wouter Bancken <wouter.bancken@aca-it.be
>>>>>>>
>>>>>> wrote:
>>>>>>
>>>>>>> Can anyone clarify if this is a bug in Kafka or the expected
>>>>>>> behavior?
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Wouter
>>>>>>>
>>>>>>>
>>>>>>> On 30 January 2018 at 21:04, Wouter Bancken <
>>>>>>> wouter.bancken@aca-it.be
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I'm trying to write an external tool to monitor consumer lag on
>>>>>>>>
>>>>>>> Apache
>>>>
>>>>> Kafka.
>>>>>>>>
>>>>>>>> For this purpose, I'm using the kafka-consumer-groups tool to fetch
>>>>>>>>
>>>>>>> the
>>>>
>>>>> consumer offsets.
>>>>>>>>
>>>>>>>> When using this tool, partition assignments seem to be unavailable
>>>>>>>> temporarily during the creation of a new topic even if the consumer
>>>>>>>>
>>>>>>> group
>>>>>>
>>>>>>> has no subscription on this new topic. This seems to match the
>>>>>>>> documentation
>>>>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/
>>>>>>>>
>>>>>>> Kafka+Client-side+Assignment+Proposal>
>>>>>>
>>>>>>> saying *"Topic metadata changes which have no impact on subscriptions
>>>>>>>> cause resync"*.
>>>>>>>>
>>>>>>>> However, when this occurs I'd expect the state of the consumer to be
>>>>>>>> "PreparingRebalance" or "AwaitingSync" but it is simply "Stable".
>>>>>>>>
>>>>>>>> Is this a bug in the tooling or is there a different way to obtain
>>>>>>>>
>>>>>>> the
>>>>
>>>>> correct offsets for a consumer group during a rebalance?
>>>>>>>>
>>>>>>>> I'm using Kafka 10.2.1 but I haven't found any related issues in
>>>>>>>>
>>>>>>> recent
>>>>
>>>>> changelogs.
>>>>>>>> Best regards,
>>>>>>>> Wouter
>>>>>>>>
>>>>>>>>
>>>
>
Re: Kafka Consumer Offsets unavailable during rebalancing
Posted by Jan Filipiak <Ja...@trivago.com>.
I would encourage you todo so.
I also think its not reasonable behavior
On 13.02.2018 11:28, Wouter Bancken wrote:
> We have upgraded our Kafka version as an attempt to solve this issue.
> However, the issue is still present in Kafka 1.0.0.
>
> Can I log a bug for this in JIRA?
>
> Wouter
>
> On 5 February 2018 at 09:22, Wouter Bancken <wo...@aca-it.be>
> wrote:
>
>> The consumers in consumer group 'X' do not have a regex subscription
>> matching the newly created topic 'C'. They simply subscribe with
>> the subscribe(java.util.Collection<java.lang.String> topics) method on
>> topics 'A' and 'B'.
>>
>> Shouldn't the consumer group have a different state from "Stable" during a
>> rebalancing regardless of the cause? How else can we determine the consumer
>> lag of the group during the rebalancing?
>>
>> Best regards,
>> Wouter
>>
>> Have a look at our brand NEW job website: jobs.aca-it.be !
>>
>>
>> *ACA IT-Solutions NV*
>> *HQ:* Herkenrodesingel 8B 2.01 | 3500 Hasselt
>> T +32(0)11 26 50 10 | F +32(0)11 26 50 11
>> www.aca-it.be | Twitter <https://twitter.com/dorien_jorissen> | Facebook
>> <http://www.facebook.com/pages/ACA-IT-Solutions/278520212202909> |
>> Linkedin <http://www.linkedin.com/company/6524>
>>
>> On 5 February 2018 at 00:13, Hans Jespersen <ha...@confluent.io> wrote:
>>
>>> Do the consumers in consumer group ‘X’ have a regex subscription that
>>> matches the newly created topic ‘C’?
>>>
>>> If they do then they will only discover this new topic once their ‘
>>> metadata.max.age.ms’ metadata refresh interval has passed, which
>>> defaults to 5 minutes.
>>>
>>> metadata.max.age.ms The period of time in milliseconds after which
>>> we force a refresh of metadata even if we haven't seen any partition
>>> leadership changes to proactively discover any new brokers or partitions
>>> -hans
>>>
>>>
>>>> On Feb 4, 2018, at 2:16 PM, Wouter Bancken <wo...@aca-it.be>
>>> wrote:
>>>> Hi Hans,
>>>>
>>>> Thanks for the response!
>>>>
>>>> However, I get this result for all topics, not just for the newly
>>> created
>>>> topic.
>>>>
>>>> Situation sketch:
>>>> 1. I have a consumer group 'X' subscribed to topics 'A' and 'B' with
>>>> partition assignments and lag information. Consumer group 'X' is
>>> "Stable".
>>>> 2a. Topic 'C' is (being) created.
>>>> 2b. During this creation, I do not have a partition assignment for
>>> consumer
>>>> group 'X' for topics 'A' and 'B' but the consumer group is still
>>> "Stable".
>>>> 3. A second later: I have a partition assignment for consumer group 'X'
>>> for
>>>> topics 'A' and 'B' again and the consumer group is still "Stable".
>>>>
>>>> I expected the state of consumer group 'X' during step 2b to be
>>>> "PreparingRebalance" or "AwaitingSync".
>>>>
>>>> Best regards,
>>>> Wouter
>>>>
>>>>> On 4 February 2018 at 21:25, Hans Jespersen <ha...@confluent.io> wrote:
>>>>>
>>>>> I believe this is expected behavior.
>>>>>
>>>>> If there are no subscriptions to a new topic, and therefor no partition
>>>>> assignments, and definitely no committed offsets, then lag is an
>>> undefined
>>>>> concept. When the consumers subscribe to this new topic they may chose
>>> to
>>>>> start at the beginning or end of the commit log so the lag cannot be
>>>>> predicted in advance.
>>>>>
>>>>> -hans
>>>>>
>>>>>> On Feb 4, 2018, at 11:51 AM, Wouter Bancken <wouter.bancken@aca-it.be
>>>>> wrote:
>>>>>> Can anyone clarify if this is a bug in Kafka or the expected behavior?
>>>>>>
>>>>>> Best regards,
>>>>>> Wouter
>>>>>>
>>>>>>
>>>>>> On 30 January 2018 at 21:04, Wouter Bancken <wouter.bancken@aca-it.be
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I'm trying to write an external tool to monitor consumer lag on
>>> Apache
>>>>>>> Kafka.
>>>>>>>
>>>>>>> For this purpose, I'm using the kafka-consumer-groups tool to fetch
>>> the
>>>>>>> consumer offsets.
>>>>>>>
>>>>>>> When using this tool, partition assignments seem to be unavailable
>>>>>>> temporarily during the creation of a new topic even if the consumer
>>>>> group
>>>>>>> has no subscription on this new topic. This seems to match the
>>>>>>> documentation
>>>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/
>>>>> Kafka+Client-side+Assignment+Proposal>
>>>>>>> saying *"Topic metadata changes which have no impact on subscriptions
>>>>>>> cause resync"*.
>>>>>>>
>>>>>>> However, when this occurs I'd expect the state of the consumer to be
>>>>>>> "PreparingRebalance" or "AwaitingSync" but it is simply "Stable".
>>>>>>>
>>>>>>> Is this a bug in the tooling or is there a different way to obtain
>>> the
>>>>>>> correct offsets for a consumer group during a rebalance?
>>>>>>>
>>>>>>> I'm using Kafka 10.2.1 but I haven't found any related issues in
>>> recent
>>>>>>> changelogs.
>>>>>>> Best regards,
>>>>>>> Wouter
>>>>>>>
>>
Re: Kafka Consumer Offsets unavailable during rebalancing
Posted by Wouter Bancken <wo...@aca-it.be>.
We have upgraded our Kafka version as an attempt to solve this issue.
However, the issue is still present in Kafka 1.0.0.
Can I log a bug for this in JIRA?
Wouter
On 5 February 2018 at 09:22, Wouter Bancken <wo...@aca-it.be>
wrote:
> The consumers in consumer group 'X' do not have a regex subscription
> matching the newly created topic 'C'. They simply subscribe with
> the subscribe(java.util.Collection<java.lang.String> topics) method on
> topics 'A' and 'B'.
>
> Shouldn't the consumer group have a different state from "Stable" during a
> rebalancing regardless of the cause? How else can we determine the consumer
> lag of the group during the rebalancing?
>
> Best regards,
> Wouter
>
> Have a look at our brand NEW job website: jobs.aca-it.be !
>
>
> *ACA IT-Solutions NV*
> *HQ:* Herkenrodesingel 8B 2.01 | 3500 Hasselt
> T +32(0)11 26 50 10 | F +32(0)11 26 50 11
> www.aca-it.be | Twitter <https://twitter.com/dorien_jorissen> | Facebook
> <http://www.facebook.com/pages/ACA-IT-Solutions/278520212202909> |
> Linkedin <http://www.linkedin.com/company/6524>
>
> On 5 February 2018 at 00:13, Hans Jespersen <ha...@confluent.io> wrote:
>
>> Do the consumers in consumer group ‘X’ have a regex subscription that
>> matches the newly created topic ‘C’?
>>
>> If they do then they will only discover this new topic once their ‘
>> metadata.max.age.ms’ metadata refresh interval has passed, which
>> defaults to 5 minutes.
>>
>> metadata.max.age.ms The period of time in milliseconds after which
>> we force a refresh of metadata even if we haven't seen any partition
>> leadership changes to proactively discover any new brokers or partitions
>> -hans
>>
>>
>> > On Feb 4, 2018, at 2:16 PM, Wouter Bancken <wo...@aca-it.be>
>> wrote:
>> >
>> > Hi Hans,
>> >
>> > Thanks for the response!
>> >
>> > However, I get this result for all topics, not just for the newly
>> created
>> > topic.
>> >
>> > Situation sketch:
>> > 1. I have a consumer group 'X' subscribed to topics 'A' and 'B' with
>> > partition assignments and lag information. Consumer group 'X' is
>> "Stable".
>> > 2a. Topic 'C' is (being) created.
>> > 2b. During this creation, I do not have a partition assignment for
>> consumer
>> > group 'X' for topics 'A' and 'B' but the consumer group is still
>> "Stable".
>> > 3. A second later: I have a partition assignment for consumer group 'X'
>> for
>> > topics 'A' and 'B' again and the consumer group is still "Stable".
>> >
>> > I expected the state of consumer group 'X' during step 2b to be
>> > "PreparingRebalance" or "AwaitingSync".
>> >
>> > Best regards,
>> > Wouter
>> >
>> >> On 4 February 2018 at 21:25, Hans Jespersen <ha...@confluent.io> wrote:
>> >>
>> >> I believe this is expected behavior.
>> >>
>> >> If there are no subscriptions to a new topic, and therefor no partition
>> >> assignments, and definitely no committed offsets, then lag is an
>> undefined
>> >> concept. When the consumers subscribe to this new topic they may chose
>> to
>> >> start at the beginning or end of the commit log so the lag cannot be
>> >> predicted in advance.
>> >>
>> >> -hans
>> >>
>> >>> On Feb 4, 2018, at 11:51 AM, Wouter Bancken <wouter.bancken@aca-it.be
>> >
>> >> wrote:
>> >>>
>> >>> Can anyone clarify if this is a bug in Kafka or the expected behavior?
>> >>>
>> >>> Best regards,
>> >>> Wouter
>> >>>
>> >>>
>> >>> On 30 January 2018 at 21:04, Wouter Bancken <wouter.bancken@aca-it.be
>> >
>> >>> wrote:
>> >>>
>> >>>> Hi,
>> >>>>
>> >>>> I'm trying to write an external tool to monitor consumer lag on
>> Apache
>> >>>> Kafka.
>> >>>>
>> >>>> For this purpose, I'm using the kafka-consumer-groups tool to fetch
>> the
>> >>>> consumer offsets.
>> >>>>
>> >>>> When using this tool, partition assignments seem to be unavailable
>> >>>> temporarily during the creation of a new topic even if the consumer
>> >> group
>> >>>> has no subscription on this new topic. This seems to match the
>> >>>> documentation
>> >>>> <https://cwiki.apache.org/confluence/display/KAFKA/
>> >> Kafka+Client-side+Assignment+Proposal>
>> >>>> saying *"Topic metadata changes which have no impact on subscriptions
>> >>>> cause resync"*.
>> >>>>
>> >>>> However, when this occurs I'd expect the state of the consumer to be
>> >>>> "PreparingRebalance" or "AwaitingSync" but it is simply "Stable".
>> >>>>
>> >>>> Is this a bug in the tooling or is there a different way to obtain
>> the
>> >>>> correct offsets for a consumer group during a rebalance?
>> >>>>
>> >>>> I'm using Kafka 10.2.1 but I haven't found any related issues in
>> recent
>> >>>> changelogs.
>> >>>> Best regards,
>> >>>> Wouter
>> >>>>
>> >>
>>
>
>
Re: Kafka Consumer Offsets unavailable during rebalancing
Posted by Wouter Bancken <wo...@aca-it.be>.
The consumers in consumer group 'X' do not have a regex subscription
matching the newly created topic 'C'. They simply subscribe with
the subscribe(java.util.Collection<java.lang.String> topics) method on
topics 'A' and 'B'.
Shouldn't the consumer group have a different state from "Stable" during a
rebalancing regardless of the cause? How else can we determine the consumer
lag of the group during the rebalancing?
Best regards,
Wouter
Have a look at our brand NEW job website: jobs.aca-it.be !
*ACA IT-Solutions NV*
*HQ:* Herkenrodesingel 8B 2.01 | 3500 Hasselt
T +32(0)11 26 50 10 | F +32(0)11 26 50 11
www.aca-it.be | Twitter <https://twitter.com/dorien_jorissen> | Facebook
<http://www.facebook.com/pages/ACA-IT-Solutions/278520212202909> | Linkedin
<http://www.linkedin.com/company/6524>
On 5 February 2018 at 00:13, Hans Jespersen <ha...@confluent.io> wrote:
> Do the consumers in consumer group ‘X’ have a regex subscription that
> matches the newly created topic ‘C’?
>
> If they do then they will only discover this new topic once their ‘
> metadata.max.age.ms’ metadata refresh interval has passed, which
> defaults to 5 minutes.
>
> metadata.max.age.ms The period of time in milliseconds after which we
> force a refresh of metadata even if we haven't seen any partition
> leadership changes to proactively discover any new brokers or partitions
> -hans
>
>
> > On Feb 4, 2018, at 2:16 PM, Wouter Bancken <wo...@aca-it.be>
> wrote:
> >
> > Hi Hans,
> >
> > Thanks for the response!
> >
> > However, I get this result for all topics, not just for the newly created
> > topic.
> >
> > Situation sketch:
> > 1. I have a consumer group 'X' subscribed to topics 'A' and 'B' with
> > partition assignments and lag information. Consumer group 'X' is
> "Stable".
> > 2a. Topic 'C' is (being) created.
> > 2b. During this creation, I do not have a partition assignment for
> consumer
> > group 'X' for topics 'A' and 'B' but the consumer group is still
> "Stable".
> > 3. A second later: I have a partition assignment for consumer group 'X'
> for
> > topics 'A' and 'B' again and the consumer group is still "Stable".
> >
> > I expected the state of consumer group 'X' during step 2b to be
> > "PreparingRebalance" or "AwaitingSync".
> >
> > Best regards,
> > Wouter
> >
> >> On 4 February 2018 at 21:25, Hans Jespersen <ha...@confluent.io> wrote:
> >>
> >> I believe this is expected behavior.
> >>
> >> If there are no subscriptions to a new topic, and therefor no partition
> >> assignments, and definitely no committed offsets, then lag is an
> undefined
> >> concept. When the consumers subscribe to this new topic they may chose
> to
> >> start at the beginning or end of the commit log so the lag cannot be
> >> predicted in advance.
> >>
> >> -hans
> >>
> >>> On Feb 4, 2018, at 11:51 AM, Wouter Bancken <wo...@aca-it.be>
> >> wrote:
> >>>
> >>> Can anyone clarify if this is a bug in Kafka or the expected behavior?
> >>>
> >>> Best regards,
> >>> Wouter
> >>>
> >>>
> >>> On 30 January 2018 at 21:04, Wouter Bancken <wo...@aca-it.be>
> >>> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> I'm trying to write an external tool to monitor consumer lag on Apache
> >>>> Kafka.
> >>>>
> >>>> For this purpose, I'm using the kafka-consumer-groups tool to fetch
> the
> >>>> consumer offsets.
> >>>>
> >>>> When using this tool, partition assignments seem to be unavailable
> >>>> temporarily during the creation of a new topic even if the consumer
> >> group
> >>>> has no subscription on this new topic. This seems to match the
> >>>> documentation
> >>>> <https://cwiki.apache.org/confluence/display/KAFKA/
> >> Kafka+Client-side+Assignment+Proposal>
> >>>> saying *"Topic metadata changes which have no impact on subscriptions
> >>>> cause resync"*.
> >>>>
> >>>> However, when this occurs I'd expect the state of the consumer to be
> >>>> "PreparingRebalance" or "AwaitingSync" but it is simply "Stable".
> >>>>
> >>>> Is this a bug in the tooling or is there a different way to obtain the
> >>>> correct offsets for a consumer group during a rebalance?
> >>>>
> >>>> I'm using Kafka 10.2.1 but I haven't found any related issues in
> recent
> >>>> changelogs.
> >>>> Best regards,
> >>>> Wouter
> >>>>
> >>
>
Re: Kafka Consumer Offsets unavailable during rebalancing
Posted by Hans Jespersen <ha...@confluent.io>.
Do the consumers in consumer group ‘X’ have a regex subscription that matches the newly created topic ‘C’?
If they do then they will only discover this new topic once their ‘metadata.max.age.ms’ metadata refresh interval has passed, which defaults to 5 minutes.
metadata.max.age.ms The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new brokers or partitions
-hans
> On Feb 4, 2018, at 2:16 PM, Wouter Bancken <wo...@aca-it.be> wrote:
>
> Hi Hans,
>
> Thanks for the response!
>
> However, I get this result for all topics, not just for the newly created
> topic.
>
> Situation sketch:
> 1. I have a consumer group 'X' subscribed to topics 'A' and 'B' with
> partition assignments and lag information. Consumer group 'X' is "Stable".
> 2a. Topic 'C' is (being) created.
> 2b. During this creation, I do not have a partition assignment for consumer
> group 'X' for topics 'A' and 'B' but the consumer group is still "Stable".
> 3. A second later: I have a partition assignment for consumer group 'X' for
> topics 'A' and 'B' again and the consumer group is still "Stable".
>
> I expected the state of consumer group 'X' during step 2b to be
> "PreparingRebalance" or "AwaitingSync".
>
> Best regards,
> Wouter
>
>> On 4 February 2018 at 21:25, Hans Jespersen <ha...@confluent.io> wrote:
>>
>> I believe this is expected behavior.
>>
>> If there are no subscriptions to a new topic, and therefor no partition
>> assignments, and definitely no committed offsets, then lag is an undefined
>> concept. When the consumers subscribe to this new topic they may chose to
>> start at the beginning or end of the commit log so the lag cannot be
>> predicted in advance.
>>
>> -hans
>>
>>> On Feb 4, 2018, at 11:51 AM, Wouter Bancken <wo...@aca-it.be>
>> wrote:
>>>
>>> Can anyone clarify if this is a bug in Kafka or the expected behavior?
>>>
>>> Best regards,
>>> Wouter
>>>
>>>
>>> On 30 January 2018 at 21:04, Wouter Bancken <wo...@aca-it.be>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I'm trying to write an external tool to monitor consumer lag on Apache
>>>> Kafka.
>>>>
>>>> For this purpose, I'm using the kafka-consumer-groups tool to fetch the
>>>> consumer offsets.
>>>>
>>>> When using this tool, partition assignments seem to be unavailable
>>>> temporarily during the creation of a new topic even if the consumer
>> group
>>>> has no subscription on this new topic. This seems to match the
>>>> documentation
>>>> <https://cwiki.apache.org/confluence/display/KAFKA/
>> Kafka+Client-side+Assignment+Proposal>
>>>> saying *"Topic metadata changes which have no impact on subscriptions
>>>> cause resync"*.
>>>>
>>>> However, when this occurs I'd expect the state of the consumer to be
>>>> "PreparingRebalance" or "AwaitingSync" but it is simply "Stable".
>>>>
>>>> Is this a bug in the tooling or is there a different way to obtain the
>>>> correct offsets for a consumer group during a rebalance?
>>>>
>>>> I'm using Kafka 10.2.1 but I haven't found any related issues in recent
>>>> changelogs.
>>>> Best regards,
>>>> Wouter
>>>>
>>
Re: Kafka Consumer Offsets unavailable during rebalancing
Posted by Wouter Bancken <wo...@aca-it.be>.
Hi Hans,
Thanks for the response!
However, I get this result for all topics, not just for the newly created
topic.
Situation sketch:
1. I have a consumer group 'X' subscribed to topics 'A' and 'B' with
partition assignments and lag information. Consumer group 'X' is "Stable".
2a. Topic 'C' is (being) created.
2b. During this creation, I do not have a partition assignment for consumer
group 'X' for topics 'A' and 'B' but the consumer group is still "Stable".
3. A second later: I have a partition assignment for consumer group 'X' for
topics 'A' and 'B' again and the consumer group is still "Stable".
I expected the state of consumer group 'X' during step 2b to be
"PreparingRebalance" or "AwaitingSync".
Best regards,
Wouter
On 4 February 2018 at 21:25, Hans Jespersen <ha...@confluent.io> wrote:
> I believe this is expected behavior.
>
> If there are no subscriptions to a new topic, and therefor no partition
> assignments, and definitely no committed offsets, then lag is an undefined
> concept. When the consumers subscribe to this new topic they may chose to
> start at the beginning or end of the commit log so the lag cannot be
> predicted in advance.
>
> -hans
>
> > On Feb 4, 2018, at 11:51 AM, Wouter Bancken <wo...@aca-it.be>
> wrote:
> >
> > Can anyone clarify if this is a bug in Kafka or the expected behavior?
> >
> > Best regards,
> > Wouter
> >
> >
> > On 30 January 2018 at 21:04, Wouter Bancken <wo...@aca-it.be>
> > wrote:
> >
> >> Hi,
> >>
> >> I'm trying to write an external tool to monitor consumer lag on Apache
> >> Kafka.
> >>
> >> For this purpose, I'm using the kafka-consumer-groups tool to fetch the
> >> consumer offsets.
> >>
> >> When using this tool, partition assignments seem to be unavailable
> >> temporarily during the creation of a new topic even if the consumer
> group
> >> has no subscription on this new topic. This seems to match the
> >> documentation
> >> <https://cwiki.apache.org/confluence/display/KAFKA/
> Kafka+Client-side+Assignment+Proposal>
> >> saying *"Topic metadata changes which have no impact on subscriptions
> >> cause resync"*.
> >>
> >> However, when this occurs I'd expect the state of the consumer to be
> >> "PreparingRebalance" or "AwaitingSync" but it is simply "Stable".
> >>
> >> Is this a bug in the tooling or is there a different way to obtain the
> >> correct offsets for a consumer group during a rebalance?
> >>
> >> I'm using Kafka 10.2.1 but I haven't found any related issues in recent
> >> changelogs.
> >> Best regards,
> >> Wouter
> >>
>
Re: Kafka Consumer Offsets unavailable during rebalancing
Posted by Hans Jespersen <ha...@confluent.io>.
I believe this is expected behavior.
If there are no subscriptions to a new topic, and therefor no partition assignments, and definitely no committed offsets, then lag is an undefined concept. When the consumers subscribe to this new topic they may chose to start at the beginning or end of the commit log so the lag cannot be predicted in advance.
-hans
> On Feb 4, 2018, at 11:51 AM, Wouter Bancken <wo...@aca-it.be> wrote:
>
> Can anyone clarify if this is a bug in Kafka or the expected behavior?
>
> Best regards,
> Wouter
>
>
> On 30 January 2018 at 21:04, Wouter Bancken <wo...@aca-it.be>
> wrote:
>
>> Hi,
>>
>> I'm trying to write an external tool to monitor consumer lag on Apache
>> Kafka.
>>
>> For this purpose, I'm using the kafka-consumer-groups tool to fetch the
>> consumer offsets.
>>
>> When using this tool, partition assignments seem to be unavailable
>> temporarily during the creation of a new topic even if the consumer group
>> has no subscription on this new topic. This seems to match the
>> documentation
>> <https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Client-side+Assignment+Proposal>
>> saying *"Topic metadata changes which have no impact on subscriptions
>> cause resync"*.
>>
>> However, when this occurs I'd expect the state of the consumer to be
>> "PreparingRebalance" or "AwaitingSync" but it is simply "Stable".
>>
>> Is this a bug in the tooling or is there a different way to obtain the
>> correct offsets for a consumer group during a rebalance?
>>
>> I'm using Kafka 10.2.1 but I haven't found any related issues in recent
>> changelogs.
>> Best regards,
>> Wouter
>>