You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by shargan <ka...@shargan.me> on 2017/09/20 20:40:17 UTC
Can't reset consumer group offsets in Kafka 0.11.0.1
Hi there,
The Kafka servers in question were 0.11.0.0 this morning. I upgraded to
0.11.0.1 after discovering this issue but I don’t see any change. Scala
2.11.
A client using the new-style consumer groups is not consuming any new
messages because its current committed offset is out of range. I shut down
the clients to ensure the consumer group was inactive and checked the
offset state:
$ kafka-consumer-groups.sh --new-consumer --bootstrap-server
kafka.service:9092 --group router —describe | egrep 'TOPIC|app-router-logs’
Note: This will only show information about consumers that use the Java
consumer API (non-ZooKeeper-based consumers).
Consumer group 'router' has no active members.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET
LAG CONSUMER-ID HOST
CLIENT-ID
app-router-logs 0 728660106 2324523934
1595863828 - -
-
app-router-logs 1 728342585 2324526718
1596184133 - -
-
app-router-logs 2 728698812 2324430267
1595731455 - -
-
Resetting the offsets acts like it worked, but doesn’t:
$ kafka-consumer-groups.sh --new-consumer --bootstrap-server
kafka.service:9092 --group router --reset-offsets --topic app-router-logs
--to-latest
Note: This will only show information about consumers that use the Java
consumer API (non-ZooKeeper-based consumers).
TOPIC PARTITION NEW-OFFSET
app-router-logs 2 2325666004
app-router-logs 1 2325765832
app-router-logs 0 2325763132
kafka-consumer-groups.sh --new-consumer --bootstrap-server
kafka.service:9092 --group router --describe | egrep 'TOPIC|app-router-logs'
Note: This will only show information about consumers that use the Java
consumer API (non-ZooKeeper-based consumers).
Consumer group 'router' has no active members.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET
LAG CONSUMER-ID HOST
CLIENT-ID
app-router-logs 0 728660106 2326329513
1597669407 - -
-
app-router-logs 1 728342585 2326334386
1597991801 - -
-
app-router-logs 2 728698812 2326237143
1597538331 - -
-
I’ve also tried —to-earliest, —to-current, and individually setting each
partition via —to-offset and there’s no difference. What’s going on here?
-shargan
Re: Can't reset consumer group offsets in Kafka 0.11.0.1
Posted by shargan <ka...@shargan.me>.
I completely missed that in the docs (for me, j think it's because this is
the only operation I know of that requires that kind of confirmation).
Thanks!
-shargan
On Thu, Sep 21, 2017 at 12:41 Hans Jespersen <ha...@confluent.io> wrote:
> Did you add the --execute flag?
>
> -hans
>
> > On Sep 21, 2017, at 11:37 AM, shargan <ka...@shargan.me> wrote:
> >
> > Testing kafka-consumer-groups.sh in my dev environment, I'm unable to
> reset
> > offsets even when CURRENT-OFFSET is inbounds. Again, it returns as if the
> > change took effect but describe still shows the original value and client
> > behavior bears this out.
> >
> > -shargan
> >
> >> On Wed, Sep 20, 2017 at 13:40 shargan <ka...@shargan.me> wrote:
> >>
> >> Hi there,
> >>
> >> The Kafka servers in question were 0.11.0.0 this morning. I upgraded to
> >> 0.11.0.1 after discovering this issue but I don’t see any change. Scala
> >> 2.11.
> >>
> >> A client using the new-style consumer groups is not consuming any new
> >> messages because its current committed offset is out of range. I shut
> down
> >> the clients to ensure the consumer group was inactive and checked the
> >> offset state:
> >>
> >> $ kafka-consumer-groups.sh --new-consumer --bootstrap-server
> >> kafka.service:9092 --group router —describe | egrep
> 'TOPIC|app-router-logs’
> >> Note: This will only show information about consumers that use the Java
> >> consumer API (non-ZooKeeper-based consumers).
> >> Consumer group 'router' has no active members.
> >> TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET
> >> LAG CONSUMER-ID HOST
> >> CLIENT-ID
> >> app-router-logs 0 728660106 2324523934
> >> 1595863828 - -
> >> -
> >> app-router-logs 1 728342585 2324526718
> >> 1596184133 - -
> >> -
> >> app-router-logs 2 728698812 2324430267
> >> 1595731455 - -
> >> -
> >>
> >>
> >>
> >>
> >> Resetting the offsets acts like it worked, but doesn’t:
> >>
> >> $ kafka-consumer-groups.sh --new-consumer --bootstrap-server
> >> kafka.service:9092 --group router --reset-offsets --topic
> app-router-logs
> >> --to-latest
> >> Note: This will only show information about consumers that use the Java
> >> consumer API (non-ZooKeeper-based consumers).
> >>
> >> TOPIC PARTITION NEW-OFFSET
> >> app-router-logs 2 2325666004
> >> app-router-logs 1 2325765832
> >> app-router-logs 0 2325763132
> >>
> >>
> >>
> >> kafka-consumer-groups.sh --new-consumer --bootstrap-server
> >> kafka.service:9092 --group router --describe | egrep
> 'TOPIC|app-router-logs'
> >> Note: This will only show information about consumers that use the Java
> >> consumer API (non-ZooKeeper-based consumers).
> >> Consumer group 'router' has no active members.
> >> TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET
> >> LAG CONSUMER-ID HOST
> >> CLIENT-ID
> >> app-router-logs 0 728660106 2326329513
> >> 1597669407 - -
> >> -
> >> app-router-logs 1 728342585 2326334386
> >> 1597991801 - -
> >> -
> >> app-router-logs 2 728698812 2326237143
> >> 1597538331 - -
> >> -
> >>
> >>
> >>
> >> I’ve also tried —to-earliest, —to-current, and individually setting each
> >> partition via —to-offset and there’s no difference. What’s going on
> here?
> >>
> >>
> >> -shargan
> >>
>
Re: Can't reset consumer group offsets in Kafka 0.11.0.1
Posted by Hans Jespersen <ha...@confluent.io>.
Did you add the --execute flag?
-hans
> On Sep 21, 2017, at 11:37 AM, shargan <ka...@shargan.me> wrote:
>
> Testing kafka-consumer-groups.sh in my dev environment, I'm unable to reset
> offsets even when CURRENT-OFFSET is inbounds. Again, it returns as if the
> change took effect but describe still shows the original value and client
> behavior bears this out.
>
> -shargan
>
>> On Wed, Sep 20, 2017 at 13:40 shargan <ka...@shargan.me> wrote:
>>
>> Hi there,
>>
>> The Kafka servers in question were 0.11.0.0 this morning. I upgraded to
>> 0.11.0.1 after discovering this issue but I don’t see any change. Scala
>> 2.11.
>>
>> A client using the new-style consumer groups is not consuming any new
>> messages because its current committed offset is out of range. I shut down
>> the clients to ensure the consumer group was inactive and checked the
>> offset state:
>>
>> $ kafka-consumer-groups.sh --new-consumer --bootstrap-server
>> kafka.service:9092 --group router —describe | egrep 'TOPIC|app-router-logs’
>> Note: This will only show information about consumers that use the Java
>> consumer API (non-ZooKeeper-based consumers).
>> Consumer group 'router' has no active members.
>> TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET
>> LAG CONSUMER-ID HOST
>> CLIENT-ID
>> app-router-logs 0 728660106 2324523934
>> 1595863828 - -
>> -
>> app-router-logs 1 728342585 2324526718
>> 1596184133 - -
>> -
>> app-router-logs 2 728698812 2324430267
>> 1595731455 - -
>> -
>>
>>
>>
>>
>> Resetting the offsets acts like it worked, but doesn’t:
>>
>> $ kafka-consumer-groups.sh --new-consumer --bootstrap-server
>> kafka.service:9092 --group router --reset-offsets --topic app-router-logs
>> --to-latest
>> Note: This will only show information about consumers that use the Java
>> consumer API (non-ZooKeeper-based consumers).
>>
>> TOPIC PARTITION NEW-OFFSET
>> app-router-logs 2 2325666004
>> app-router-logs 1 2325765832
>> app-router-logs 0 2325763132
>>
>>
>>
>> kafka-consumer-groups.sh --new-consumer --bootstrap-server
>> kafka.service:9092 --group router --describe | egrep 'TOPIC|app-router-logs'
>> Note: This will only show information about consumers that use the Java
>> consumer API (non-ZooKeeper-based consumers).
>> Consumer group 'router' has no active members.
>> TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET
>> LAG CONSUMER-ID HOST
>> CLIENT-ID
>> app-router-logs 0 728660106 2326329513
>> 1597669407 - -
>> -
>> app-router-logs 1 728342585 2326334386
>> 1597991801 - -
>> -
>> app-router-logs 2 728698812 2326237143
>> 1597538331 - -
>> -
>>
>>
>>
>> I’ve also tried —to-earliest, —to-current, and individually setting each
>> partition via —to-offset and there’s no difference. What’s going on here?
>>
>>
>> -shargan
>>
Re: Can't reset consumer group offsets in Kafka 0.11.0.1
Posted by shargan <ka...@shargan.me>.
Testing kafka-consumer-groups.sh in my dev environment, I'm unable to reset
offsets even when CURRENT-OFFSET is inbounds. Again, it returns as if the
change took effect but describe still shows the original value and client
behavior bears this out.
-shargan
On Wed, Sep 20, 2017 at 13:40 shargan <ka...@shargan.me> wrote:
> Hi there,
>
> The Kafka servers in question were 0.11.0.0 this morning. I upgraded to
> 0.11.0.1 after discovering this issue but I don’t see any change. Scala
> 2.11.
>
> A client using the new-style consumer groups is not consuming any new
> messages because its current committed offset is out of range. I shut down
> the clients to ensure the consumer group was inactive and checked the
> offset state:
>
> $ kafka-consumer-groups.sh --new-consumer --bootstrap-server
> kafka.service:9092 --group router —describe | egrep 'TOPIC|app-router-logs’
> Note: This will only show information about consumers that use the Java
> consumer API (non-ZooKeeper-based consumers).
> Consumer group 'router' has no active members.
> TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET
> LAG CONSUMER-ID HOST
> CLIENT-ID
> app-router-logs 0 728660106 2324523934
> 1595863828 - -
> -
> app-router-logs 1 728342585 2324526718
> 1596184133 - -
> -
> app-router-logs 2 728698812 2324430267
> 1595731455 - -
> -
>
>
>
>
> Resetting the offsets acts like it worked, but doesn’t:
>
> $ kafka-consumer-groups.sh --new-consumer --bootstrap-server
> kafka.service:9092 --group router --reset-offsets --topic app-router-logs
> --to-latest
> Note: This will only show information about consumers that use the Java
> consumer API (non-ZooKeeper-based consumers).
>
> TOPIC PARTITION NEW-OFFSET
> app-router-logs 2 2325666004
> app-router-logs 1 2325765832
> app-router-logs 0 2325763132
>
>
>
> kafka-consumer-groups.sh --new-consumer --bootstrap-server
> kafka.service:9092 --group router --describe | egrep 'TOPIC|app-router-logs'
> Note: This will only show information about consumers that use the Java
> consumer API (non-ZooKeeper-based consumers).
> Consumer group 'router' has no active members.
> TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET
> LAG CONSUMER-ID HOST
> CLIENT-ID
> app-router-logs 0 728660106 2326329513
> 1597669407 - -
> -
> app-router-logs 1 728342585 2326334386
> 1597991801 - -
> -
> app-router-logs 2 728698812 2326237143
> 1597538331 - -
> -
>
>
>
> I’ve also tried —to-earliest, —to-current, and individually setting each
> partition via —to-offset and there’s no difference. What’s going on here?
>
>
> -shargan
>