You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by shargan <ka...@shargan.me> on 2017/09/20 20:40:17 UTC

Can't reset consumer group offsets in Kafka 0.11.0.1

Hi there,

The Kafka servers in question were 0.11.0.0 this morning. I upgraded to
0.11.0.1 after discovering this issue but I don’t see any change. Scala
2.11.

A client using the new-style consumer groups is not consuming any new
messages because its current committed offset is out of range. I shut down
the clients to ensure the consumer group was inactive and checked the
offset state:

$ kafka-consumer-groups.sh --new-consumer --bootstrap-server
kafka.service:9092 --group router —describe | egrep 'TOPIC|app-router-logs’
Note: This will only show information about consumers that use the Java
consumer API (non-ZooKeeper-based consumers).
Consumer group 'router' has no active members.
TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET
 LAG        CONSUMER-ID                                       HOST
                  CLIENT-ID
app-router-logs                0          728660106       2324523934
 1595863828 -                                                 -
                 -
app-router-logs                1          728342585       2324526718
 1596184133 -                                                 -
                 -
app-router-logs                2          728698812       2324430267
 1595731455 -                                                 -
                 -




Resetting the offsets acts like it worked, but doesn’t:

$ kafka-consumer-groups.sh --new-consumer --bootstrap-server
kafka.service:9092 --group router --reset-offsets --topic app-router-logs
--to-latest
Note: This will only show information about consumers that use the Java
consumer API (non-ZooKeeper-based consumers).

TOPIC                          PARTITION  NEW-OFFSET
app-router-logs                2          2325666004
app-router-logs                1          2325765832
app-router-logs                0          2325763132



kafka-consumer-groups.sh --new-consumer --bootstrap-server
kafka.service:9092 --group router --describe | egrep 'TOPIC|app-router-logs'
Note: This will only show information about consumers that use the Java
consumer API (non-ZooKeeper-based consumers).
Consumer group 'router' has no active members.
TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET
 LAG        CONSUMER-ID                                       HOST
                  CLIENT-ID
app-router-logs                0          728660106       2326329513
 1597669407 -                                                 -
                 -
app-router-logs                1          728342585       2326334386
 1597991801 -                                                 -
                 -
app-router-logs                2          728698812       2326237143
 1597538331 -                                                 -
                 -



I’ve also tried —to-earliest, —to-current, and individually setting each
partition via —to-offset and there’s no difference. What’s going on here?

-shargan

Re: Can't reset consumer group offsets in Kafka 0.11.0.1

Posted by shargan <ka...@shargan.me>.
I completely missed that in the docs (for me, j think it's because this is
the only operation I know of that requires that kind of confirmation).
Thanks!

-shargan

On Thu, Sep 21, 2017 at 12:41 Hans Jespersen <ha...@confluent.io> wrote:

> Did you add the --execute flag?
>
> -hans
>
> > On Sep 21, 2017, at 11:37 AM, shargan <ka...@shargan.me> wrote:
> >
> > Testing kafka-consumer-groups.sh in my dev environment, I'm unable to
> reset
> > offsets even when CURRENT-OFFSET is inbounds. Again, it returns as if the
> > change took effect but describe still shows the original value and client
> > behavior bears this out.
> >
> > -shargan
> >
> >> On Wed, Sep 20, 2017 at 13:40 shargan <ka...@shargan.me> wrote:
> >>
> >> Hi there,
> >>
> >> The Kafka servers in question were 0.11.0.0 this morning. I upgraded to
> >> 0.11.0.1 after discovering this issue but I don’t see any change. Scala
> >> 2.11.
> >>
> >> A client using the new-style consumer groups is not consuming any new
> >> messages because its current committed offset is out of range. I shut
> down
> >> the clients to ensure the consumer group was inactive and checked the
> >> offset state:
> >>
> >> $ kafka-consumer-groups.sh --new-consumer --bootstrap-server
> >> kafka.service:9092 --group router —describe | egrep
> 'TOPIC|app-router-logs’
> >> Note: This will only show information about consumers that use the Java
> >> consumer API (non-ZooKeeper-based consumers).
> >> Consumer group 'router' has no active members.
> >> TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET
> >> LAG        CONSUMER-ID                                       HOST
> >>                  CLIENT-ID
> >> app-router-logs                0          728660106       2324523934
> >> 1595863828 -                                                 -
> >>                 -
> >> app-router-logs                1          728342585       2324526718
> >> 1596184133 -                                                 -
> >>                 -
> >> app-router-logs                2          728698812       2324430267
> >> 1595731455 -                                                 -
> >>                 -
> >>
> >>
> >>
> >>
> >> Resetting the offsets acts like it worked, but doesn’t:
> >>
> >> $ kafka-consumer-groups.sh --new-consumer --bootstrap-server
> >> kafka.service:9092 --group router --reset-offsets --topic
> app-router-logs
> >> --to-latest
> >> Note: This will only show information about consumers that use the Java
> >> consumer API (non-ZooKeeper-based consumers).
> >>
> >> TOPIC                          PARTITION  NEW-OFFSET
> >> app-router-logs                2          2325666004
> >> app-router-logs                1          2325765832
> >> app-router-logs                0          2325763132
> >>
> >>
> >>
> >> kafka-consumer-groups.sh --new-consumer --bootstrap-server
> >> kafka.service:9092 --group router --describe | egrep
> 'TOPIC|app-router-logs'
> >> Note: This will only show information about consumers that use the Java
> >> consumer API (non-ZooKeeper-based consumers).
> >> Consumer group 'router' has no active members.
> >> TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET
> >> LAG        CONSUMER-ID                                       HOST
> >>                  CLIENT-ID
> >> app-router-logs                0          728660106       2326329513
> >> 1597669407 -                                                 -
> >>                 -
> >> app-router-logs                1          728342585       2326334386
> >> 1597991801 -                                                 -
> >>                 -
> >> app-router-logs                2          728698812       2326237143
> >> 1597538331 -                                                 -
> >>                 -
> >>
> >>
> >>
> >> I’ve also tried —to-earliest, —to-current, and individually setting each
> >> partition via —to-offset and there’s no difference. What’s going on
> here?
> >>
> >>
> >> -shargan
> >>
>

Re: Can't reset consumer group offsets in Kafka 0.11.0.1

Posted by Hans Jespersen <ha...@confluent.io>.
Did you add the --execute flag?

-hans

> On Sep 21, 2017, at 11:37 AM, shargan <ka...@shargan.me> wrote:
> 
> Testing kafka-consumer-groups.sh in my dev environment, I'm unable to reset
> offsets even when CURRENT-OFFSET is inbounds. Again, it returns as if the
> change took effect but describe still shows the original value and client
> behavior bears this out.
> 
> -shargan
> 
>> On Wed, Sep 20, 2017 at 13:40 shargan <ka...@shargan.me> wrote:
>> 
>> Hi there,
>> 
>> The Kafka servers in question were 0.11.0.0 this morning. I upgraded to
>> 0.11.0.1 after discovering this issue but I don’t see any change. Scala
>> 2.11.
>> 
>> A client using the new-style consumer groups is not consuming any new
>> messages because its current committed offset is out of range. I shut down
>> the clients to ensure the consumer group was inactive and checked the
>> offset state:
>> 
>> $ kafka-consumer-groups.sh --new-consumer --bootstrap-server
>> kafka.service:9092 --group router —describe | egrep 'TOPIC|app-router-logs’
>> Note: This will only show information about consumers that use the Java
>> consumer API (non-ZooKeeper-based consumers).
>> Consumer group 'router' has no active members.
>> TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET
>> LAG        CONSUMER-ID                                       HOST
>>                  CLIENT-ID
>> app-router-logs                0          728660106       2324523934
>> 1595863828 -                                                 -
>>                 -
>> app-router-logs                1          728342585       2324526718
>> 1596184133 -                                                 -
>>                 -
>> app-router-logs                2          728698812       2324430267
>> 1595731455 -                                                 -
>>                 -
>> 
>> 
>> 
>> 
>> Resetting the offsets acts like it worked, but doesn’t:
>> 
>> $ kafka-consumer-groups.sh --new-consumer --bootstrap-server
>> kafka.service:9092 --group router --reset-offsets --topic app-router-logs
>> --to-latest
>> Note: This will only show information about consumers that use the Java
>> consumer API (non-ZooKeeper-based consumers).
>> 
>> TOPIC                          PARTITION  NEW-OFFSET
>> app-router-logs                2          2325666004
>> app-router-logs                1          2325765832
>> app-router-logs                0          2325763132
>> 
>> 
>> 
>> kafka-consumer-groups.sh --new-consumer --bootstrap-server
>> kafka.service:9092 --group router --describe | egrep 'TOPIC|app-router-logs'
>> Note: This will only show information about consumers that use the Java
>> consumer API (non-ZooKeeper-based consumers).
>> Consumer group 'router' has no active members.
>> TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET
>> LAG        CONSUMER-ID                                       HOST
>>                  CLIENT-ID
>> app-router-logs                0          728660106       2326329513
>> 1597669407 -                                                 -
>>                 -
>> app-router-logs                1          728342585       2326334386
>> 1597991801 -                                                 -
>>                 -
>> app-router-logs                2          728698812       2326237143
>> 1597538331 -                                                 -
>>                 -
>> 
>> 
>> 
>> I’ve also tried —to-earliest, —to-current, and individually setting each
>> partition via —to-offset and there’s no difference. What’s going on here?
>> 
>> 
>> -shargan
>> 

Re: Can't reset consumer group offsets in Kafka 0.11.0.1

Posted by shargan <ka...@shargan.me>.
Testing kafka-consumer-groups.sh in my dev environment, I'm unable to reset
offsets even when CURRENT-OFFSET is inbounds. Again, it returns as if the
change took effect but describe still shows the original value and client
behavior bears this out.

-shargan

On Wed, Sep 20, 2017 at 13:40 shargan <ka...@shargan.me> wrote:

> Hi there,
>
> The Kafka servers in question were 0.11.0.0 this morning. I upgraded to
> 0.11.0.1 after discovering this issue but I don’t see any change. Scala
> 2.11.
>
> A client using the new-style consumer groups is not consuming any new
> messages because its current committed offset is out of range. I shut down
> the clients to ensure the consumer group was inactive and checked the
> offset state:
>
> $ kafka-consumer-groups.sh --new-consumer --bootstrap-server
> kafka.service:9092 --group router —describe | egrep 'TOPIC|app-router-logs’
> Note: This will only show information about consumers that use the Java
> consumer API (non-ZooKeeper-based consumers).
> Consumer group 'router' has no active members.
> TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET
>  LAG        CONSUMER-ID                                       HOST
>                   CLIENT-ID
> app-router-logs                0          728660106       2324523934
>  1595863828 -                                                 -
>                  -
> app-router-logs                1          728342585       2324526718
>  1596184133 -                                                 -
>                  -
> app-router-logs                2          728698812       2324430267
>  1595731455 -                                                 -
>                  -
>
>
>
>
> Resetting the offsets acts like it worked, but doesn’t:
>
> $ kafka-consumer-groups.sh --new-consumer --bootstrap-server
> kafka.service:9092 --group router --reset-offsets --topic app-router-logs
> --to-latest
> Note: This will only show information about consumers that use the Java
> consumer API (non-ZooKeeper-based consumers).
>
> TOPIC                          PARTITION  NEW-OFFSET
> app-router-logs                2          2325666004
> app-router-logs                1          2325765832
> app-router-logs                0          2325763132
>
>
>
> kafka-consumer-groups.sh --new-consumer --bootstrap-server
> kafka.service:9092 --group router --describe | egrep 'TOPIC|app-router-logs'
> Note: This will only show information about consumers that use the Java
> consumer API (non-ZooKeeper-based consumers).
> Consumer group 'router' has no active members.
> TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET
>  LAG        CONSUMER-ID                                       HOST
>                   CLIENT-ID
> app-router-logs                0          728660106       2326329513
>  1597669407 -                                                 -
>                  -
> app-router-logs                1          728342585       2326334386
>  1597991801 -                                                 -
>                  -
> app-router-logs                2          728698812       2326237143
>  1597538331 -                                                 -
>                  -
>
>
>
> I’ve also tried —to-earliest, —to-current, and individually setting each
> partition via —to-offset and there’s no difference. What’s going on here?
>
>
> -shargan
>