You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Tarun Garg <bi...@live.com> on 2017/07/25 16:28:34 UTC

consumer group offset lag

I am sending this mail to DEV because i think this is a bug in AdminClient while getting the group offset.

Hi,

I run through a issue.
i am trying to get the status of a consumer group.

for this i have consumer and producer with topic named test_7 with partition count 1. It is working fine.

1. then i stoped my services and increased the number of partition to to and then check the status it doesn’t showed me the new partitions.
2. i started my producer to populated data still the consumer group status check gives me the same response.
3. finally i started my consumer and then i am able to see the data and lag

This is not giving me the correct lag.

Any idea how to get that information out.

Thanks
Tarun

tarun.garg:tgarg kafka_2.12-0.10.2.0> bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group ReactiveConsumerTest_3
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).

Consumer group 'ReactiveConsumerTest_3' has no active members.

TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID                                       HOST                           CLIENT-ID
test_7                         0          113720          114324          604        -                                                 -                              -

tarun.garg:tgarg kafka_2.12-0.10.2.0> bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group ReactiveConsumerTest_3
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).


TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID                                       HOST                           CLIENT-ID
test_7                         0          114700          114744          44         consumer-1-47b3f36-f897-460d-b178-d03f7f11a749   /172.17.0.1                    consumer-1
test_7                         1          245             256             11         consumer-1-47b3f36-f897-460d-b178-d03f7f11a749   /172.17.0.1                    consumer-1
test_7                         2          980             1024            44         consumer-1-47b3f36-f897-460d-b178-d03f7f11a749   /172.17.0.1                    consumer-1
test_7                         3          1470            1536            66         consumer-1-47b3f36-f897-460d-b178-d03f7f11a749   /172.17.0.1                    consumer-1
test_7                         4          245             256             11         consumer-1-47b3f36-f897-460d-b178-d03f7f11a749   /172.17.0.1                    consumer-1
test_7                         5          1225            1280            55         consumer-1-47b3f36-f897-460d-b178-d03f7f11a749   /172.17.0.1                    consumer-1
test_7                         6          490             512             22         consumer-1-47b3f36-f897-460d-b178-d03f7f11a749   /172.17.0.1                    consumer-1
test_7                         7          1715            1792            77         consumer-1-47b3f36-f897-460d-b178-d03f7f11a749   /172.17.0.1                    consumer-1
test_7                         8          1715            1792            77         consumer-1-47b3f36-f897-460d-b178-d03f7f11a749   /172.17.0.1                    consumer-1
test_7                         9          735             768             33         consumer-1-47b3f36-f897-460d-b178-d03f7f11a749   /172.17.0.1                    consumer-1


Re: consumer group offset lag

Posted by Abhimanyu Nagrath <ab...@gmail.com>.
I am also facing the same issue.

On Tue, Jul 25, 2017 at 9:58 PM, Tarun Garg <bi...@live.com> wrote:

> I am sending this mail to DEV because i think this is a bug in AdminClient
> while getting the group offset.
>
> Hi,
>
> I run through a issue.
> i am trying to get the status of a consumer group.
>
> for this i have consumer and producer with topic named test_7 with
> partition count 1. It is working fine.
>
> 1. then i stoped my services and increased the number of partition to to
> and then check the status it doesn’t showed me the new partitions.
> 2. i started my producer to populated data still the consumer group status
> check gives me the same response.
> 3. finally i started my consumer and then i am able to see the data and lag
>
> This is not giving me the correct lag.
>
> Any idea how to get that information out.
>
> Thanks
> Tarun
>
> tarun.garg:tgarg kafka_2.12-0.10.2.0> bin/kafka-consumer-groups.sh
> --bootstrap-server localhost:9092 --describe --group ReactiveConsumerTest_3
> Note: This will only show information about consumers that use the Java
> consumer API (non-ZooKeeper-based consumers).
>
> Consumer group 'ReactiveConsumerTest_3' has no active members.
>
> TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET
> LAG        CONSUMER-ID                                       HOST
>                  CLIENT-ID
> test_7                         0          113720          114324
> 604        -                                                 -
>                 -
>
> tarun.garg:tgarg kafka_2.12-0.10.2.0> bin/kafka-consumer-groups.sh
> --bootstrap-server localhost:9092 --describe --group ReactiveConsumerTest_3
> Note: This will only show information about consumers that use the Java
> consumer API (non-ZooKeeper-based consumers).
>
>
> TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET
> LAG        CONSUMER-ID                                       HOST
>                  CLIENT-ID
> test_7                         0          114700          114744
> 44         consumer-1-47b3f36-f897-460d-b178-d03f7f11a749   /172.17.0.1
>                   consumer-1
> test_7                         1          245             256
>  11         consumer-1-47b3f36-f897-460d-b178-d03f7f11a749   /172.17.0.1
>                   consumer-1
> test_7                         2          980             1024
> 44         consumer-1-47b3f36-f897-460d-b178-d03f7f11a749   /172.17.0.1
>                   consumer-1
> test_7                         3          1470            1536
> 66         consumer-1-47b3f36-f897-460d-b178-d03f7f11a749   /172.17.0.1
>                   consumer-1
> test_7                         4          245             256
>  11         consumer-1-47b3f36-f897-460d-b178-d03f7f11a749   /172.17.0.1
>                   consumer-1
> test_7                         5          1225            1280
> 55         consumer-1-47b3f36-f897-460d-b178-d03f7f11a749   /172.17.0.1
>                   consumer-1
> test_7                         6          490             512
>  22         consumer-1-47b3f36-f897-460d-b178-d03f7f11a749   /172.17.0.1
>                   consumer-1
> test_7                         7          1715            1792
> 77         consumer-1-47b3f36-f897-460d-b178-d03f7f11a749   /172.17.0.1
>                   consumer-1
> test_7                         8          1715            1792
> 77         consumer-1-47b3f36-f897-460d-b178-d03f7f11a749   /172.17.0.1
>                   consumer-1
> test_7                         9          735             768
>  33         consumer-1-47b3f36-f897-460d-b178-d03f7f11a749   /172.17.0.1
>                   consumer-1
>
>

Re: consumer group offset lag

Posted by Abhimanyu Nagrath <ab...@gmail.com>.
I am also facing the same issue.

On Tue, Jul 25, 2017 at 9:58 PM, Tarun Garg <bi...@live.com> wrote:

> I am sending this mail to DEV because i think this is a bug in AdminClient
> while getting the group offset.
>
> Hi,
>
> I run through a issue.
> i am trying to get the status of a consumer group.
>
> for this i have consumer and producer with topic named test_7 with
> partition count 1. It is working fine.
>
> 1. then i stoped my services and increased the number of partition to to
> and then check the status it doesn’t showed me the new partitions.
> 2. i started my producer to populated data still the consumer group status
> check gives me the same response.
> 3. finally i started my consumer and then i am able to see the data and lag
>
> This is not giving me the correct lag.
>
> Any idea how to get that information out.
>
> Thanks
> Tarun
>
> tarun.garg:tgarg kafka_2.12-0.10.2.0> bin/kafka-consumer-groups.sh
> --bootstrap-server localhost:9092 --describe --group ReactiveConsumerTest_3
> Note: This will only show information about consumers that use the Java
> consumer API (non-ZooKeeper-based consumers).
>
> Consumer group 'ReactiveConsumerTest_3' has no active members.
>
> TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET
> LAG        CONSUMER-ID                                       HOST
>                  CLIENT-ID
> test_7                         0          113720          114324
> 604        -                                                 -
>                 -
>
> tarun.garg:tgarg kafka_2.12-0.10.2.0> bin/kafka-consumer-groups.sh
> --bootstrap-server localhost:9092 --describe --group ReactiveConsumerTest_3
> Note: This will only show information about consumers that use the Java
> consumer API (non-ZooKeeper-based consumers).
>
>
> TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET
> LAG        CONSUMER-ID                                       HOST
>                  CLIENT-ID
> test_7                         0          114700          114744
> 44         consumer-1-47b3f36-f897-460d-b178-d03f7f11a749   /172.17.0.1
>                   consumer-1
> test_7                         1          245             256
>  11         consumer-1-47b3f36-f897-460d-b178-d03f7f11a749   /172.17.0.1
>                   consumer-1
> test_7                         2          980             1024
> 44         consumer-1-47b3f36-f897-460d-b178-d03f7f11a749   /172.17.0.1
>                   consumer-1
> test_7                         3          1470            1536
> 66         consumer-1-47b3f36-f897-460d-b178-d03f7f11a749   /172.17.0.1
>                   consumer-1
> test_7                         4          245             256
>  11         consumer-1-47b3f36-f897-460d-b178-d03f7f11a749   /172.17.0.1
>                   consumer-1
> test_7                         5          1225            1280
> 55         consumer-1-47b3f36-f897-460d-b178-d03f7f11a749   /172.17.0.1
>                   consumer-1
> test_7                         6          490             512
>  22         consumer-1-47b3f36-f897-460d-b178-d03f7f11a749   /172.17.0.1
>                   consumer-1
> test_7                         7          1715            1792
> 77         consumer-1-47b3f36-f897-460d-b178-d03f7f11a749   /172.17.0.1
>                   consumer-1
> test_7                         8          1715            1792
> 77         consumer-1-47b3f36-f897-460d-b178-d03f7f11a749   /172.17.0.1
>                   consumer-1
> test_7                         9          735             768
>  33         consumer-1-47b3f36-f897-460d-b178-d03f7f11a749   /172.17.0.1
>                   consumer-1
>
>