You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Tarun Garg <bi...@live.com> on 2017/07/25 16:28:34 UTC
consumer group offset lag
I am sending this mail to DEV because i think this is a bug in AdminClient while getting the group offset.
Hi,
I run through a issue.
i am trying to get the status of a consumer group.
for this i have consumer and producer with topic named test_7 with partition count 1. It is working fine.
1. then i stoped my services and increased the number of partition to to and then check the status it doesn’t showed me the new partitions.
2. i started my producer to populated data still the consumer group status check gives me the same response.
3. finally i started my consumer and then i am able to see the data and lag
This is not giving me the correct lag.
Any idea how to get that information out.
Thanks
Tarun
tarun.garg:tgarg kafka_2.12-0.10.2.0> bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group ReactiveConsumerTest_3
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
Consumer group 'ReactiveConsumerTest_3' has no active members.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
test_7 0 113720 114324 604 - - -
tarun.garg:tgarg kafka_2.12-0.10.2.0> bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group ReactiveConsumerTest_3
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
test_7 0 114700 114744 44 consumer-1-47b3f36-f897-460d-b178-d03f7f11a749 /172.17.0.1 consumer-1
test_7 1 245 256 11 consumer-1-47b3f36-f897-460d-b178-d03f7f11a749 /172.17.0.1 consumer-1
test_7 2 980 1024 44 consumer-1-47b3f36-f897-460d-b178-d03f7f11a749 /172.17.0.1 consumer-1
test_7 3 1470 1536 66 consumer-1-47b3f36-f897-460d-b178-d03f7f11a749 /172.17.0.1 consumer-1
test_7 4 245 256 11 consumer-1-47b3f36-f897-460d-b178-d03f7f11a749 /172.17.0.1 consumer-1
test_7 5 1225 1280 55 consumer-1-47b3f36-f897-460d-b178-d03f7f11a749 /172.17.0.1 consumer-1
test_7 6 490 512 22 consumer-1-47b3f36-f897-460d-b178-d03f7f11a749 /172.17.0.1 consumer-1
test_7 7 1715 1792 77 consumer-1-47b3f36-f897-460d-b178-d03f7f11a749 /172.17.0.1 consumer-1
test_7 8 1715 1792 77 consumer-1-47b3f36-f897-460d-b178-d03f7f11a749 /172.17.0.1 consumer-1
test_7 9 735 768 33 consumer-1-47b3f36-f897-460d-b178-d03f7f11a749 /172.17.0.1 consumer-1
Re: consumer group offset lag
Posted by Abhimanyu Nagrath <ab...@gmail.com>.
I am also facing the same issue.
On Tue, Jul 25, 2017 at 9:58 PM, Tarun Garg <bi...@live.com> wrote:
> I am sending this mail to DEV because i think this is a bug in AdminClient
> while getting the group offset.
>
> Hi,
>
> I run through a issue.
> i am trying to get the status of a consumer group.
>
> for this i have consumer and producer with topic named test_7 with
> partition count 1. It is working fine.
>
> 1. then i stoped my services and increased the number of partition to to
> and then check the status it doesn’t showed me the new partitions.
> 2. i started my producer to populated data still the consumer group status
> check gives me the same response.
> 3. finally i started my consumer and then i am able to see the data and lag
>
> This is not giving me the correct lag.
>
> Any idea how to get that information out.
>
> Thanks
> Tarun
>
> tarun.garg:tgarg kafka_2.12-0.10.2.0> bin/kafka-consumer-groups.sh
> --bootstrap-server localhost:9092 --describe --group ReactiveConsumerTest_3
> Note: This will only show information about consumers that use the Java
> consumer API (non-ZooKeeper-based consumers).
>
> Consumer group 'ReactiveConsumerTest_3' has no active members.
>
> TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET
> LAG CONSUMER-ID HOST
> CLIENT-ID
> test_7 0 113720 114324
> 604 - -
> -
>
> tarun.garg:tgarg kafka_2.12-0.10.2.0> bin/kafka-consumer-groups.sh
> --bootstrap-server localhost:9092 --describe --group ReactiveConsumerTest_3
> Note: This will only show information about consumers that use the Java
> consumer API (non-ZooKeeper-based consumers).
>
>
> TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET
> LAG CONSUMER-ID HOST
> CLIENT-ID
> test_7 0 114700 114744
> 44 consumer-1-47b3f36-f897-460d-b178-d03f7f11a749 /172.17.0.1
> consumer-1
> test_7 1 245 256
> 11 consumer-1-47b3f36-f897-460d-b178-d03f7f11a749 /172.17.0.1
> consumer-1
> test_7 2 980 1024
> 44 consumer-1-47b3f36-f897-460d-b178-d03f7f11a749 /172.17.0.1
> consumer-1
> test_7 3 1470 1536
> 66 consumer-1-47b3f36-f897-460d-b178-d03f7f11a749 /172.17.0.1
> consumer-1
> test_7 4 245 256
> 11 consumer-1-47b3f36-f897-460d-b178-d03f7f11a749 /172.17.0.1
> consumer-1
> test_7 5 1225 1280
> 55 consumer-1-47b3f36-f897-460d-b178-d03f7f11a749 /172.17.0.1
> consumer-1
> test_7 6 490 512
> 22 consumer-1-47b3f36-f897-460d-b178-d03f7f11a749 /172.17.0.1
> consumer-1
> test_7 7 1715 1792
> 77 consumer-1-47b3f36-f897-460d-b178-d03f7f11a749 /172.17.0.1
> consumer-1
> test_7 8 1715 1792
> 77 consumer-1-47b3f36-f897-460d-b178-d03f7f11a749 /172.17.0.1
> consumer-1
> test_7 9 735 768
> 33 consumer-1-47b3f36-f897-460d-b178-d03f7f11a749 /172.17.0.1
> consumer-1
>
>
Re: consumer group offset lag
Posted by Abhimanyu Nagrath <ab...@gmail.com>.
I am also facing the same issue.
On Tue, Jul 25, 2017 at 9:58 PM, Tarun Garg <bi...@live.com> wrote:
> I am sending this mail to DEV because i think this is a bug in AdminClient
> while getting the group offset.
>
> Hi,
>
> I run through a issue.
> i am trying to get the status of a consumer group.
>
> for this i have consumer and producer with topic named test_7 with
> partition count 1. It is working fine.
>
> 1. then i stoped my services and increased the number of partition to to
> and then check the status it doesn’t showed me the new partitions.
> 2. i started my producer to populated data still the consumer group status
> check gives me the same response.
> 3. finally i started my consumer and then i am able to see the data and lag
>
> This is not giving me the correct lag.
>
> Any idea how to get that information out.
>
> Thanks
> Tarun
>
> tarun.garg:tgarg kafka_2.12-0.10.2.0> bin/kafka-consumer-groups.sh
> --bootstrap-server localhost:9092 --describe --group ReactiveConsumerTest_3
> Note: This will only show information about consumers that use the Java
> consumer API (non-ZooKeeper-based consumers).
>
> Consumer group 'ReactiveConsumerTest_3' has no active members.
>
> TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET
> LAG CONSUMER-ID HOST
> CLIENT-ID
> test_7 0 113720 114324
> 604 - -
> -
>
> tarun.garg:tgarg kafka_2.12-0.10.2.0> bin/kafka-consumer-groups.sh
> --bootstrap-server localhost:9092 --describe --group ReactiveConsumerTest_3
> Note: This will only show information about consumers that use the Java
> consumer API (non-ZooKeeper-based consumers).
>
>
> TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET
> LAG CONSUMER-ID HOST
> CLIENT-ID
> test_7 0 114700 114744
> 44 consumer-1-47b3f36-f897-460d-b178-d03f7f11a749 /172.17.0.1
> consumer-1
> test_7 1 245 256
> 11 consumer-1-47b3f36-f897-460d-b178-d03f7f11a749 /172.17.0.1
> consumer-1
> test_7 2 980 1024
> 44 consumer-1-47b3f36-f897-460d-b178-d03f7f11a749 /172.17.0.1
> consumer-1
> test_7 3 1470 1536
> 66 consumer-1-47b3f36-f897-460d-b178-d03f7f11a749 /172.17.0.1
> consumer-1
> test_7 4 245 256
> 11 consumer-1-47b3f36-f897-460d-b178-d03f7f11a749 /172.17.0.1
> consumer-1
> test_7 5 1225 1280
> 55 consumer-1-47b3f36-f897-460d-b178-d03f7f11a749 /172.17.0.1
> consumer-1
> test_7 6 490 512
> 22 consumer-1-47b3f36-f897-460d-b178-d03f7f11a749 /172.17.0.1
> consumer-1
> test_7 7 1715 1792
> 77 consumer-1-47b3f36-f897-460d-b178-d03f7f11a749 /172.17.0.1
> consumer-1
> test_7 8 1715 1792
> 77 consumer-1-47b3f36-f897-460d-b178-d03f7f11a749 /172.17.0.1
> consumer-1
> test_7 9 735 768
> 33 consumer-1-47b3f36-f897-460d-b178-d03f7f11a749 /172.17.0.1
> consumer-1
>
>