You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Abhimanyu Nagrath (JIRA)" <ji...@apache.org> on 2017/05/12 07:48:04 UTC
[jira] [Updated] (KAFKA-5221) Kafka Consumer Attached to partition
but not consuming messages
[ https://issues.apache.org/jira/browse/KAFKA-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Abhimanyu Nagrath updated KAFKA-5221:
-------------------------------------
Description:
I have a single node Kafka broker and a single zookeeper (3.4.9). I am using new Kafka Consumer APIs. One strange thing I observed is when I am starting multiple Kafka consumers for multiple topics placed in a single group and on hitting ./kafka-consumer-groups.sh this script for the group. Few of the consumers are attached to the group but they do not consume any message.
Below are the stats of group command.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST
topic1 0 288 288 0 consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae /<serverip> consumer-8
topic1 1 283 283 0 consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae /<serverip> consumer-8
topic1 2 279 279 0 consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae /<serverip> consumer-8
topic2 0 - 9 - consumer-1-b0476dc8-099c-4a62-a68c-e9dc9c0a5bed /<serverip> consumer-1
topic2 1 - 2 - consumer-1-b0476dc8-099c-4a62-a68c-e9dc9c0a5bed /<serverip> consumer-1
topic3 0 450 450 0 consumer-3-63c07703-17d0-471b-8c5f-17347699f108 /<serverip> consumer-3
topic4 1 - 54
- consumer-2-94dcc209-8377-45ce-8473-9ab0d85951c4 /<serverip>
topic2 2 441 441 0 consumer-5-bcfffc99-5915-41f4-b3e4-970baa204c14 /<serverip>
So can someone help me that why for topic **topic2** partition 0 **current-offset** is showing - and **lag** is showing - but messages are still there in the server as **LOG-END-OFFSET** is showing 9.
I am having 600 GB Data distributed among 280 topics and each topic is having approx. 50 partitions.
This is happening very frequently and restarting the consumers solves the issue temporarily.
was:
I have a single node Kafka broker and a single zookeeper (3.4.9). I am using new Kafka Consumer APIs. One strange thing I observed is when I am starting multiple Kafka consumers for multiple topics placed in a single group and on hitting ./kafka-consumer-groups.sh this script for the group. Few of the consumers are attached to the group but they do not consume any message.
Below are the stats of group command.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST
topic1 0 288 288 0 consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae /<serverip> consumer-8
topic1 1 283 283 0 consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae /<serverip> consumer-8
topic1 2 279 279 0 consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae /<serverip> consumer-8
topic2 0 - 9 - consumer-1-b0476dc8-099c-4a62-a68c-e9dc9c0a5bed /<serverip> consumer-1
topic2 1 - 2 - consumer-1-b0476dc8-099c-4a62-a68c-e9dc9c0a5bed /<serverip> consumer-1
topic3 0 450 450 0 consumer-3-63c07703-17d0-471b-8c5f-17347699f108 /<serverip> consumer-3
topic4 1 - 54
- consumer-2-94dcc209-8377-45ce-8473-9ab0d85951c4 /<serverip>
topic2 2 441 441 0 consumer-5-bcfffc99-5915-41f4-b3e4-970baa204c14 /<serverip>
So can someone help me that why for topic **topic2** partition 0 **current-offset** is showing - and **lag** is showing - but messages are still there in the server as **LOG-END-OFFSET** is showing 9.
I am having 600 GB Data distributed among 380 topics and each topic is having approx. 100 partitions.
This is happening very frequently and restarting the consumers solves the issue temporarily.
> Kafka Consumer Attached to partition but not consuming messages
> ---------------------------------------------------------------
>
> Key: KAFKA-5221
> URL: https://issues.apache.org/jira/browse/KAFKA-5221
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 0.10.2.0
> Environment: Amazon-Linux AMI
> Hardware Specifications - (8 core, 16 GB RAM , 1 TB HardDisk)
> Reporter: Abhimanyu Nagrath
>
> I have a single node Kafka broker and a single zookeeper (3.4.9). I am using new Kafka Consumer APIs. One strange thing I observed is when I am starting multiple Kafka consumers for multiple topics placed in a single group and on hitting ./kafka-consumer-groups.sh this script for the group. Few of the consumers are attached to the group but they do not consume any message.
> Below are the stats of group command.
> TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST
> topic1 0 288 288 0 consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae /<serverip> consumer-8
>
> topic1 1 283 283 0 consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae /<serverip> consumer-8
>
> topic1 2 279 279 0 consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae /<serverip> consumer-8
>
> topic2 0 - 9 - consumer-1-b0476dc8-099c-4a62-a68c-e9dc9c0a5bed /<serverip> consumer-1
>
> topic2 1 - 2 - consumer-1-b0476dc8-099c-4a62-a68c-e9dc9c0a5bed /<serverip> consumer-1
>
> topic3 0 450 450 0 consumer-3-63c07703-17d0-471b-8c5f-17347699f108 /<serverip> consumer-3
>
> topic4 1 - 54
> - consumer-2-94dcc209-8377-45ce-8473-9ab0d85951c4 /<serverip>
>
> topic2 2 441 441 0 consumer-5-bcfffc99-5915-41f4-b3e4-970baa204c14 /<serverip>
> So can someone help me that why for topic **topic2** partition 0 **current-offset** is showing - and **lag** is showing - but messages are still there in the server as **LOG-END-OFFSET** is showing 9.
> I am having 600 GB Data distributed among 280 topics and each topic is having approx. 50 partitions.
> This is happening very frequently and restarting the consumers solves the issue temporarily.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)