You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "huxi (JIRA)" <ji...@apache.org> on 2017/05/12 07:38:04 UTC
[jira] [Commented] (KAFKA-5221) Kafka Consumer Attached to
partition but not consuming messages
[ https://issues.apache.org/jira/browse/KAFKA-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16007746#comment-16007746 ]
huxi commented on KAFKA-5221:
-----------------------------
Do you mean all these 38,000 partitions reside on a single broker machine? Is it possible to hit any resource limit?
> Kafka Consumer Attached to partition but not consuming messages
> ---------------------------------------------------------------
>
> Key: KAFKA-5221
> URL: https://issues.apache.org/jira/browse/KAFKA-5221
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 0.10.2.0
> Environment: Amazon-Linux AMI
> Hardware Specifications - (8 core, 16 GB RAM , 1 TB HardDisk)
> Reporter: Abhimanyu Nagrath
>
> I have a single node Kafka broker and a single zookeeper (3.4.9). I am using new Kafka Consumer APIs. One strange thing I observed is when I am starting multiple Kafka consumers for multiple topics placed in a single group and on hitting ./kafka-consumer-groups.sh this script for the group. Few of the consumers are attached to the group but they do not consume any message.
> Below are the stats of group command.
> TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST
> topic1 0 288 288 0 consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae /<serverip> consumer-8
>
> topic1 1 283 283 0 consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae /<serverip> consumer-8
>
> topic1 2 279 279 0 consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae /<serverip> consumer-8
>
> topic2 0 - 9 - consumer-1-b0476dc8-099c-4a62-a68c-e9dc9c0a5bed /<serverip> consumer-1
>
> topic2 1 - 2 - consumer-1-b0476dc8-099c-4a62-a68c-e9dc9c0a5bed /<serverip> consumer-1
>
> topic3 0 450 450 0 consumer-3-63c07703-17d0-471b-8c5f-17347699f108 /<serverip> consumer-3
>
> topic4 1 - 54
> - consumer-2-94dcc209-8377-45ce-8473-9ab0d85951c4 /<serverip>
>
> topic2 2 441 441 0 consumer-5-bcfffc99-5915-41f4-b3e4-970baa204c14 /<serverip>
> So can someone help me that why for topic **topic2** partition 0 **current-offset** is showing - and **lag** is showing - but messages are still there in the server as **LOG-END-OFFSET** is showing 9.
> I am having 600 GB Data distributed among 380 topics and each topic is having approx. 100 partitions.
> This is happening very frequently and restarting the consumers solves the issue temporarily.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)