[jira] [Created] (KAFKA-5221) Kafka Consumer Attached to partition but not consuming messages

["Abhimanyu Nagrath (JIRA)" <ji...@apache.org>]

Abhimanyu Nagrath created KAFKA-5221:
----------------------------------------

             Summary: Kafka Consumer Attached to partition but not consuming messages
                 Key: KAFKA-5221
                 URL: https://issues.apache.org/jira/browse/KAFKA-5221
             Project: Kafka
          Issue Type: Bug
          Components: consumer
    Affects Versions: 0.10.2.0
         Environment: Amazon-Linux AMI 
Hardware Specifications - (8 core, 16 GB RAM , 1 TB HardDisk)
            Reporter: Abhimanyu Nagrath


I have a single node Kafka broker and a single zookeeper (3.4.9). I am using new Kafka Consumer APIs. One strange thing I observed is when I am starting multiple Kafka consumers for multiple topics placed in a single group and on hitting ./kafka-consumer-groups.sh this script for the group. Few of the consumers are attached to the group but they do not consume any message. 
Below are the stats of group command.

        TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID                                 HOST  
    topic1 0          288             288             0          consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae   /<serverip>          consumer-8 
    
    topic1 1          283             283             0         consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae   /<serverip>          consumer-8 
    
    topic1 2          279             279             0         consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae   /<serverip>          consumer-8 
    
    topic2 0          -               9               -         consumer-1-b0476dc8-099c-4a62-a68c-e9dc9c0a5bed   /<serverip>          consumer-1 
    
    topic2 1          -               2               -         consumer-1-b0476dc8-099c-4a62-a68c-e9dc9c0a5bed   /<serverip>          consumer-1 
    
    topic3 0          450             450             0         consumer-3-63c07703-17d0-471b-8c5f-17347699f108   /<serverip>          consumer-3 
    
    topic4                1          -               54        
        -        consumer-2-94dcc209-8377-45ce-8473-9ab0d85951c4   /<serverip>      
    


    topic2 2          441             441             0          consumer-5-bcfffc99-5915-41f4-b3e4-970baa204c14   /<serverip>

So can someone help me that why for topic **topic2** partition 0 **current-offset** is showing - and **lag** is showing - but messages are still there in the server as  **LOG-END-OFFSET** is showing 9.

I am having 600 GB Data distributed among 380 topics and each topic is having approx. 100 partitions.

This is happening very frequently and restarting the consumers solves the issue temporarily.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)