Kafka Not sending few partitions messages to consumers

[Gnanasoundari Soundarajan <gn...@man-es.com>]

Hi all,

Kafka is not sending messages to consumer though the consumer is still active and subscribed to the partition. Please help us to know why this behavior is happening in kafka.


Infrastructure:
Kafka is running in cluster mode with 3 brokers and 3 zookeeper instances.
Kafka broker is running in t2.xlarge AWS instance.
3 Zookeeper instances is running in single t2.xlarge AWS instance.
Kafka Version: 1.1.0
How to reproduce:
*        Continuously send 50,000 records/sec where each record size is about 200 bytes approx. for 24hour duration.
*        We have 25 topics with 4 partitions each. All 25 topics are consumed by 4 consumers with auto assignment of partition by kafka.
*        Kafka clears all records in partitions except 2 partitions. For two partitions , only 27 messages are delivered. Remaining messages has not delivered to consumer.
*        During the run kafka brokers cpu utilization is 5%
Problematic partitions details:
TOPIC

PARTITION

CURRENT-OFFSET

LOG-END-OFFSET

LAG

CONSUMER-ID

HOST

CLIENT-ID

topic-di14

3

27

3919844

3919817

consumer-7-dfaf4c6f-5b71-455e-ab12-9984446a9bf4

/10.10.9.22

consumer-7


topic-di9

0

27

12231117

12231090

consumer-22-7922babd-8f8b-49db-a111-df523962cca8

/10.10.9.22

consumer-22

I am not able to see any error in kafka as well as my consumer application.  Consumer is active when i ran ./bin/kafka-consumer-groups.sh command in kafka.

Kafka Consumer settings:
enable.auto.commit = true
session.timeout.ms = 90000
heartbeat.interval.ms = 30000
max.poll.records = 1000
poll-interval = 60ms
poll-timeout = 60ms



Regards,
Gnana