[jira] [Created] (KAFKA-5287) Messages getting repeated in kafka

["Abhimanyu Nagrath (JIRA)" <ji...@apache.org>]

Abhimanyu Nagrath created KAFKA-5287:
----------------------------------------

             Summary: Messages getting repeated in kafka
                 Key: KAFKA-5287
                 URL: https://issues.apache.org/jira/browse/KAFKA-5287
             Project: Kafka
          Issue Type: Bug
          Components: consumer
    Affects Versions: 0.10.2.0
         Environment: Hardware specification(8 Cores , 16 GB RAM,1 TB Harddisk)
            Reporter: Abhimanyu Nagrath


I have a topic with 200 partition in which messages contains the total of 3 Million messages. It took 5 days to completely process all the messages and as soon as message got processed i.e. Kafka-consumer-groups.sh showed 0 lag in all the partition of the topic I stopped the consumer .but after 6 hrs again it was showing the lag of 2 million message which I found that were duplicate messages. This thing is happening very frequently. My offsets are stored on Kafka broker itself. 
My server configuration is:

    broker.id=1
    delete.topic.enable=true
    #listeners=PLAINTEXT://:9092
    #advertised.listeners=PLAINTEXT://your.host.name:9092
    num.network.threads=3
    num.io.threads=8
    socket.send.buffer.bytes=102400
    socket.receive.buffer.bytes=102400
    socket.request.max.bytes=104857600
    log.dirs=/kafka/data/logs
    num.partitions=1
    num.recovery.threads.per.data.dir=5
    log.flush.interval.messages=10000
    #log.flush.interval.ms=1000
    log.retention.hours=480
    log.retention.bytes=1073741824
    log.segment.bytes=1073741824
    log.retention.check.interval.ms=300000
    zookeeper.connect=<zkIp>:2181
    zookeeper.connection.timeout.ms=6000

Is there in the configuration that I am missing? 
Any help is appreciated 





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)