You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "huxihx (JIRA)" <ji...@apache.org> on 2017/06/03 06:20:04 UTC
[jira] [Commented] (KAFKA-5287) Messages getting repeated in kafka
[ https://issues.apache.org/jira/browse/KAFKA-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16035864#comment-16035864 ]
huxihx commented on KAFKA-5287:
-------------------------------
Do you ensure you check status for the same group?
> Messages getting repeated in kafka
> ----------------------------------
>
> Key: KAFKA-5287
> URL: https://issues.apache.org/jira/browse/KAFKA-5287
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 0.10.2.0
> Environment: Hardware specification(8 Cores , 16 GB RAM,1 TB Harddisk)
> Reporter: Abhimanyu Nagrath
>
> I have a topic with 200 partition in which messages contains the total of 3 Million messages. It took 5 days to completely process all the messages and as soon as message got processed i.e. Kafka-consumer-groups.sh showed 0 lag in all the partition of the topic I stopped the consumer .but after 6 hrs again it was showing the lag of 2 million message which I found that were duplicate messages. This thing is happening very frequently. My offsets are stored on Kafka broker itself.
> My server configuration is:
> broker.id=1
> delete.topic.enable=true
> #listeners=PLAINTEXT://:9092
> #advertised.listeners=PLAINTEXT://your.host.name:9092
> num.network.threads=3
> num.io.threads=8
> socket.send.buffer.bytes=102400
> socket.receive.buffer.bytes=102400
> socket.request.max.bytes=104857600
> log.dirs=/kafka/data/logs
> num.partitions=1
> num.recovery.threads.per.data.dir=5
> log.flush.interval.messages=10000
> #log.flush.interval.ms=1000
> log.retention.hours=480
> log.retention.bytes=1073741824
> log.segment.bytes=1073741824
> log.retention.check.interval.ms=300000
> zookeeper.connect=<zkIp>:2181
> zookeeper.connection.timeout.ms=6000
> Is there in the configuration that I am missing?
> Any help is appreciated
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)