[jira] [Comment Edited] (KAFKA-4373) Kafka Consumer API jumping offsets

["Conor Hughes (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-4373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677107#comment-15677107 ] 

Conor Hughes edited comment on KAFKA-4373 at 11/18/16 4:42 PM:
---------------------------------------------------------------

I have experienced a similar issue using Kafka 0.10.1.0 and Kafka Streams 0.10.1.0.

I have 2 topologies running:

Topology number 1: Reads messages from Kafka topic, processes them and outputs them to another Kafka topic.
Topology number 2: Reads messages from topology number 1's output topic and inserts them into a database.

I am tracking the number of messages topology number 2 has read in both inside and after the consumer's deserialiser as well as the topology's offsets in Kafka.
These counts will incrementally match for a number of minutes then suddenly the offsets will jump by up to 40,000 and but the topology's counts remain incrementing normally as if this jump never happen so messages are being lost.


was (Author: thatguyhughesy):
I have experienced a similar issue using Kafka 0.10.1.0 and Kafka Stream 0.10.1.0.

I have 2 topologies running:

Topology number 1: Reads messages from Kafka topic, processes them and outputs them to another Kafka topic.
Topology number 2: Reads messages from topology number 1's output topic and inserts them into a database.

I am tracking the number of messages topology number 2 has read in both inside and after the consumer's deserialiser as well as the topology's offsets in Kafka.
These counts will incrementally match for a number of minutes then suddenly the offsets will jump by up to 40,000 and but the topology's counts remain incrementing normally as if this jump never happen so messages are being lost.

> Kafka Consumer API jumping offsets
> ----------------------------------
>
>                 Key: KAFKA-4373
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4373
>             Project: Kafka
>          Issue Type: Bug
>          Components: consumer
>            Reporter: Srinivasan Venkatraman
>
> Hi,
> I am using Kafka Version 0.10.0.1 and java consumer API to consume messages from a topic. We are using a single node kafka and zookeeper. It is sometimes observed that the consumer is losing a bulk of message. We are unable to find the exact reason to replicate the issue.
> The scenario is:
> Consumer polls the topic.
> Fetches the messages and gives it to a thread pool to handle the message.
> Waits for the threads to return and then commits the offsets.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)