Consumer stops reading messages after some time

[Alexey Sverdelov <al...@googlemail.com>]

Hi,

I'm facing an issue with high level kafka consumer (0.8.2.0) - after
consuming some amount of data one of our consumers stops. After restart it
consumes some messages and stops again with no error/exception or warning.

After some investigation I found that the "ConsumerFetcherThread" for my
main topic gets stuck: it makes no progress and writes no log messages.


Consumer for my second topic seems to be ok - I can see some FetchRequests.

Any thoughts?

Alexey

Re: Consumer stops reading messages after some time

[Alexey Sverdelov <al...@googlemail.com>]

Answering my own message, the problem with consumer was this exception:

==
ERROR c.u.u.e.impl.kafka.KafkaConsumer  - Error consuming message stream:
 # kafka.message.InvalidMessageException: Message is corrupt (stored crc =
3801080313, computed crc = 2728178222)
 # \x09at kafka.message.Message.ensureValid(Message.scala:166)
~[org.apache.kafka.kafka_2.11-0.8.2.0.jar:na]
==

Any ideas how can I simple ignore such messages at all?

Thanks.

On Thu, Oct 1, 2015 at 1:05 PM, Alexey Sverdelov <
alexey.sverdelov@googlemail.com> wrote:

> Hi,
>
> I'm facing an issue with high level kafka consumer (0.8.2.0) - after
> consuming some amount of data one of our consumers stops. After restart it
> consumes some messages and stops again with no error/exception or warning.
>
> After some investigation I found that the "ConsumerFetcherThread" for my
> main topic gets stuck: it makes no progress and writes no log messages.
>
>
> Consumer for my second topic seems to be ok - I can see some FetchRequests.
>
> Any thoughts?
>
> Alexey
>