[jira] [Updated] (KAFKA-1200) inconsistent log levels when consumed offset is reset

["Joe Stein (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-1200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joe Stein updated KAFKA-1200:
-----------------------------

    Fix Version/s: 0.8.1

the consumer should log the error too, yeah.  should be able to patch this too if no one else is already working on it

> inconsistent log levels when consumed offset is reset
> -----------------------------------------------------
>
>                 Key: KAFKA-1200
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1200
>             Project: Kafka
>          Issue Type: Bug
>          Components: consumer
>    Affects Versions: 0.8.0
>            Reporter: Jason Rosenberg
>            Assignee: Neha Narkhede
>             Fix For: 0.8.1
>
>
> I've recently been dealing with the issue where my consumer falls behind and essentially loses data when the broker deletes data, due to it's retention policy.
> On the broker, this is logged as an ERROR:
> 2013-12-23 05:02:08,456 ERROR [kafka-request-handler-2] server.KafkaApis - [KafkaApi-45] Error when processing fetch request for partition [mytopic,0] offset 204243601 from consumer with correlation id 130341
> kafka.common.OffsetOutOfRangeException: Request for offset 204243601 but we only have log segments in the range 204343397 to 207423640.
> But on the consumer, this same event is logged as a WARN:
> 2013-12-23 05:02:08,797  WARN [ConsumerFetcherThread-myconsumergroup-1387353494862-7aa0c61d-0-45] consumer.ConsumerFetcherThread - [ConsumerFetcherThread-myconsumergroup-1387353494862-7aa0c61d-0-45], Current offset 204243601 for partition [mytopic,0] out of range; reset offset to 204343397
> It seems this should also be an ERROR condition (it would seem the consumer would care more about this than the broker, at least!).
> Also, sometimes (but not always) there is also this log message on the consumer, which does log as an ERROR (I'm not sure why this log line doesn't always appear after the above WARN?):
> 2014-01-08 02:31:47,681 ERROR [myconsumerthread-0]
> consumer.ConsumerIterator - consumed offset: 16163904970 doesn't match
> fetch offset: 16175326044 for mytopic:0: fetched offset = 16175330598:
> consumed offset = 16163904970;
>  Consumer may lose data
> In this message, there is the "Consumer may lose data" message, which makes sense.  Seems the fetcher thread above should also log something like that, and be an ERROR.
> This would allow for more consistent alerting, in this case.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)