[jira] [Commented] (KAFKA-1585) Client: Infinite "conflict in /consumers/"

["Joe Stein (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-1585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092050#comment-14092050 ] 

Joe Stein commented on KAFKA-1585:
----------------------------------

FWIW there were a lot of bug fixes released in the Zookeeper 3.4.6 http://zookeeper.apache.org/doc/r3.4.6/releasenotes.html from 3.4.5 version.

You could be hitting ZOOKEEPER-1382 which was fixed in the 3.4.6 release

Current Kafka 0.8.1.1 zookeeper recommend https://kafka.apache.org/documentation.html#zk though folks are using 3.4.6 in production and that should be Zookeeper version for 0.8.2

In regards to your logs, before this happened it looks like you had errors and then a reconnect and consumer shutdown

Line 132356: 18:31:38,948 [7-cloudera:2181] INFO  kafka.utils.Logging$class - [Q_dev-1407608193903-1cb30b18], Q_dev-1407608193903-1cb30b18-0 attempting to claim partition 0
Line 132357: 18:31:38,975 [26-d7f0e66a-0-0] ERROR kafka.utils.Logging$class - [ConsumerFetcherThread-Q_dev-1407608195226-d7f0e66a-0-0], Current offset 15 for partition [gk.q.event,0] out of range; reset offset to 0
Line 132358: 18:31:38,980 [62-1d81f64b-0-0] ERROR kafka.utils.Logging$class - [ConsumerFetcherThread-Q_dev-1407608193962-1d81f64b-0-0], Current offset 4 for partition [gk.q.mail.api,0] out of range; reset offset to 0
Line 132359: 18:31:38,994 [84-ceea5788-0-0] WARN  kafka.utils.Logging$class - Reconnect due to socket error: null
Line 132360: 18:31:38,995 [84-ceea5788-0-0] INFO  kafka.utils.Logging$class - [ConsumerFetcherThread-dev_dev-1407608194884-ceea5788-0-0], Stopped 
Line 132361: 18:31:38,995 [atcher_executor] INFO  kafka.utils.Logging$class - [ConsumerFetcherThread-dev_dev-1407608194884-ceea5788-0-0], Shutdown completed
Line 132362: 18:31:38,995 [atcher_executor] INFO  kafka.utils.Logging$class - [ConsumerFetcherManager-1407608194890] All connections stopped
Line 132363: 18:31:38,996 [atcher_executor] INFO  kafka.utils.Logging$class - [dev_dev-1407608194884-ceea5788], Cleared all relevant queues for this fetcher
Line 132364: 18:31:38,996 [atcher_executor] INFO  kafka.utils.Logging$class - [dev_dev-1407608194884-ceea5788], Cleared the data chunks in all the consumer message iterators
Line 132365: 18:31:38,996 [atcher_executor] INFO  kafka.utils.Logging$class - [dev_dev-1407608194884-ceea5788], Committing all offsets after clearing the fetcher queues
Line 132366: 18:31:38,996 [atcher_executor] INFO  kafka.utils.Logging$class - [dev_dev-1407608194884-ceea5788], Releasing partition ownership
Line 132367: 18:31:39,005 [7-cloudera:2181] INFO  kafka.utils.Logging$class - conflict in /consumers/Q/owners/gk.q.log/0 data: Q_dev-1407608193903-1cb30b18-0 stored data: Q_dev-1407608205503-9cfb99aa-0

likely what happened is when it reconnected the timeout with zk never occurred and it got stuck there.  Could be the Zk bug, could also be related somewhat to KAFKA-1387 or KAFKA-1451 I will link the JIRAs so when we test 0.8.2 see about reproducing this on a good zk version

To resolve that you can stop the consumer, wait for the zk nodes to expire and start up the consumers again.



> Client: Infinite "conflict in /consumers/"
> ------------------------------------------
>
>                 Key: KAFKA-1585
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1585
>             Project: Kafka
>          Issue Type: Bug
>          Components: consumer
>    Affects Versions: 0.8.1.1
>            Reporter: Artur Denysenko
>            Priority: Critical
>             Fix For: 0.8.2
>
>         Attachments: kafka_consumer_ephemeral_node_extract.zip
>
>
> Periodically we have kafka consumers cycling in "conflict in /consumers/" and "I wrote this conflicted ephemeral node". 
> Please see attached log extract.
> After restarting the process kafka consumers are working perfectly. 
> We are using Zookeeper 3.4.5



--
This message was sent by Atlassian JIRA
(v6.2#6252)