[jira] [Commented] (SPARK-2383) With auto.offset.reset, KafkaReceiver potentially deletes Consumer nodes from Zookeeper

["Tathagata Das (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14224571#comment-14224571 ] 

Tathagata Das commented on SPARK-2383:
--------------------------------------

This issue has been resolved in Spark 1.2.0. I am marking this JIRA as duplicate of that SPARK-2492 and closing this.

> With auto.offset.reset, KafkaReceiver potentially deletes Consumer nodes from Zookeeper
> ---------------------------------------------------------------------------------------
>
>                 Key: SPARK-2383
>                 URL: https://issues.apache.org/jira/browse/SPARK-2383
>             Project: Spark
>          Issue Type: Bug
>          Components: Streaming
>            Reporter: Tobias Pfeiffer
>
> When auto.offset.reset is set in the Kafka configuration, then {{KafkaReceiver}}'s {{tryZookeeperConsumerGroupCleanup()}} will delete the whole /consume/<groupId> tree in Zookeeper before creating consumer nodes. If there are already consumer nodes present (this may happen when multiple KafkaReceivers in the same consumer group are launched), they are deleted as well, leading to subsequent NoNode exceptions, for example, on rebalance.
> There should be a check before the delete like {{if (zk.countChildren(dir + "/ids") == 0) ...}} (ideally in an atomic way) in order to prevent deleting existing consumer nodes.
> (Also note that the behavior of auto.offset.reset as realized by Spark's Kafka receiver differs from the behavior defined in Kafka's documentation.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org