[jira] [Created] (SPARK-5505) ConsumerRebalanceFailedException from Kafka consumer

["Greg Temchenko (JIRA)" <ji...@apache.org>]

Greg Temchenko created SPARK-5505:
-------------------------------------

             Summary: ConsumerRebalanceFailedException from Kafka consumer
                 Key: SPARK-5505
                 URL: https://issues.apache.org/jira/browse/SPARK-5505
             Project: Spark
          Issue Type: Bug
          Components: Streaming
    Affects Versions: 1.2.0
         Environment: CentOS6 / Linux 2.6.32-358.2.1.el6.x86_64
java version "1.7.0_21"
Scala compiler version 2.9.3
2 cores Intel(R) Xeon(R) CPU E5620  @ 2.40GHz / 16G RAM
VMWare VM.
            Reporter: Greg Temchenko
            Priority: Critical


>From time to time Spark streaming produces a ConsumerRebalanceFailedException and stops receiving messages. After that all consequential RDDs are empty.

{code}
15/01/30 18:18:36 ERROR consumer.ZookeeperConsumerConnector: [terran_vmname-1422670149779-243b4e10], error during syncedRebalance
kafka.common.ConsumerRebalanceFailedException: terran_vmname-1422670149779-243b4e10 can't rebalance after 4 retries
	at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:432)
	at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(ZookeeperConsumerConnector.scala:355)
{code}

The problem is also described in the mailing list: http://apache-spark-user-list.1001560.n3.nabble.com/Error-when-Spark-streaming-consumes-from-Kafka-td19570.html

As I understand it's a critical blocker for kafka-spark streaming production use.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org