Fetcher threads blocked, no messages read from stream

[Bryan Baugher <bj...@gmail.com>]

Hi everyone,

I'm currently experiencing a situation that looks very similar to this[1].
I'm running Kafka 0.8.2.1 and we use it with Storm but don't use Storm's
kafka spout since it doesn't quite work for our needs. I've been able to
reproduce the issue locally by creating a storm topology with 1 worker
(jvm) which has 8 spouts (threads) and each spout has its own
ConsumerConnector and stream. The issue occurs when downstream processing
continuously fails in which our spouts shut down their connector, waits 5s
and then creates a new connector in order to replay the messages. After a
flurry of error messages usually (syncedRebalance exceptions) everything
settles down and our spouts continue to try and read new messages from the
stream but nothing comes through. Looking at the thread dump shows all
consumer fetcher threads parked unable to add more to their queue which
could mean the queues are full,

   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000007bcc62e88> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350)
at kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:60)
at
kafka.consumer.ConsumerFetcherThread.processPartitionData(ConsumerFetcherThread.scala:49)

I realize this setup isn't the best and I would love to move to storm's
kafka spout which looks like it avoids some of these problems but that may
be a ways off. Any ideas or things to try?

[1] -
https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CB0QFjAAahUKEwjL5Z7T9bXIAhUY7WMKHQ35BJU&url=http%3A%2F%2Fmail-archives.apache.org%2Fmod_mbox%2Fkafka-users%2F201307.mbox%2F%253CCAFbh0Q3%3DugxWp-L%2BGaoycNbdW3WyCbNU4CrZ3qDWbP8uXFADRA%40mail.gmail.com%253E&usg=AFQjCNHF6myACUd50GrAqdgpwZflOsjYKg&sig2=61k2oqI9havLlhHuhdl3hw