Kafka startup, no broker partitions consumed by consumer thread

[Viktor Kolodrevskiy <vi...@gmail.com>]

On our current project we want to have a single shell script that:
first starts kafka with zookeeper and second strarts tomcat server
with our webapp.
We run kafka and tomcat on single machine, use kafka 0.8
We set num.partitions=4 in server.properties.
In webapp we start 4 threads that do
kafka.consumer.Consumer.createJavaConsumerConnector to create consumer
connectors.

The logic of script is following:
1. run zookeeper-server-start.sh and wait until zk process
appears(with linux pgrep command).
2. run kafka-server-start.sh and wait until kafka process appears.
3. sleep a fixed timeout(say 10 seconds) to let kafka completely initialize
4. start tomcat

If timeout in step#3 is smaller we get following warning and consumers
do not receive messages:
WARN  [kafka.consumer.ZookeeperConsumerConnector]
[test-1381405301679-580f9706], No broker partitions consumed by
consumer thread test-1381405301679-580f9706-3 for topic test-topic

Is there a way to avoid this hardcoded timeout? How to check that
partitions are initialized?


Thanks,
Viktor

Re: Kafka startup, no broker partitions consumed by consumer thread

[Viktor Kolodrevskiy <vi...@gmail.com>]

Jun,

Thank you! That helped.


Thanks,
Viktor

Re: Kafka startup, no broker partitions consumed by consumer thread

[Jun Rao <ju...@gmail.com>]

Could you try the latest code in the 0.8 branch? We recently fixed a
consumer side issue (KAFKA-1030) that seems related.

Thanks,

Jun

On Thu, Oct 10, 2013 at 6:36 AM, Viktor Kolodrevskiy <
viktor.kolodrevskiy@gmail.com> wrote:

> On our current project we want to have a single shell script that:
> first starts kafka with zookeeper and second strarts tomcat server
> with our webapp.
> We run kafka and tomcat on single machine, use kafka 0.8
> We set num.partitions=4 in server.properties.
> In webapp we start 4 threads that do
> kafka.consumer.Consumer.createJavaConsumerConnector to create consumer
> connectors.
>
> The logic of script is following:
> 1. run zookeeper-server-start.sh and wait until zk process
> appears(with linux pgrep command).
> 2. run kafka-server-start.sh and wait until kafka process appears.
> 3. sleep a fixed timeout(say 10 seconds) to let kafka completely initialize
> 4. start tomcat
>
> If timeout in step#3 is smaller we get following warning and consumers
> do not receive messages:
> WARN  [kafka.consumer.ZookeeperConsumerConnector]
> [test-1381405301679-580f9706], No broker partitions consumed by
> consumer thread test-1381405301679-580f9706-3 for topic test-topic
>
> Is there a way to avoid this hardcoded timeout? How to check that
> partitions are initialized?
>
>
> Thanks,
> Viktor
>