Skipping assignment for topic * since no metadata is available

[Patrick Kaufmann <pa...@innoq.com>]

Hello

Recently we’ve run into a problem when starting our application for the first time.

At the moment all our topics are auto-created. Now, at the first start there are no topics, so naturally some consumers try to connect to topics which don’t exist.
Those consumers now fail quite consistently with the following error:
Skipping assignment for topic <topic-name> since no metadata is available

This then leads to the consumer not consuming any messages on that topic.

This only happens when the topic does not exist. When we restart the failing consumer it then can connect correctly to the topic and consume it.
How can this error be prevented?

Best regards

Patrick

Re: Skipping assignment for topic * since no metadata is available

[Jaikiran Pai <ja...@gmail.com>]

On Thursday 09 June 2016 08:00 PM, Patrick Kaufmann wrote:
> Hello
>
> Recently we\u2019ve run into a problem when starting our application for the first time.
>
> At the moment all our topics are auto-created. Now, at the first start there are no topics, so naturally some consumers try to connect to topics which don\u2019t exist.
> Those consumers now fail quite consistently with the following error:
> Skipping assignment for topic <topic-name> since no metadata is available
>
> This then leads to the consumer not consuming any messages on that topic.
Take a look at "auto.offset.reset" property that you can pass to the 
consumers which is meant to deal with cases where the offsets aren't 
known/available for the topic. Depending on the version of Kafka you are 
using, the value of the property will be different, so check the 
relevant version documentation and see what value makes sense in your 
application.

-Jaikiran

Re: Skipping assignment for topic * since no metadata is available

[Kaufman Ng <ka...@confluent.io>]

For consumers connecting to non-existent topics there's no metadata
available initially, that's the reason why you are seeing this message.

There are multiple ways to work around this:
- create the topics manually (e.g. kafka-topics.sh --create --topic <topic>
...)
- start your producer *before* your consumers so that topics will be
populated with messages (and therefore metadata).  By the time consumers
start topics will exist already with messages and metadata
- do nothing and wait for consumer group to rebalance itself after 5
minutes (default).

Hope this helps.

On Thu, Jun 9, 2016 at 10:30 AM, Patrick Kaufmann <
patrick.kaufmann@innoq.com> wrote:

> Hello
>
> Recently we’ve run into a problem when starting our application for the
> first time.
>
> At the moment all our topics are auto-created. Now, at the first start
> there are no topics, so naturally some consumers try to connect to topics
> which don’t exist.
> Those consumers now fail quite consistently with the following error:
> Skipping assignment for topic <topic-name> since no metadata is available
>
> This then leads to the consumer not consuming any messages on that topic.
>
> This only happens when the topic does not exist. When we restart the
> failing consumer it then can connect correctly to the topic and consume it.
> How can this error be prevented?
>
> Best regards
>
> Patrick




-- 
Kaufman Ng | Solutions Architect | Confluent
kaufman@confluent.io | +1 646 961 8063