You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Ali Akhtar <al...@gmail.com> on 2017/07/07 20:46:31 UTC
Kafka behavior when consuming a topic which doesn't exist?
Sometimes I see warnings in my logs if i create a consumer for a topic
which doesn't exist. Such as:
org.apache.kafka.clients.NetworkClient - Error while fetching metadata
with correlation id 1 : {example_topic=LEADER_NOT_AVAILABLE}
If later messages are posted to that topic (which will create it), will my
consumer receive those messages?
Re[2]: Kafka behavior when consuming a topic which doesn't exist?
Posted by "M. Manna" <ma...@gmail.com>.
Please ensure that you set the desired minimum partition and replication
settings in the server.properties file. These are the ones which will be
used for auto creation of topics.
KR,
------ Original Message ------
From: "Ali Akhtar" <al...@gmail.com>
To: users@kafka.apache.org
Sent: 7/8/2017 4:27:55 PM
Subject: Re: Kafka behavior when consuming a topic which doesn't exist?
>Oh gotcha, thanks. So a topic will be created if topic creation is
>enabled.
>
>On Sat, Jul 8, 2017 at 8:14 PM, M. Manna <ma...@gmail.com> wrote:
>
>>Please check my previous email.
>>
>>On Sat, 8 Jul 2017 at 2:32 am, Ali Akhtar <al...@gmail.com>
>>wrote:
>>
>> > What happens if auto creation is enabled but the topic doesn't
>>exist.
>> > Consumers subscribe to that topic which doesn't exist. Then messages
>>are
>> > posted to that topic.
>> >
>> > Will the consumer receive those messages in this scenario?
>> >
>> >
>> >
>> > On 8 Jul 2017 4:38 a.m., "M. Manna" <ma...@gmail.com> wrote:
>> >
>> > That depends.
>> >
>> > If auto creation of non-existent topic enabled (check docs), then it
>>will
>> > simple use the minimum partiotion and replication settings defined
>>in
>> > broker config to create a topic. If auto creation is disabled, your
>> > consumer group won't do anything.
>> >
>> > With auto creation enable - It's the same if you publish to a topic
>>that
>> > doesn't yet exist. But that means topic will get created first, and
>>then
>> > the messages are sent on the bus. If you subscribe to that topic
>>later
>>you
>> > will get messages. Check below:
>> >
>> > http://grokbase.com/t/kafka/users/1648kbr04c/subscribe-on-
>> > a-topic-that-does-not-exist
>> >
>> >
>> >
>> > On 7 Jul 2017 9:46 pm, "Ali Akhtar" <al...@gmail.com> wrote:
>> >
>> > > Sometimes I see warnings in my logs if i create a consumer for a
>>topic
>> > > which doesn't exist. Such as:
>> > >
>> > > org.apache.kafka.clients.NetworkClient - Error while fetching
>>metadata
>> > > with correlation id 1 : {example_topic=LEADER_NOT_AVAILABLE}
>> > >
>> > > If later messages are posted to that topic (which will create it),
>>will
>> > my
>> > > consumer receive those messages?
>> > >
>> >
>>
Re: Kafka behavior when consuming a topic which doesn't exist?
Posted by Ali Akhtar <al...@gmail.com>.
Oh gotcha, thanks. So a topic will be created if topic creation is enabled.
On Sat, Jul 8, 2017 at 8:14 PM, M. Manna <ma...@gmail.com> wrote:
> Please check my previous email.
>
> On Sat, 8 Jul 2017 at 2:32 am, Ali Akhtar <al...@gmail.com> wrote:
>
> > What happens if auto creation is enabled but the topic doesn't exist.
> > Consumers subscribe to that topic which doesn't exist. Then messages are
> > posted to that topic.
> >
> > Will the consumer receive those messages in this scenario?
> >
> >
> >
> > On 8 Jul 2017 4:38 a.m., "M. Manna" <ma...@gmail.com> wrote:
> >
> > That depends.
> >
> > If auto creation of non-existent topic enabled (check docs), then it will
> > simple use the minimum partiotion and replication settings defined in
> > broker config to create a topic. If auto creation is disabled, your
> > consumer group won't do anything.
> >
> > With auto creation enable - It's the same if you publish to a topic that
> > doesn't yet exist. But that means topic will get created first, and then
> > the messages are sent on the bus. If you subscribe to that topic later
> you
> > will get messages. Check below:
> >
> > http://grokbase.com/t/kafka/users/1648kbr04c/subscribe-on-
> > a-topic-that-does-not-exist
> >
> >
> >
> > On 7 Jul 2017 9:46 pm, "Ali Akhtar" <al...@gmail.com> wrote:
> >
> > > Sometimes I see warnings in my logs if i create a consumer for a topic
> > > which doesn't exist. Such as:
> > >
> > > org.apache.kafka.clients.NetworkClient - Error while fetching
> metadata
> > > with correlation id 1 : {example_topic=LEADER_NOT_AVAILABLE}
> > >
> > > If later messages are posted to that topic (which will create it), will
> > my
> > > consumer receive those messages?
> > >
> >
>
Re: Kafka behavior when consuming a topic which doesn't exist?
Posted by "M. Manna" <ma...@gmail.com>.
Please check my previous email.
On Sat, 8 Jul 2017 at 2:32 am, Ali Akhtar <al...@gmail.com> wrote:
> What happens if auto creation is enabled but the topic doesn't exist.
> Consumers subscribe to that topic which doesn't exist. Then messages are
> posted to that topic.
>
> Will the consumer receive those messages in this scenario?
>
>
>
> On 8 Jul 2017 4:38 a.m., "M. Manna" <ma...@gmail.com> wrote:
>
> That depends.
>
> If auto creation of non-existent topic enabled (check docs), then it will
> simple use the minimum partiotion and replication settings defined in
> broker config to create a topic. If auto creation is disabled, your
> consumer group won't do anything.
>
> With auto creation enable - It's the same if you publish to a topic that
> doesn't yet exist. But that means topic will get created first, and then
> the messages are sent on the bus. If you subscribe to that topic later you
> will get messages. Check below:
>
> http://grokbase.com/t/kafka/users/1648kbr04c/subscribe-on-
> a-topic-that-does-not-exist
>
>
>
> On 7 Jul 2017 9:46 pm, "Ali Akhtar" <al...@gmail.com> wrote:
>
> > Sometimes I see warnings in my logs if i create a consumer for a topic
> > which doesn't exist. Such as:
> >
> > org.apache.kafka.clients.NetworkClient - Error while fetching metadata
> > with correlation id 1 : {example_topic=LEADER_NOT_AVAILABLE}
> >
> > If later messages are posted to that topic (which will create it), will
> my
> > consumer receive those messages?
> >
>
Re: Kafka behavior when consuming a topic which doesn't exist?
Posted by Ali Akhtar <al...@gmail.com>.
What happens if auto creation is enabled but the topic doesn't exist.
Consumers subscribe to that topic which doesn't exist. Then messages are
posted to that topic.
Will the consumer receive those messages in this scenario?
On 8 Jul 2017 4:38 a.m., "M. Manna" <ma...@gmail.com> wrote:
That depends.
If auto creation of non-existent topic enabled (check docs), then it will
simple use the minimum partiotion and replication settings defined in
broker config to create a topic. If auto creation is disabled, your
consumer group won't do anything.
With auto creation enable - It's the same if you publish to a topic that
doesn't yet exist. But that means topic will get created first, and then
the messages are sent on the bus. If you subscribe to that topic later you
will get messages. Check below:
http://grokbase.com/t/kafka/users/1648kbr04c/subscribe-on-
a-topic-that-does-not-exist
On 7 Jul 2017 9:46 pm, "Ali Akhtar" <al...@gmail.com> wrote:
> Sometimes I see warnings in my logs if i create a consumer for a topic
> which doesn't exist. Such as:
>
> org.apache.kafka.clients.NetworkClient - Error while fetching metadata
> with correlation id 1 : {example_topic=LEADER_NOT_AVAILABLE}
>
> If later messages are posted to that topic (which will create it), will my
> consumer receive those messages?
>
Re: Kafka behavior when consuming a topic which doesn't exist?
Posted by "M. Manna" <ma...@gmail.com>.
That depends.
If auto creation of non-existent topic enabled (check docs), then it will
simple use the minimum partiotion and replication settings defined in
broker config to create a topic. If auto creation is disabled, your
consumer group won't do anything.
With auto creation enable - It's the same if you publish to a topic that
doesn't yet exist. But that means topic will get created first, and then
the messages are sent on the bus. If you subscribe to that topic later you
will get messages. Check below:
http://grokbase.com/t/kafka/users/1648kbr04c/subscribe-on-a-topic-that-does-not-exist
On 7 Jul 2017 9:46 pm, "Ali Akhtar" <al...@gmail.com> wrote:
> Sometimes I see warnings in my logs if i create a consumer for a topic
> which doesn't exist. Such as:
>
> org.apache.kafka.clients.NetworkClient - Error while fetching metadata
> with correlation id 1 : {example_topic=LEADER_NOT_AVAILABLE}
>
> If later messages are posted to that topic (which will create it), will my
> consumer receive those messages?
>