You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by David Morales de Frías <dm...@paradigmatecnologico.com> on 2014/03/06 11:10:07 UTC
consumer reaction to auto-create topics
Hi there,
If i start a consumer in a non-existent topic (auto-create true) before the
producer, the consumer never gets the messages.
These are the steps:
1) kafka-console-consumer --topic newTopic (it doesn't exist)
2) kafka-console-producer --topic newTopic
3) Send some messages
4) I can see this in the log and the consumer never receives them
[2014-03-06 10:58:18,641] WARN Error while fetching metadata
[{TopicMetadata for topic newTopic ->
No partition metadata for topic newTopic due to
kafka.common.LeaderNotAvailableException}] for topic [newTopic]: class
kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartitionInfo)
5) restart kafka-console-consumer --topic newTopic
6) Send some messages from producer
7) Consumer receives messages.
Is that ok?
Where are the produced messages? I'm sure that consumer never see them, but
are they stored in brokers?
Regards.
Re: consumer reaction to auto-create topics
Posted by David Morales de Frías <dm...@paradigmatecnologico.com>.
Thanks for your support.
I have explained the issue by using the console producer&consumer, in terms
of simplicity, but i have just found it in my application.
I will update this thread later with more tests.
Regards.
2014-03-06 18:56 GMT+01:00 Neha Narkhede <ne...@gmail.com>:
> I've seen this behavior when the broker was not functional. Basically what
> you see is that the console producer appears to have sent some messages.
> These messages may not have reached the server. Due to the bad state of the
> server, the consumer cannot get these messages and also runs into issues
> while rebalancing and issuing fetch requests for the topic.
>
> Could you try to run this from scratch (following the quickstart). Ensure
> that the broker and zookeeper have been started successfully.
>
> Thanks,
> Neha
>
>
> On Thu, Mar 6, 2014 at 9:34 AM, David Morales de Frías <
> dmorales@paradigmatecnologico.com> wrote:
>
> > 0.8, thanks.
> >
> >
> > 2014-03-06 18:27 GMT+01:00 Jun Rao <ju...@gmail.com>:
> >
> > > Which version of Kafka are you using?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Thu, Mar 6, 2014 at 2:10 AM, David Morales de Frías <
> > > dmorales@paradigmatecnologico.com> wrote:
> > >
> > > > Hi there,
> > > >
> > > > If i start a consumer in a non-existent topic (auto-create true)
> before
> > > the
> > > > producer, the consumer never gets the messages.
> > > >
> > > > These are the steps:
> > > >
> > > > 1) kafka-console-consumer --topic newTopic (it doesn't exist)
> > > > 2) kafka-console-producer --topic newTopic
> > > > 3) Send some messages
> > > > 4) I can see this in the log and the consumer never receives them
> > > >
> > > > [2014-03-06 10:58:18,641] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic newTopic ->
> > > > No partition metadata for topic newTopic due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [newTopic]:
> class
> > > > kafka.common.LeaderNotAvailableException
> > > > (kafka.producer.BrokerPartitionInfo)
> > > >
> > > >
> > > >
> > > > 5) restart kafka-console-consumer --topic newTopic
> > > > 6) Send some messages from producer
> > > > 7) Consumer receives messages.
> > > >
> > > >
> > > >
> > > > Is that ok?
> > > > Where are the produced messages? I'm sure that consumer never see
> them,
> > > but
> > > > are they stored in brokers?
> > > >
> > > >
> > > > Regards.
> > > >
> > >
> >
>
Re: consumer reaction to auto-create topics
Posted by Neha Narkhede <ne...@gmail.com>.
I've seen this behavior when the broker was not functional. Basically what
you see is that the console producer appears to have sent some messages.
These messages may not have reached the server. Due to the bad state of the
server, the consumer cannot get these messages and also runs into issues
while rebalancing and issuing fetch requests for the topic.
Could you try to run this from scratch (following the quickstart). Ensure
that the broker and zookeeper have been started successfully.
Thanks,
Neha
On Thu, Mar 6, 2014 at 9:34 AM, David Morales de Frías <
dmorales@paradigmatecnologico.com> wrote:
> 0.8, thanks.
>
>
> 2014-03-06 18:27 GMT+01:00 Jun Rao <ju...@gmail.com>:
>
> > Which version of Kafka are you using?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Thu, Mar 6, 2014 at 2:10 AM, David Morales de Frías <
> > dmorales@paradigmatecnologico.com> wrote:
> >
> > > Hi there,
> > >
> > > If i start a consumer in a non-existent topic (auto-create true) before
> > the
> > > producer, the consumer never gets the messages.
> > >
> > > These are the steps:
> > >
> > > 1) kafka-console-consumer --topic newTopic (it doesn't exist)
> > > 2) kafka-console-producer --topic newTopic
> > > 3) Send some messages
> > > 4) I can see this in the log and the consumer never receives them
> > >
> > > [2014-03-06 10:58:18,641] WARN Error while fetching metadata
> > > [{TopicMetadata for topic newTopic ->
> > > No partition metadata for topic newTopic due to
> > > kafka.common.LeaderNotAvailableException}] for topic [newTopic]: class
> > > kafka.common.LeaderNotAvailableException
> > > (kafka.producer.BrokerPartitionInfo)
> > >
> > >
> > >
> > > 5) restart kafka-console-consumer --topic newTopic
> > > 6) Send some messages from producer
> > > 7) Consumer receives messages.
> > >
> > >
> > >
> > > Is that ok?
> > > Where are the produced messages? I'm sure that consumer never see them,
> > but
> > > are they stored in brokers?
> > >
> > >
> > > Regards.
> > >
> >
>
Re: consumer reaction to auto-create topics
Posted by Jun Rao <ju...@gmail.com>.
Ok. The is fixed in the 0.8.1 release, which is being voted now.
Thanks,
Jun
On Thu, Mar 6, 2014 at 9:34 AM, David Morales de Frías <
dmorales@paradigmatecnologico.com> wrote:
> 0.8, thanks.
>
>
> 2014-03-06 18:27 GMT+01:00 Jun Rao <ju...@gmail.com>:
>
> > Which version of Kafka are you using?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Thu, Mar 6, 2014 at 2:10 AM, David Morales de Frías <
> > dmorales@paradigmatecnologico.com> wrote:
> >
> > > Hi there,
> > >
> > > If i start a consumer in a non-existent topic (auto-create true) before
> > the
> > > producer, the consumer never gets the messages.
> > >
> > > These are the steps:
> > >
> > > 1) kafka-console-consumer --topic newTopic (it doesn't exist)
> > > 2) kafka-console-producer --topic newTopic
> > > 3) Send some messages
> > > 4) I can see this in the log and the consumer never receives them
> > >
> > > [2014-03-06 10:58:18,641] WARN Error while fetching metadata
> > > [{TopicMetadata for topic newTopic ->
> > > No partition metadata for topic newTopic due to
> > > kafka.common.LeaderNotAvailableException}] for topic [newTopic]: class
> > > kafka.common.LeaderNotAvailableException
> > > (kafka.producer.BrokerPartitionInfo)
> > >
> > >
> > >
> > > 5) restart kafka-console-consumer --topic newTopic
> > > 6) Send some messages from producer
> > > 7) Consumer receives messages.
> > >
> > >
> > >
> > > Is that ok?
> > > Where are the produced messages? I'm sure that consumer never see them,
> > but
> > > are they stored in brokers?
> > >
> > >
> > > Regards.
> > >
> >
>
Re: consumer reaction to auto-create topics
Posted by David Morales de Frías <dm...@paradigmatecnologico.com>.
0.8, thanks.
2014-03-06 18:27 GMT+01:00 Jun Rao <ju...@gmail.com>:
> Which version of Kafka are you using?
>
> Thanks,
>
> Jun
>
>
> On Thu, Mar 6, 2014 at 2:10 AM, David Morales de Frías <
> dmorales@paradigmatecnologico.com> wrote:
>
> > Hi there,
> >
> > If i start a consumer in a non-existent topic (auto-create true) before
> the
> > producer, the consumer never gets the messages.
> >
> > These are the steps:
> >
> > 1) kafka-console-consumer --topic newTopic (it doesn't exist)
> > 2) kafka-console-producer --topic newTopic
> > 3) Send some messages
> > 4) I can see this in the log and the consumer never receives them
> >
> > [2014-03-06 10:58:18,641] WARN Error while fetching metadata
> > [{TopicMetadata for topic newTopic ->
> > No partition metadata for topic newTopic due to
> > kafka.common.LeaderNotAvailableException}] for topic [newTopic]: class
> > kafka.common.LeaderNotAvailableException
> > (kafka.producer.BrokerPartitionInfo)
> >
> >
> >
> > 5) restart kafka-console-consumer --topic newTopic
> > 6) Send some messages from producer
> > 7) Consumer receives messages.
> >
> >
> >
> > Is that ok?
> > Where are the produced messages? I'm sure that consumer never see them,
> but
> > are they stored in brokers?
> >
> >
> > Regards.
> >
>
Re: consumer reaction to auto-create topics
Posted by Jun Rao <ju...@gmail.com>.
Which version of Kafka are you using?
Thanks,
Jun
On Thu, Mar 6, 2014 at 2:10 AM, David Morales de Frías <
dmorales@paradigmatecnologico.com> wrote:
> Hi there,
>
> If i start a consumer in a non-existent topic (auto-create true) before the
> producer, the consumer never gets the messages.
>
> These are the steps:
>
> 1) kafka-console-consumer --topic newTopic (it doesn't exist)
> 2) kafka-console-producer --topic newTopic
> 3) Send some messages
> 4) I can see this in the log and the consumer never receives them
>
> [2014-03-06 10:58:18,641] WARN Error while fetching metadata
> [{TopicMetadata for topic newTopic ->
> No partition metadata for topic newTopic due to
> kafka.common.LeaderNotAvailableException}] for topic [newTopic]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
>
>
>
> 5) restart kafka-console-consumer --topic newTopic
> 6) Send some messages from producer
> 7) Consumer receives messages.
>
>
>
> Is that ok?
> Where are the produced messages? I'm sure that consumer never see them, but
> are they stored in brokers?
>
>
> Regards.
>