You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Andras Hatvani <an...@andrashatvani.com> on 2014/09/26 14:43:36 UTC
LeaderNotAvailableException, although leader elected
Hi,
What does the exception mean in the following context?
I've got two Kafka brokers out which the second one will be elected as leader:
kafka2_1 | [2014-09-26 12:35:07,289] INFO [Kafka Server 2], started (kafka.server.KafkaServer)
kafka2_1 | [2014-09-26 12:35:07,394] INFO New leader is 2 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
Then, when I try to produce a message, I only get the following error:
23182 [main] INFO kafka.producer.async.DefaultEventHandler - Back off for 100 ms before retrying send. Remaining retries = 0
23286 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from broker id:0,host:192.168.59.103,port:9092 with correlation id 8 for 1 topic(s) Set(inputTopic)
23287 [main] INFO kafka.producer.SyncProducer - Connected to 192.168.59.103:9092 for producing
23300 [main] INFO kafka.producer.SyncProducer - Disconnecting from 192.168.59.103:9092
23301 [main] WARN kafka.producer.BrokerPartitionInfo - Error while fetching metadata [{TopicMetadata for topic inputTopic ->
No partition metadata for topic inputTopic due to kafka.common.LeaderNotAvailableException}] for topic [inputTopic]: class kafka.common.LeaderNotAvailableException
23303 [main] ERROR kafka.producer.async.DefaultEventHandler - Failed to send requests for topics inputTopic with correlation ids in [0,8]
The property metadata.broker.list contains the host:port values of both brokers.
What can be wrong/missing?
Thanks,
Andras
Re: LeaderNotAvailableException, although leader elected
Posted by Neha Narkhede <ne...@gmail.com>.
Andras,
Thanks for your feedback!
In my opinion programmatic message sending must work out of the box on the
first try, without any exceptions, warnings or the need for additional
configuration.
I'd be glad to support/contribute.
I agree that the behavior of the producer for the first message on a topic
is awkward and I'd encourage feedback from you. We certainly are interested
in improving user experience. Could you please file a JIRA so we can
discuss alternatives there?
Thanks,
Neha
On Sat, Sep 27, 2014 at 12:58 AM, Andras Hatvani <
andras.hatvani@andrashatvani.com> wrote:
> AFAIK not topics, but only partitions of topics have leaders. What
> controllers do you mean? I haven't read about such.
>
> Thanks for the explanation regarding the metadata request, in the meantime
> I found out that this is an expected (!) failure (
> http://qnalist.com/questions/4787268/kafka-common-leadernotavailableexception
> ).
>
> For me this is isn't an acceptable way to communicate that the leader
> election is in progress. There is not a single hint to this fact, but only
> an exception.
> If this is an expected behavior, then it not only mustn't be an exception,
> but it also has to be communicated that there is something in progress.
> Furthermore, suggestions regarding changing the values variables I
> mentioned in my solution should be mandatory.
>
> This was my case:
> - OK, let's use Kafka
> - Create an infrastructure
> - Create a programmatic producer
> - Send a message
> - Message sending fails.
> - Retry
> - Message sending works!
> - Look for answers on the internet and in the docs
> - Read the configuration
> - Play around with configuration values.
>
> This is bad user experience especially for a newbie and involves a lot of
> effort.
>
> In my opinion programmatic message sending must work out of the box on the
> first try, without any exceptions, warnings or the need for additional
> configuration.
>
> I'd be glad to support/contribute.
>
> Regards,
> Andras
>
> > On 26 Sep 2014, at 19:53, Joel Koshy <jj...@gmail.com> wrote:
> >
> >>> kafka2_1 | [2014-09-26 12:35:07,289] INFO [Kafka Server 2],
> started (kafka.server.KafkaServer)
> >>> kafka2_1 | [2014-09-26 12:35:07,394] INFO New leader is 2
> (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> >
> > The above logs are for controller election. Not leader election for
> > the topic you are producing to.
> >
> > When you producer starts it will issue a metadata request and it
> > should auto-create the topic (provided auto-create is on which is
> > default). The first metadata request for a non-existent topic always
> > returns LeaderNotAvailable because the controller then has to elect a
> > leader for the new topic.
> >
> > Joel
> >
> > On Fri, Sep 26, 2014 at 04:07:58PM +0200, Andras Hatvani wrote:
> >> And the solution was:
> >> Increase retry.backoff.ms from the default 100 ms to 1000 ms, so the
> output is:
> >>
> >> 11891 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from
> broker id:0,host:192.168.59.103,port:9092 with correlation id 0 for 1
> topic(s) Set(inputTopic)
> >> 11893 [main] INFO kafka.producer.SyncProducer - Connected to
> 192.168.59.103:9092 for producing
> >> 12045 [main] INFO kafka.producer.SyncProducer - Disconnecting from
> 192.168.59.103:9092
> >> 12062 [main] WARN kafka.producer.BrokerPartitionInfo - Error while
> fetching metadata [{TopicMetadata for topic inputTopic ->
> >> No partition metadata for topic inputTopic due to
> kafka.common.LeaderNotAvailableException}] for topic [inputTopic]: class
> kafka.common.LeaderNotAvailableException
> >> 12066 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from
> broker id:0,host:192.168.59.103,port:9092 with correlation id 1 for 1
> topic(s) Set(inputTopic)
> >> 12067 [main] INFO kafka.producer.SyncProducer - Connected to
> 192.168.59.103:9092 for producing
> >> 12097 [main] INFO kafka.producer.SyncProducer - Disconnecting from
> 192.168.59.103:9092
> >> 12098 [main] WARN kafka.producer.BrokerPartitionInfo - Error while
> fetching metadata [{TopicMetadata for topic inputTopic ->
> >> No partition metadata for topic inputTopic due to
> kafka.common.LeaderNotAvailableException}] for topic [inputTopic]: class
> kafka.common.LeaderNotAvailableException
> >> 12098 [main] ERROR kafka.producer.async.DefaultEventHandler - Failed to
> collate messages by topic, partition due to: Failed to fetch topic metadata
> for topic: inputTopic
> >> 12099 [main] INFO kafka.producer.async.DefaultEventHandler - Back off
> for 1000 ms before retrying send. Remaining retries = 3
> >> 13104 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from
> broker id:0,host:192.168.59.103,port:9092 with correlation id 2 for 1
> topic(s) Set(inputTopic)
> >> 13111 [main] INFO kafka.producer.SyncProducer - Connected to
> 192.168.59.103:9092 for producing
> >> 13137 [main] INFO kafka.producer.SyncProducer - Disconnecting from
> 192.168.59.103:9092
> >> 13161 [main] INFO kafka.producer.SyncProducer - Connected to
> 192.168.59.103:9092 for producing
> >>
> >> In case this would be not enough for you, you can try to change the
> values of
> >> - message.send.max.retries from the default 5 to e.g. 10 and
> >> - topic.metadata.refresh.interval.ms to 0
> >>
> >>
> >> It is still unclear, why it takes 2 tries i.e. 2 seconds to fetch the
> topic metadata…
> >>
> >>
> >> Cheers,
> >> Andras
> >>
> >>> On 26 Sep 2014, at 14:43, Andras Hatvani <
> andras.hatvani@andrashatvani.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>> What does the exception mean in the following context?
> >>>
> >>> I've got two Kafka brokers out which the second one will be elected as
> leader:
> >>>
> >>> kafka2_1 | [2014-09-26 12:35:07,289] INFO [Kafka Server 2],
> started (kafka.server.KafkaServer)
> >>> kafka2_1 | [2014-09-26 12:35:07,394] INFO New leader is 2
> (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> >>>
> >>> Then, when I try to produce a message, I only get the following error:
> >>>
> >>> 23182 [main] INFO kafka.producer.async.DefaultEventHandler - Back off
> for 100 ms before retrying send. Remaining retries = 0
> >>> 23286 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from
> broker id:0,host:192.168.59.103,port:9092 with correlation id 8 for 1
> topic(s) Set(inputTopic)
> >>> 23287 [main] INFO kafka.producer.SyncProducer - Connected to
> 192.168.59.103:9092 for producing
> >>> 23300 [main] INFO kafka.producer.SyncProducer - Disconnecting from
> 192.168.59.103:9092
> >>> 23301 [main] WARN kafka.producer.BrokerPartitionInfo - Error while
> fetching metadata [{TopicMetadata for topic inputTopic ->
> >>> No partition metadata for topic inputTopic due to
> kafka.common.LeaderNotAvailableException}] for topic [inputTopic]: class
> kafka.common.LeaderNotAvailableException
> >>> 23303 [main] ERROR kafka.producer.async.DefaultEventHandler - Failed
> to send requests for topics inputTopic with correlation ids in [0,8]
> >>>
> >>> The property metadata.broker.list contains the host:port values of
> both brokers.
> >>>
> >>> What can be wrong/missing?
> >>> Thanks,
> >>> Andras
> >>
> >
>
>
Re: LeaderNotAvailableException, although leader elected
Posted by Neha Narkhede <ne...@gmail.com>.
Andras,
Thanks for your feedback!
In my opinion programmatic message sending must work out of the box on the
first try, without any exceptions, warnings or the need for additional
configuration.
I'd be glad to support/contribute.
I agree that the behavior of the producer for the first message on a topic
is awkward and I'd encourage feedback from you. We certainly are interested
in improving user experience. Could you please file a JIRA so we can
discuss alternatives there?
Thanks,
Neha
On Sat, Sep 27, 2014 at 12:58 AM, Andras Hatvani <
andras.hatvani@andrashatvani.com> wrote:
> AFAIK not topics, but only partitions of topics have leaders. What
> controllers do you mean? I haven't read about such.
>
> Thanks for the explanation regarding the metadata request, in the meantime
> I found out that this is an expected (!) failure (
> http://qnalist.com/questions/4787268/kafka-common-leadernotavailableexception
> ).
>
> For me this is isn't an acceptable way to communicate that the leader
> election is in progress. There is not a single hint to this fact, but only
> an exception.
> If this is an expected behavior, then it not only mustn't be an exception,
> but it also has to be communicated that there is something in progress.
> Furthermore, suggestions regarding changing the values variables I
> mentioned in my solution should be mandatory.
>
> This was my case:
> - OK, let's use Kafka
> - Create an infrastructure
> - Create a programmatic producer
> - Send a message
> - Message sending fails.
> - Retry
> - Message sending works!
> - Look for answers on the internet and in the docs
> - Read the configuration
> - Play around with configuration values.
>
> This is bad user experience especially for a newbie and involves a lot of
> effort.
>
> In my opinion programmatic message sending must work out of the box on the
> first try, without any exceptions, warnings or the need for additional
> configuration.
>
> I'd be glad to support/contribute.
>
> Regards,
> Andras
>
> > On 26 Sep 2014, at 19:53, Joel Koshy <jj...@gmail.com> wrote:
> >
> >>> kafka2_1 | [2014-09-26 12:35:07,289] INFO [Kafka Server 2],
> started (kafka.server.KafkaServer)
> >>> kafka2_1 | [2014-09-26 12:35:07,394] INFO New leader is 2
> (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> >
> > The above logs are for controller election. Not leader election for
> > the topic you are producing to.
> >
> > When you producer starts it will issue a metadata request and it
> > should auto-create the topic (provided auto-create is on which is
> > default). The first metadata request for a non-existent topic always
> > returns LeaderNotAvailable because the controller then has to elect a
> > leader for the new topic.
> >
> > Joel
> >
> > On Fri, Sep 26, 2014 at 04:07:58PM +0200, Andras Hatvani wrote:
> >> And the solution was:
> >> Increase retry.backoff.ms from the default 100 ms to 1000 ms, so the
> output is:
> >>
> >> 11891 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from
> broker id:0,host:192.168.59.103,port:9092 with correlation id 0 for 1
> topic(s) Set(inputTopic)
> >> 11893 [main] INFO kafka.producer.SyncProducer - Connected to
> 192.168.59.103:9092 for producing
> >> 12045 [main] INFO kafka.producer.SyncProducer - Disconnecting from
> 192.168.59.103:9092
> >> 12062 [main] WARN kafka.producer.BrokerPartitionInfo - Error while
> fetching metadata [{TopicMetadata for topic inputTopic ->
> >> No partition metadata for topic inputTopic due to
> kafka.common.LeaderNotAvailableException}] for topic [inputTopic]: class
> kafka.common.LeaderNotAvailableException
> >> 12066 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from
> broker id:0,host:192.168.59.103,port:9092 with correlation id 1 for 1
> topic(s) Set(inputTopic)
> >> 12067 [main] INFO kafka.producer.SyncProducer - Connected to
> 192.168.59.103:9092 for producing
> >> 12097 [main] INFO kafka.producer.SyncProducer - Disconnecting from
> 192.168.59.103:9092
> >> 12098 [main] WARN kafka.producer.BrokerPartitionInfo - Error while
> fetching metadata [{TopicMetadata for topic inputTopic ->
> >> No partition metadata for topic inputTopic due to
> kafka.common.LeaderNotAvailableException}] for topic [inputTopic]: class
> kafka.common.LeaderNotAvailableException
> >> 12098 [main] ERROR kafka.producer.async.DefaultEventHandler - Failed to
> collate messages by topic, partition due to: Failed to fetch topic metadata
> for topic: inputTopic
> >> 12099 [main] INFO kafka.producer.async.DefaultEventHandler - Back off
> for 1000 ms before retrying send. Remaining retries = 3
> >> 13104 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from
> broker id:0,host:192.168.59.103,port:9092 with correlation id 2 for 1
> topic(s) Set(inputTopic)
> >> 13111 [main] INFO kafka.producer.SyncProducer - Connected to
> 192.168.59.103:9092 for producing
> >> 13137 [main] INFO kafka.producer.SyncProducer - Disconnecting from
> 192.168.59.103:9092
> >> 13161 [main] INFO kafka.producer.SyncProducer - Connected to
> 192.168.59.103:9092 for producing
> >>
> >> In case this would be not enough for you, you can try to change the
> values of
> >> - message.send.max.retries from the default 5 to e.g. 10 and
> >> - topic.metadata.refresh.interval.ms to 0
> >>
> >>
> >> It is still unclear, why it takes 2 tries i.e. 2 seconds to fetch the
> topic metadata…
> >>
> >>
> >> Cheers,
> >> Andras
> >>
> >>> On 26 Sep 2014, at 14:43, Andras Hatvani <
> andras.hatvani@andrashatvani.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>> What does the exception mean in the following context?
> >>>
> >>> I've got two Kafka brokers out which the second one will be elected as
> leader:
> >>>
> >>> kafka2_1 | [2014-09-26 12:35:07,289] INFO [Kafka Server 2],
> started (kafka.server.KafkaServer)
> >>> kafka2_1 | [2014-09-26 12:35:07,394] INFO New leader is 2
> (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> >>>
> >>> Then, when I try to produce a message, I only get the following error:
> >>>
> >>> 23182 [main] INFO kafka.producer.async.DefaultEventHandler - Back off
> for 100 ms before retrying send. Remaining retries = 0
> >>> 23286 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from
> broker id:0,host:192.168.59.103,port:9092 with correlation id 8 for 1
> topic(s) Set(inputTopic)
> >>> 23287 [main] INFO kafka.producer.SyncProducer - Connected to
> 192.168.59.103:9092 for producing
> >>> 23300 [main] INFO kafka.producer.SyncProducer - Disconnecting from
> 192.168.59.103:9092
> >>> 23301 [main] WARN kafka.producer.BrokerPartitionInfo - Error while
> fetching metadata [{TopicMetadata for topic inputTopic ->
> >>> No partition metadata for topic inputTopic due to
> kafka.common.LeaderNotAvailableException}] for topic [inputTopic]: class
> kafka.common.LeaderNotAvailableException
> >>> 23303 [main] ERROR kafka.producer.async.DefaultEventHandler - Failed
> to send requests for topics inputTopic with correlation ids in [0,8]
> >>>
> >>> The property metadata.broker.list contains the host:port values of
> both brokers.
> >>>
> >>> What can be wrong/missing?
> >>> Thanks,
> >>> Andras
> >>
> >
>
>
Re: LeaderNotAvailableException, although leader elected
Posted by Andras Hatvani <an...@andrashatvani.com>.
AFAIK not topics, but only partitions of topics have leaders. What controllers do you mean? I haven't read about such.
Thanks for the explanation regarding the metadata request, in the meantime I found out that this is an expected (!) failure (http://qnalist.com/questions/4787268/kafka-common-leadernotavailableexception).
For me this is isn't an acceptable way to communicate that the leader election is in progress. There is not a single hint to this fact, but only an exception.
If this is an expected behavior, then it not only mustn't be an exception, but it also has to be communicated that there is something in progress. Furthermore, suggestions regarding changing the values variables I mentioned in my solution should be mandatory.
This was my case:
- OK, let's use Kafka
- Create an infrastructure
- Create a programmatic producer
- Send a message
- Message sending fails.
- Retry
- Message sending works!
- Look for answers on the internet and in the docs
- Read the configuration
- Play around with configuration values.
This is bad user experience especially for a newbie and involves a lot of effort.
In my opinion programmatic message sending must work out of the box on the first try, without any exceptions, warnings or the need for additional configuration.
I'd be glad to support/contribute.
Regards,
Andras
> On 26 Sep 2014, at 19:53, Joel Koshy <jj...@gmail.com> wrote:
>
>>> kafka2_1 | [2014-09-26 12:35:07,289] INFO [Kafka Server 2], started (kafka.server.KafkaServer)
>>> kafka2_1 | [2014-09-26 12:35:07,394] INFO New leader is 2 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
>
> The above logs are for controller election. Not leader election for
> the topic you are producing to.
>
> When you producer starts it will issue a metadata request and it
> should auto-create the topic (provided auto-create is on which is
> default). The first metadata request for a non-existent topic always
> returns LeaderNotAvailable because the controller then has to elect a
> leader for the new topic.
>
> Joel
>
> On Fri, Sep 26, 2014 at 04:07:58PM +0200, Andras Hatvani wrote:
>> And the solution was:
>> Increase retry.backoff.ms from the default 100 ms to 1000 ms, so the output is:
>>
>> 11891 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from broker id:0,host:192.168.59.103,port:9092 with correlation id 0 for 1 topic(s) Set(inputTopic)
>> 11893 [main] INFO kafka.producer.SyncProducer - Connected to 192.168.59.103:9092 for producing
>> 12045 [main] INFO kafka.producer.SyncProducer - Disconnecting from 192.168.59.103:9092
>> 12062 [main] WARN kafka.producer.BrokerPartitionInfo - Error while fetching metadata [{TopicMetadata for topic inputTopic ->
>> No partition metadata for topic inputTopic due to kafka.common.LeaderNotAvailableException}] for topic [inputTopic]: class kafka.common.LeaderNotAvailableException
>> 12066 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from broker id:0,host:192.168.59.103,port:9092 with correlation id 1 for 1 topic(s) Set(inputTopic)
>> 12067 [main] INFO kafka.producer.SyncProducer - Connected to 192.168.59.103:9092 for producing
>> 12097 [main] INFO kafka.producer.SyncProducer - Disconnecting from 192.168.59.103:9092
>> 12098 [main] WARN kafka.producer.BrokerPartitionInfo - Error while fetching metadata [{TopicMetadata for topic inputTopic ->
>> No partition metadata for topic inputTopic due to kafka.common.LeaderNotAvailableException}] for topic [inputTopic]: class kafka.common.LeaderNotAvailableException
>> 12098 [main] ERROR kafka.producer.async.DefaultEventHandler - Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: inputTopic
>> 12099 [main] INFO kafka.producer.async.DefaultEventHandler - Back off for 1000 ms before retrying send. Remaining retries = 3
>> 13104 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from broker id:0,host:192.168.59.103,port:9092 with correlation id 2 for 1 topic(s) Set(inputTopic)
>> 13111 [main] INFO kafka.producer.SyncProducer - Connected to 192.168.59.103:9092 for producing
>> 13137 [main] INFO kafka.producer.SyncProducer - Disconnecting from 192.168.59.103:9092
>> 13161 [main] INFO kafka.producer.SyncProducer - Connected to 192.168.59.103:9092 for producing
>>
>> In case this would be not enough for you, you can try to change the values of
>> - message.send.max.retries from the default 5 to e.g. 10 and
>> - topic.metadata.refresh.interval.ms to 0
>>
>>
>> It is still unclear, why it takes 2 tries i.e. 2 seconds to fetch the topic metadata…
>>
>>
>> Cheers,
>> Andras
>>
>>> On 26 Sep 2014, at 14:43, Andras Hatvani <an...@andrashatvani.com> wrote:
>>>
>>> Hi,
>>>
>>> What does the exception mean in the following context?
>>>
>>> I've got two Kafka brokers out which the second one will be elected as leader:
>>>
>>> kafka2_1 | [2014-09-26 12:35:07,289] INFO [Kafka Server 2], started (kafka.server.KafkaServer)
>>> kafka2_1 | [2014-09-26 12:35:07,394] INFO New leader is 2 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
>>>
>>> Then, when I try to produce a message, I only get the following error:
>>>
>>> 23182 [main] INFO kafka.producer.async.DefaultEventHandler - Back off for 100 ms before retrying send. Remaining retries = 0
>>> 23286 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from broker id:0,host:192.168.59.103,port:9092 with correlation id 8 for 1 topic(s) Set(inputTopic)
>>> 23287 [main] INFO kafka.producer.SyncProducer - Connected to 192.168.59.103:9092 for producing
>>> 23300 [main] INFO kafka.producer.SyncProducer - Disconnecting from 192.168.59.103:9092
>>> 23301 [main] WARN kafka.producer.BrokerPartitionInfo - Error while fetching metadata [{TopicMetadata for topic inputTopic ->
>>> No partition metadata for topic inputTopic due to kafka.common.LeaderNotAvailableException}] for topic [inputTopic]: class kafka.common.LeaderNotAvailableException
>>> 23303 [main] ERROR kafka.producer.async.DefaultEventHandler - Failed to send requests for topics inputTopic with correlation ids in [0,8]
>>>
>>> The property metadata.broker.list contains the host:port values of both brokers.
>>>
>>> What can be wrong/missing?
>>> Thanks,
>>> Andras
>>
>
Re: LeaderNotAvailableException, although leader elected
Posted by Andras Hatvani <an...@andrashatvani.com>.
AFAIK not topics, but only partitions of topics have leaders. What controllers do you mean? I haven't read about such.
Thanks for the explanation regarding the metadata request, in the meantime I found out that this is an expected (!) failure (http://qnalist.com/questions/4787268/kafka-common-leadernotavailableexception).
For me this is isn't an acceptable way to communicate that the leader election is in progress. There is not a single hint to this fact, but only an exception.
If this is an expected behavior, then it not only mustn't be an exception, but it also has to be communicated that there is something in progress. Furthermore, suggestions regarding changing the values variables I mentioned in my solution should be mandatory.
This was my case:
- OK, let's use Kafka
- Create an infrastructure
- Create a programmatic producer
- Send a message
- Message sending fails.
- Retry
- Message sending works!
- Look for answers on the internet and in the docs
- Read the configuration
- Play around with configuration values.
This is bad user experience especially for a newbie and involves a lot of effort.
In my opinion programmatic message sending must work out of the box on the first try, without any exceptions, warnings or the need for additional configuration.
I'd be glad to support/contribute.
Regards,
Andras
> On 26 Sep 2014, at 19:53, Joel Koshy <jj...@gmail.com> wrote:
>
>>> kafka2_1 | [2014-09-26 12:35:07,289] INFO [Kafka Server 2], started (kafka.server.KafkaServer)
>>> kafka2_1 | [2014-09-26 12:35:07,394] INFO New leader is 2 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
>
> The above logs are for controller election. Not leader election for
> the topic you are producing to.
>
> When you producer starts it will issue a metadata request and it
> should auto-create the topic (provided auto-create is on which is
> default). The first metadata request for a non-existent topic always
> returns LeaderNotAvailable because the controller then has to elect a
> leader for the new topic.
>
> Joel
>
> On Fri, Sep 26, 2014 at 04:07:58PM +0200, Andras Hatvani wrote:
>> And the solution was:
>> Increase retry.backoff.ms from the default 100 ms to 1000 ms, so the output is:
>>
>> 11891 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from broker id:0,host:192.168.59.103,port:9092 with correlation id 0 for 1 topic(s) Set(inputTopic)
>> 11893 [main] INFO kafka.producer.SyncProducer - Connected to 192.168.59.103:9092 for producing
>> 12045 [main] INFO kafka.producer.SyncProducer - Disconnecting from 192.168.59.103:9092
>> 12062 [main] WARN kafka.producer.BrokerPartitionInfo - Error while fetching metadata [{TopicMetadata for topic inputTopic ->
>> No partition metadata for topic inputTopic due to kafka.common.LeaderNotAvailableException}] for topic [inputTopic]: class kafka.common.LeaderNotAvailableException
>> 12066 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from broker id:0,host:192.168.59.103,port:9092 with correlation id 1 for 1 topic(s) Set(inputTopic)
>> 12067 [main] INFO kafka.producer.SyncProducer - Connected to 192.168.59.103:9092 for producing
>> 12097 [main] INFO kafka.producer.SyncProducer - Disconnecting from 192.168.59.103:9092
>> 12098 [main] WARN kafka.producer.BrokerPartitionInfo - Error while fetching metadata [{TopicMetadata for topic inputTopic ->
>> No partition metadata for topic inputTopic due to kafka.common.LeaderNotAvailableException}] for topic [inputTopic]: class kafka.common.LeaderNotAvailableException
>> 12098 [main] ERROR kafka.producer.async.DefaultEventHandler - Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: inputTopic
>> 12099 [main] INFO kafka.producer.async.DefaultEventHandler - Back off for 1000 ms before retrying send. Remaining retries = 3
>> 13104 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from broker id:0,host:192.168.59.103,port:9092 with correlation id 2 for 1 topic(s) Set(inputTopic)
>> 13111 [main] INFO kafka.producer.SyncProducer - Connected to 192.168.59.103:9092 for producing
>> 13137 [main] INFO kafka.producer.SyncProducer - Disconnecting from 192.168.59.103:9092
>> 13161 [main] INFO kafka.producer.SyncProducer - Connected to 192.168.59.103:9092 for producing
>>
>> In case this would be not enough for you, you can try to change the values of
>> - message.send.max.retries from the default 5 to e.g. 10 and
>> - topic.metadata.refresh.interval.ms to 0
>>
>>
>> It is still unclear, why it takes 2 tries i.e. 2 seconds to fetch the topic metadata…
>>
>>
>> Cheers,
>> Andras
>>
>>> On 26 Sep 2014, at 14:43, Andras Hatvani <an...@andrashatvani.com> wrote:
>>>
>>> Hi,
>>>
>>> What does the exception mean in the following context?
>>>
>>> I've got two Kafka brokers out which the second one will be elected as leader:
>>>
>>> kafka2_1 | [2014-09-26 12:35:07,289] INFO [Kafka Server 2], started (kafka.server.KafkaServer)
>>> kafka2_1 | [2014-09-26 12:35:07,394] INFO New leader is 2 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
>>>
>>> Then, when I try to produce a message, I only get the following error:
>>>
>>> 23182 [main] INFO kafka.producer.async.DefaultEventHandler - Back off for 100 ms before retrying send. Remaining retries = 0
>>> 23286 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from broker id:0,host:192.168.59.103,port:9092 with correlation id 8 for 1 topic(s) Set(inputTopic)
>>> 23287 [main] INFO kafka.producer.SyncProducer - Connected to 192.168.59.103:9092 for producing
>>> 23300 [main] INFO kafka.producer.SyncProducer - Disconnecting from 192.168.59.103:9092
>>> 23301 [main] WARN kafka.producer.BrokerPartitionInfo - Error while fetching metadata [{TopicMetadata for topic inputTopic ->
>>> No partition metadata for topic inputTopic due to kafka.common.LeaderNotAvailableException}] for topic [inputTopic]: class kafka.common.LeaderNotAvailableException
>>> 23303 [main] ERROR kafka.producer.async.DefaultEventHandler - Failed to send requests for topics inputTopic with correlation ids in [0,8]
>>>
>>> The property metadata.broker.list contains the host:port values of both brokers.
>>>
>>> What can be wrong/missing?
>>> Thanks,
>>> Andras
>>
>
Re: LeaderNotAvailableException, although leader elected
Posted by Joel Koshy <jj...@gmail.com>.
> > kafka2_1 | [2014-09-26 12:35:07,289] INFO [Kafka Server 2], started (kafka.server.KafkaServer)
> > kafka2_1 | [2014-09-26 12:35:07,394] INFO New leader is 2 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
The above logs are for controller election. Not leader election for
the topic you are producing to.
When you producer starts it will issue a metadata request and it
should auto-create the topic (provided auto-create is on which is
default). The first metadata request for a non-existent topic always
returns LeaderNotAvailable because the controller then has to elect a
leader for the new topic.
Joel
On Fri, Sep 26, 2014 at 04:07:58PM +0200, Andras Hatvani wrote:
> And the solution was:
> Increase retry.backoff.ms from the default 100 ms to 1000 ms, so the output is:
>
> 11891 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from broker id:0,host:192.168.59.103,port:9092 with correlation id 0 for 1 topic(s) Set(inputTopic)
> 11893 [main] INFO kafka.producer.SyncProducer - Connected to 192.168.59.103:9092 for producing
> 12045 [main] INFO kafka.producer.SyncProducer - Disconnecting from 192.168.59.103:9092
> 12062 [main] WARN kafka.producer.BrokerPartitionInfo - Error while fetching metadata [{TopicMetadata for topic inputTopic ->
> No partition metadata for topic inputTopic due to kafka.common.LeaderNotAvailableException}] for topic [inputTopic]: class kafka.common.LeaderNotAvailableException
> 12066 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from broker id:0,host:192.168.59.103,port:9092 with correlation id 1 for 1 topic(s) Set(inputTopic)
> 12067 [main] INFO kafka.producer.SyncProducer - Connected to 192.168.59.103:9092 for producing
> 12097 [main] INFO kafka.producer.SyncProducer - Disconnecting from 192.168.59.103:9092
> 12098 [main] WARN kafka.producer.BrokerPartitionInfo - Error while fetching metadata [{TopicMetadata for topic inputTopic ->
> No partition metadata for topic inputTopic due to kafka.common.LeaderNotAvailableException}] for topic [inputTopic]: class kafka.common.LeaderNotAvailableException
> 12098 [main] ERROR kafka.producer.async.DefaultEventHandler - Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: inputTopic
> 12099 [main] INFO kafka.producer.async.DefaultEventHandler - Back off for 1000 ms before retrying send. Remaining retries = 3
> 13104 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from broker id:0,host:192.168.59.103,port:9092 with correlation id 2 for 1 topic(s) Set(inputTopic)
> 13111 [main] INFO kafka.producer.SyncProducer - Connected to 192.168.59.103:9092 for producing
> 13137 [main] INFO kafka.producer.SyncProducer - Disconnecting from 192.168.59.103:9092
> 13161 [main] INFO kafka.producer.SyncProducer - Connected to 192.168.59.103:9092 for producing
>
> In case this would be not enough for you, you can try to change the values of
> - message.send.max.retries from the default 5 to e.g. 10 and
> - topic.metadata.refresh.interval.ms to 0
>
>
> It is still unclear, why it takes 2 tries i.e. 2 seconds to fetch the topic metadata…
>
>
> Cheers,
> Andras
>
> > On 26 Sep 2014, at 14:43, Andras Hatvani <an...@andrashatvani.com> wrote:
> >
> > Hi,
> >
> > What does the exception mean in the following context?
> >
> > I've got two Kafka brokers out which the second one will be elected as leader:
> >
> > kafka2_1 | [2014-09-26 12:35:07,289] INFO [Kafka Server 2], started (kafka.server.KafkaServer)
> > kafka2_1 | [2014-09-26 12:35:07,394] INFO New leader is 2 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> >
> > Then, when I try to produce a message, I only get the following error:
> >
> > 23182 [main] INFO kafka.producer.async.DefaultEventHandler - Back off for 100 ms before retrying send. Remaining retries = 0
> > 23286 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from broker id:0,host:192.168.59.103,port:9092 with correlation id 8 for 1 topic(s) Set(inputTopic)
> > 23287 [main] INFO kafka.producer.SyncProducer - Connected to 192.168.59.103:9092 for producing
> > 23300 [main] INFO kafka.producer.SyncProducer - Disconnecting from 192.168.59.103:9092
> > 23301 [main] WARN kafka.producer.BrokerPartitionInfo - Error while fetching metadata [{TopicMetadata for topic inputTopic ->
> > No partition metadata for topic inputTopic due to kafka.common.LeaderNotAvailableException}] for topic [inputTopic]: class kafka.common.LeaderNotAvailableException
> > 23303 [main] ERROR kafka.producer.async.DefaultEventHandler - Failed to send requests for topics inputTopic with correlation ids in [0,8]
> >
> > The property metadata.broker.list contains the host:port values of both brokers.
> >
> > What can be wrong/missing?
> > Thanks,
> > Andras
>
Re: LeaderNotAvailableException, although leader elected
Posted by Andras Hatvani <an...@andrashatvani.com>.
And the solution was:
Increase retry.backoff.ms from the default 100 ms to 1000 ms, so the output is:
11891 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from broker id:0,host:192.168.59.103,port:9092 with correlation id 0 for 1 topic(s) Set(inputTopic)
11893 [main] INFO kafka.producer.SyncProducer - Connected to 192.168.59.103:9092 for producing
12045 [main] INFO kafka.producer.SyncProducer - Disconnecting from 192.168.59.103:9092
12062 [main] WARN kafka.producer.BrokerPartitionInfo - Error while fetching metadata [{TopicMetadata for topic inputTopic ->
No partition metadata for topic inputTopic due to kafka.common.LeaderNotAvailableException}] for topic [inputTopic]: class kafka.common.LeaderNotAvailableException
12066 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from broker id:0,host:192.168.59.103,port:9092 with correlation id 1 for 1 topic(s) Set(inputTopic)
12067 [main] INFO kafka.producer.SyncProducer - Connected to 192.168.59.103:9092 for producing
12097 [main] INFO kafka.producer.SyncProducer - Disconnecting from 192.168.59.103:9092
12098 [main] WARN kafka.producer.BrokerPartitionInfo - Error while fetching metadata [{TopicMetadata for topic inputTopic ->
No partition metadata for topic inputTopic due to kafka.common.LeaderNotAvailableException}] for topic [inputTopic]: class kafka.common.LeaderNotAvailableException
12098 [main] ERROR kafka.producer.async.DefaultEventHandler - Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: inputTopic
12099 [main] INFO kafka.producer.async.DefaultEventHandler - Back off for 1000 ms before retrying send. Remaining retries = 3
13104 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from broker id:0,host:192.168.59.103,port:9092 with correlation id 2 for 1 topic(s) Set(inputTopic)
13111 [main] INFO kafka.producer.SyncProducer - Connected to 192.168.59.103:9092 for producing
13137 [main] INFO kafka.producer.SyncProducer - Disconnecting from 192.168.59.103:9092
13161 [main] INFO kafka.producer.SyncProducer - Connected to 192.168.59.103:9092 for producing
In case this would be not enough for you, you can try to change the values of
- message.send.max.retries from the default 5 to e.g. 10 and
- topic.metadata.refresh.interval.ms to 0
It is still unclear, why it takes 2 tries i.e. 2 seconds to fetch the topic metadata…
Cheers,
Andras
> On 26 Sep 2014, at 14:43, Andras Hatvani <an...@andrashatvani.com> wrote:
>
> Hi,
>
> What does the exception mean in the following context?
>
> I've got two Kafka brokers out which the second one will be elected as leader:
>
> kafka2_1 | [2014-09-26 12:35:07,289] INFO [Kafka Server 2], started (kafka.server.KafkaServer)
> kafka2_1 | [2014-09-26 12:35:07,394] INFO New leader is 2 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
>
> Then, when I try to produce a message, I only get the following error:
>
> 23182 [main] INFO kafka.producer.async.DefaultEventHandler - Back off for 100 ms before retrying send. Remaining retries = 0
> 23286 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from broker id:0,host:192.168.59.103,port:9092 with correlation id 8 for 1 topic(s) Set(inputTopic)
> 23287 [main] INFO kafka.producer.SyncProducer - Connected to 192.168.59.103:9092 for producing
> 23300 [main] INFO kafka.producer.SyncProducer - Disconnecting from 192.168.59.103:9092
> 23301 [main] WARN kafka.producer.BrokerPartitionInfo - Error while fetching metadata [{TopicMetadata for topic inputTopic ->
> No partition metadata for topic inputTopic due to kafka.common.LeaderNotAvailableException}] for topic [inputTopic]: class kafka.common.LeaderNotAvailableException
> 23303 [main] ERROR kafka.producer.async.DefaultEventHandler - Failed to send requests for topics inputTopic with correlation ids in [0,8]
>
> The property metadata.broker.list contains the host:port values of both brokers.
>
> What can be wrong/missing?
> Thanks,
> Andras