You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Dmitry Minkovsky <dm...@gmail.com> on 2017/04/29 22:26:30 UTC
Kafka Producer fails to get metadata on first send attempt
I have a producer that fails to get metadata when it first attempts to send
a record to a certain topic. It fails on
ClusterAndWaitTime clusterAndWaitTime = waitOnMetadata(record.topic(),
record.partition(), maxBlockTimeMs); [0]
and yields:
org.apache.kafka.common.errors.TimeoutException: Failed to update
metadata after 30000 ms.
After the first attempt, however, the producer works fine.
This problem does not occur when I isolate the producer in a test.
However, when embedded in my service layer—which includes things like a
Kafka Streams application and a gRPC service—the above behavior occurs. I
played with this for a while and found that this occurs specifically when
the producer producers to another topic just before it produces to the one
that fails. If I remove the producing to the first topic, producing to the
second topic succeeds, even on the first try.
Please let me know what data I could provide to help debug this. I looked
at logs at DEBUG level for the NetworkClient but nothing stuck out at me. I
am on 0.10.2.1, but this happens on 0.10.2.0 and 0.10.1.0 as well. I am not
using topic auto creation.
Thank you,
Dmitry
[0]
https://github.com/apache/kafka/blob/0.10.2.1/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L450
Re: Kafka Producer fails to get metadata on first send attempt
Posted by Dmitry Minkovsky <dm...@gmail.com>.
I just posted a new message with all of this distilled into a simple test
and log output.
On Sat, Apr 29, 2017 at 6:53 PM, Dmitry Minkovsky <dm...@gmail.com>
wrote:
> It appears—at least according to debug logs—that the metadata request is
> sent after the metadata update times out:
>
>
> [... stack trace ommitted ...]
>
> Caused by: org.apache.kafka.common.errors.TimeoutException: Failed to
> update metadata after 5000 ms.
>
>
> [2017-04-29 18:47:24,264] DEBUG (org.apache.kafka.clients.NetworkClient:751)
> Sending metadata request (type=MetadataRequest,
> topics=session-create-requests,session-load-requests,*join-requests*) to
> node 0
>
>
> Note: *join-requests* is the topic I am attempting to produce to.
>
> [2017-04-29 18:47:24,272] DEBUG (org.apache.kafka.clients.Metadata:249)
> Updated cluster metadata version 4 to Cluster(id = zKIrpvAdStqc9xcfBUuRxw,
> nodes = [localhost:9092 (id: 0 rack: null)], partitions = [Partition(topic
> = session-load-requests, partition = 0, leader = 0, replicas = [0], isr =
> [0]), Partition(topic = session-create-requests, partition = 0, leader = 0,
> replicas = [0], isr = [0]), Partition(topic = join-requests, partition = 0,
> leader = 0, replicas = [0], isr = [0])])
>
>
> After this, sending with the producer to `join-requests` works.
>
> On Sat, Apr 29, 2017 at 6:26 PM, Dmitry Minkovsky <dm...@gmail.com>
> wrote:
>
>> I have a producer that fails to get metadata when it first attempts to
>> send a record to a certain topic. It fails on
>>
>> ClusterAndWaitTime clusterAndWaitTime =
>> waitOnMetadata(record.topic(), record.partition(), maxBlockTimeMs); [0]
>>
>> and yields:
>>
>> org.apache.kafka.common.errors.TimeoutException: Failed to update
>> metadata after 30000 ms.
>>
>> After the first attempt, however, the producer works fine.
>>
>> This problem does not occur when I isolate the producer in a test.
>>
>> However, when embedded in my service layer—which includes things like a
>> Kafka Streams application and a gRPC service—the above behavior occurs. I
>> played with this for a while and found that this occurs specifically when
>> the producer producers to another topic just before it produces to the one
>> that fails. If I remove the producing to the first topic, producing to the
>> second topic succeeds, even on the first try.
>>
>> Please let me know what data I could provide to help debug this. I looked
>> at logs at DEBUG level for the NetworkClient but nothing stuck out at me. I
>> am on 0.10.2.1, but this happens on 0.10.2.0 and 0.10.1.0 as well. I am not
>> using topic auto creation.
>>
>> Thank you,
>> Dmitry
>>
>>
>> [0] https://github.com/apache/kafka/blob/0.10.2.1/clients/src/ma
>> in/java/org/apache/kafka/clients/producer/KafkaProducer.java#L450
>>
>
>
Re: Kafka Producer fails to get metadata on first send attempt
Posted by Dmitry Minkovsky <dm...@gmail.com>.
It appears—at least according to debug logs—that the metadata request is
sent after the metadata update times out:
[... stack trace ommitted ...]
Caused by: org.apache.kafka.common.errors.TimeoutException: Failed to
update metadata after 5000 ms.
[2017-04-29 18:47:24,264] DEBUG (org.apache.kafka.clients.NetworkClient:751)
Sending metadata request (type=MetadataRequest,
topics=session-create-requests,session-load-requests,*join-requests*) to
node 0
Note: *join-requests* is the topic I am attempting to produce to.
[2017-04-29 18:47:24,272] DEBUG (org.apache.kafka.clients.Metadata:249)
Updated cluster metadata version 4 to Cluster(id = zKIrpvAdStqc9xcfBUuRxw,
nodes = [localhost:9092 (id: 0 rack: null)], partitions = [Partition(topic
= session-load-requests, partition = 0, leader = 0, replicas = [0], isr =
[0]), Partition(topic = session-create-requests, partition = 0, leader = 0,
replicas = [0], isr = [0]), Partition(topic = join-requests, partition = 0,
leader = 0, replicas = [0], isr = [0])])
After this, sending with the producer to `join-requests` works.
On Sat, Apr 29, 2017 at 6:26 PM, Dmitry Minkovsky <dm...@gmail.com>
wrote:
> I have a producer that fails to get metadata when it first attempts to
> send a record to a certain topic. It fails on
>
> ClusterAndWaitTime clusterAndWaitTime = waitOnMetadata(record.topic(),
> record.partition(), maxBlockTimeMs); [0]
>
> and yields:
>
> org.apache.kafka.common.errors.TimeoutException: Failed to update
> metadata after 30000 ms.
>
> After the first attempt, however, the producer works fine.
>
> This problem does not occur when I isolate the producer in a test.
>
> However, when embedded in my service layer—which includes things like a
> Kafka Streams application and a gRPC service—the above behavior occurs. I
> played with this for a while and found that this occurs specifically when
> the producer producers to another topic just before it produces to the one
> that fails. If I remove the producing to the first topic, producing to the
> second topic succeeds, even on the first try.
>
> Please let me know what data I could provide to help debug this. I looked
> at logs at DEBUG level for the NetworkClient but nothing stuck out at me. I
> am on 0.10.2.1, but this happens on 0.10.2.0 and 0.10.1.0 as well. I am not
> using topic auto creation.
>
> Thank you,
> Dmitry
>
>
> [0] https://github.com/apache/kafka/blob/0.10.2.1/clients/src/ma
> in/java/org/apache/kafka/clients/producer/KafkaProducer.java#L450
>