You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Phil Luckhurst <ph...@encycle.com> on 2016/05/04 09:12:01 UTC
RE: Metadata Request Loop?
I found that just sending one in the first 5 minutes was enough. To be more precise it needs to be within your setting for metadata.max.age.ms which by default is 5 minutes.
Phil
-----Original Message-----
From: Fumo, Vincent [mailto:Vincent_Fumo@cable.comcast.com]
Sent: 29 April 2016 15:57
To: users@kafka.apache.org
Subject: Re: Metadata Request Loop?
I'm going to try both of these ideas.
Thank you so much Phil for speaking up. I thought I was the only one with this issue. Your analysis was great. I think I can easily send a test message but would I need to send one every 5 mins? Or just in the _first_ 5 mins?
> On Apr 28, 2016, at 4:10 PM, vinay sharma <vi...@gmail.com> wrote:
>
> Can you try using 0.9.0.1 for both client and broker? I tried
> replicating the issue with both on 0.9.0.1 and i could not. I also
> tried what Phil has suggested but in my case i see only one meta data
> refresh. I have seen issues like this only for a short time when
> leader went down. Continuous occurrence of this issue i have seen only
> once where leader went down and probably there was no ISR.
>
> Vinay
>
> On Thu, Apr 28, 2016 at 9:06 AM, Phil Luckhurst
> <ph...@encycle.com>
> wrote:
>
>> I see from your debug messages it's the kafka-producer-network-thread
>> that is doing this so you might be hitting the same issue as I had in
>> https://www.mail-archive.com/users%40kafka.apache.org/msg18562.html.
>>
>> I was creating a producer when my application started but if it
>> didn't publish a message within the first 5 minutes then I also saw a
>> metadata request every 100ms.
>>
>> Regards,
>> Phil Luckhurst
>>
>> -----Original Message-----
>> From: Fumo, Vincent [mailto:Vincent_Fumo@cable.comcast.com]
>> Sent: 27 April 2016 20:48
>> To: users@kafka.apache.org
>> Subject: Re: Metadata Request Loop?
>>
>> client/producer : 0.9.0.1
>> server/broker : 0.9.0.0
>>
>>> On Apr 26, 2016, at 10:05 PM, Sharma Vinay <vi...@gmail.com>
>> wrote:
>>>
>>> What versions of kafka and client API are you using now?
>>> On Apr 26, 2016 3:35 PM, "Fumo, Vincent"
>>> <Vi...@cable.comcast.com>
>>> wrote:
>>>
>>>> I spoke too soon. It's back doing the same thing.. which is really odd.
>>>>
>>>>> On Apr 26, 2016, at 12:24 PM, Fumo, Vincent <
>>>> Vincent_Fumo@cable.comcast.com> wrote:
>>>>>
>>>>> Hello. I found the issue. The Ops team deployed kafka 0.8.1 and
>>>>> all my
>>>> code was 0.9.0. Simple mistake and one that I should have thought
>>>> of sooner. Once I had them bump up to the latest kafka all was well.
>>>> Thank you for your help!
>>>>>
>>>>> v
>>>>>
>>>>>
>>>>>> On Apr 25, 2016, at 2:54 PM, vinay sharma
>>>>>> <vi...@gmail.com>
>>>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Your code looks good. i don't see any reason there for frequent
>>>>>> meta
>>>> data
>>>>>> fetch but i will run it to verify.
>>>>>>
>>>>>> I was able to replicate this issue with my consumer today where I
>>>> killed 2
>>>>>> brokers out or 3 and ran consumer. I saw a lot of meta data
>>>>>> requests in wait for new leader for partitions for which those 2
>>>>>> brokers were
>>>> leader. i
>>>>>> see below 2 lines frequently as you mentioned in logs
>>>>>>
>>>>>> [Test-KafkaPoller-1] 04/25/16 13:12:28 DEBUG Metadata:172 -
>>>>>> Updated
>>>> cluster
>>>>>> metadata version 275 to Cluster(nodes = [Node(1,
>>>>>> node1.example.com,
>>>> 9093)],
>>>>>> partitions = [Partition(topic = kafkaPOCTopic, partition = 1,
>>>>>> leader =
>>>> 1,
>>>>>> replicas = [1,], isr = [1,], Partition(topic = kafkaPOCTopic,
>>>>>> partition
>>>> =
>>>>>> 2, leader = none, replicas = [], isr = [], Partition(topic =
>>>> kafkaPOCTopic,
>>>>>> partition = 0, leader = none, replicas = [], isr = []])
>>>>>>
>>>>>> [Test-KafkaPoller-1] 04/25/16 13:12:28 DEBUG Fetcher:453 - Leader
>>>>>> for partition kafkaPOCTopic-0 unavailable for fetching offset,
>>>>>> wait for metadata refresh
>>>>>>
>>>>>> I also saw this behavior due to below error where i might have
>>>>>> killed
>>>> the
>>>>>> only ISR available for the topic at that time.
>>>>>>
>>>>>> "Error while fetching metadata with correlation id 242 :
>>>>>> {kafkaPOCTopic=LEADER_NOT_AVAILABLE}"
>>>>>>
>>>>>> Do you see any such error in your logs?
>>>>>>
>>>>>> Regards,
>>>>>> Vinay Sharma
>>>>>>
>>>>>>
>>>>>> On Mon, Apr 25, 2016 at 9:38 AM, Fumo, Vincent <
>>>>>> Vincent_Fumo@cable.comcast.com> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> My code is very straightforward. I create a producer, and then
>>>>>>> call it
>>>> to
>>>>>>> send messages. Here is the factory method::
>>>>>>>
>>>>>>> public Producer<String, String> createProducer() {
>>>>>>>
>>>>>>> Properties props = new Properties();
>>>>>>>
>>>>>>> props.put("bootstrap.servers",
>>>>>>> "cmp-arch-kafka-01d.cc.com:9092");
>>>>>>> props.put("key.serializer",
>>>>>>> "org.apache.kafka.common.serialization.StringSerializer");
>>>>>>> props.put("value.serializer",
>>>>>>> "org.apache.kafka.common.serialization.StringSerializer");
>>>>>>> props.put("acks", "all");
>>>>>>> props.put("retries", "0");
>>>>>>> props.put("batch.size", "1638"); props.put("linger.ms", "1");
>>>>>>> props.put("buffer.memory", "33554432");
>>>>>>> props.put("compression.type", "gzip"); props.put("client.id",
>>>>>>> "sds-merdevl");
>>>>>>>
>>>>>>> return new KafkaProducer<>(props); }
>>>>>>>
>>>>>>> That is injected into a single instance of KafkaNotifier::
>>>>>>>
>>>>>>> public class KafkaNotifier implements MessageNotifier { private
>>>>>>> static Logger logger =
>>>>>>> LoggerFactory.getLogger(KafkaNotifier.class);
>>>>>>>
>>>>>>> private Producer<String, String> producer; private String
>>>>>>> topicName;
>>>>>>>
>>>>>>> @Override
>>>>>>> public void
>>>>>>> sendNotificationMessage(DataObjectModificationMessage
>>>>>>> message) {
>>>>>>>
>>>>>>> String json = message.asJSON();
>>>>>>> String key = message.getObject().getId().toString();
>>>>>>>
>>>>>>> producer.send(new ProducerRecord<>(topicName, key, json));
>>>>>>>
>>>>>>> if (logger.isDebugEnabled()) {
>>>>>>> logger.debug("Sent Kafka message for object with id={}",
>>>> key);
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>> @Required
>>>>>>> public void setProducer(Producer<String, String> producer) {
>>>>>>> this.producer = producer;
>>>>>>> }
>>>>>>>
>>>>>>> @Required
>>>>>>> public void setTopicName(String topicName) {
>>>>>>> this.topicName = topicName;
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>> Here is the topic config info :
>>>>>>>
>>>>>>> ./kafka-topics.sh -zookeeper mer-arch-zk-01d.cc.com:2181
>>>>>>> --describe --topic s.notifications.dev
>>>>>>> Topic:s.notifications.dev PartitionCount:1
>>>>>>> ReplicationFactor:1 Configs:retention.ms
>>>>>>> =172800000,cleanup.policy=compact
>>>>>>> Topic: s.notifications.dev Partition: 0 Leader: 0
>>>>>>> Replicas: 0 Isr: 0
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On Apr 22, 2016, at 6:27 PM, vinay sharma
>>>>>>>> <vi...@gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> 2 producer's to same topic should not be a problem. There can
>>>>>>>> be
>>>> multiple
>>>>>>>> producers and consumers of same kafka topic.
>>>>>>>>
>>>>>>>> I am not sure what can be wrong here. I can this at my end If
>>>>>>>> you can
>>>>>>> share
>>>>>>>> producer code and any config of topic ot broker that you
>>>>>>>> changed and
>>>> is
>>>>>>> not
>>>>>>>> default.
>>>>>>>>
>>>>>>>> Please also check that you are not creating producer every time
>>>>>>>> you
>>>> send
>>>>>>> a
>>>>>>>> message but reusing producer once created to send multiple messages.
>>>>>>> Before
>>>>>>>> a send and after creation a producers sends this request to
>>>>>>>> fetch
>>>>>>> metadata.
>>>>>>>> I ran a test to publish messages from 2 producers to a topic
>>>>>>>> with
>>>>>>>> 3 partitions on a 3 broker 1 zookeeper kafka setup. I ran test
>>>>>>>> for more
>>>>>>> than
>>>>>>>> a minute and saw just once for both producers before their 1st send.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Vinay Sharma
>>>>>>>> On Apr 22, 2016 3:15 PM, "Fumo, Vincent" <
>>>> Vincent_Fumo@cable.comcast.com
>>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi. I've not set that value. My producer properties are as
>>>>>>>>> follows
>> :
>>>>>>>>>
>>>>>>>>> acks=all
>>>>>>>>> retries=0
>>>>>>>>> bath.size=1638
>>>>>>>>> linger.ms=1
>>>>>>>>> buffer.memory=33554432
>>>>>>>>> compression.type=gzip
>>>>>>>>> client.id=sds-merdevl
>>>>>>>>>
>>>>>>>>> I have this running on two hosts with the same config. I
>>>>>>>>> thought that having the same client.id on each would just
>>>>>>>>> consolidate the
>>>> tracking
>>>>>>>>> (same logical name). You don't think there is an issue with 2
>>>> producers
>>>>>>> to
>>>>>>>>> the same topic?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Apr 22, 2016, at 3:05 PM, vinay sharma
>>>>>>>>>> <vinsharma.tech@gmail.com
>>>>>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Generally a proactive metadata refresh request is sent by
>>>>>>>>>> producer
>>>> and
>>>>>>>>>> consumer every 5 minutes but this interval can be overriden
>>>>>>>>>> with
>>>>>>>>> property "
>>>>>>>>>> metadata.max.age.ms" which has default value 300000 i.e 5
>> minutes.
>>>>>>>>> Check if
>>>>>>>>>> you have set this property very low in your producer?
>>>>>>>>>>
>>>>>>>>>> On Fri, Apr 22, 2016 at 11:46 AM, Fumo, Vincent <
>>>>>>>>>> Vincent_Fumo@cable.comcast.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> I'm testing a kafka install and using the java client. I
>>>>>>>>>>> have a
>>>> topic
>>>>>>>>> set
>>>>>>>>>>> up and it appears to work great, but after a while I noticed
>>>>>>>>>>> my log starting to fill up with what appears to be some kind
>>>>>>>>>>> of loop for
>>>>>>>>> metadata
>>>>>>>>>>> updates.
>>>>>>>>>>>
>>>>>>>>>>> example::
>>>>>>>>>>>
>>>>>>>>>>> 2016-04-22 15:43:55,139 DEBUG s=s-root_out env="md"
>>>>>>>>>>> [kafka-producer-network-thread | sds-merdevl]
>>>>>>>>>>> org.apache.kafka.clients.Metadata: Updated cluster metadata
>>>>>>>>>>> version
>>>>>>>>> 6196 to
>>>>>>>>>>> Cluster(nodes = [Node(0, cmp-arch-kafka-01d.cc.com, 9092)],
>>>>>>> partitions
>>>>>>>>> =
>>>>>>>>>>> [Partition(topic = s.notifications.dev, partition = 0,
>>>>>>>>>>> leader = 0,
>>>>>>>>> replicas
>>>>>>>>>>> = [0,], isr = [0,]])
>>>>>>>>>>> 2016-04-22 15:43:55,240 DEBUG s=s-root_out env="md"
>>>>>>>>>>> [kafka-producer-network-thread | sds-merdevl]
>>>>>>>>>>> org.apache.kafka.clients.Metadata: Updated cluster metadata
>>>>>>>>>>> version
>>>>>>>>> 6197 to
>>>>>>>>>>> Cluster(nodes = [Node(0, cmp-arch-kafka-01d.cc.com, 9092)],
>>>>>>> partitions
>>>>>>>>> =
>>>>>>>>>>> [Partition(topic = s.notifications.dev, partition = 0,
>>>>>>>>>>> leader = 0,
>>>>>>>>> replicas
>>>>>>>>>>> = [0,], isr = [0,]])
>>>>>>>>>>> 2016-04-22 15:43:55,341 DEBUG s=s-root_out env="md"
>>>>>>>>>>> [kafka-producer-network-thread | sds-merdevl]
>>>>>>>>>>> org.apache.kafka.clients.Metadata: Updated cluster metadata
>>>>>>>>>>> version
>>>>>>>>> 6198 to
>>>>>>>>>>> Cluster(nodes = [Node(0, cmp-arch-kafka-01d.cc.com, 9092)],
>>>>>>> partitions
>>>>>>>>> =
>>>>>>>>>>> [Partition(topic = s.notifications.dev, partition = 0,
>>>>>>>>>>> leader = 0,
>>>>>>>>> replicas
>>>>>>>>>>> = [0,], isr = [0,]])
>>>>>>>>>>>
>>>>>>>>>>> etc.
>>>>>>>>>>>
>>>>>>>>>>> It hasn't stopped..
>>>>>>>>>>>
>>>>>>>>>>> I'm curious about what's going on here. Can anyone help?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>
>>