You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Neeraj Vaidya <ne...@yahoo.co.in.INVALID> on 2021/05/24 23:49:13 UTC
Metadata max idle vs age ms
Hi All, (I have asked this on SO as well, but happy to paste the response there if I get a answer here or vice-versa).
I would like to know the impact of setting both of these options on the Producer API.
Can someone please let me know why metadata has not expired in the Scenario-1 even when metadata.max.age.ms has elapsed.
Scenario-1
------------
set metadata.max.age.ms to 10 seconds.
Leave metadata.max.idle.ms as default of 5 minutes.
start Kafka producer. publish some messages to a topic using this producer.
shutdown the Kafka cluster.
allow more than 10 seconds to elapse.----> I am hoping that this would cause the producer to automatically trigger a request to update Metadata.
Now,try to produce a message to the topic using KafkaProducer#send from the producer.
This operation returns immediately, without blocking.
I was expecting this to block as more than 10 seconds have elapsed and was expecting metadata to expire based on the metadata.max.age.ms being set to 10 seconds.
Scenario-2
------------
However, if do the following, then the KafkaProducer#send operation blocks :
set metadata.max.age.ms and metadata.max.idle.ms both to 10 seconds.
start Kafka producer
publish some messages to a topic using this producer.
shutdown the Kafka cluster.
allow more than 10 seconds to elapse.
try to produce a message to the topic using KafkaProducer#send from the producer.
This operation blocks trying to fetch metadata.
I am not sure why it blocks now, but not in the first scenario.
Re: Metadata max idle vs age ms
Posted by Neeraj Vaidya <ne...@yahoo.co.in.INVALID>.
Anybody has any inputs on this one?
Sent from my iPhone
> On 25 May 2021, at 9:49 am, Neeraj Vaidya <ne...@yahoo.co.in> wrote:
>
> Hi All, (I have asked this on SO as well, but happy to paste the response there if I get a answer here or vice-versa).
>
> I would like to know the impact of setting both of these options on the Producer API.
>
> Can someone please let me know why metadata has not expired in the Scenario-1 even when metadata.max.age.ms has elapsed.
>
> Scenario-1
> ------------
> set metadata.max.age.ms to 10 seconds.
> Leave metadata.max.idle.ms as default of 5 minutes.
> start Kafka producer. publish some messages to a topic using this producer.
> shutdown the Kafka cluster.
> allow more than 10 seconds to elapse.----> I am hoping that this would cause the producer to automatically trigger a request to update Metadata.
> Now,try to produce a message to the topic using KafkaProducer#send from the producer.
> This operation returns immediately, without blocking.
> I was expecting this to block as more than 10 seconds have elapsed and was expecting metadata to expire based on the metadata.max.age.ms being set to 10 seconds.
>
> Scenario-2
> ------------
> However, if do the following, then the KafkaProducer#send operation blocks :
> set metadata.max.age.ms and metadata.max.idle.ms both to 10 seconds.
> start Kafka producer
> publish some messages to a topic using this producer.
> shutdown the Kafka cluster.
> allow more than 10 seconds to elapse.
> try to produce a message to the topic using KafkaProducer#send from the producer.
> This operation blocks trying to fetch metadata.
> I am not sure why it blocks now, but not in the first scenario.