You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Bhavesh Mistry <mi...@gmail.com> on 2022/09/19 19:50:11 UTC
Metadata Refresh and TimeoutException when MAX_BLOCK_MS_CONFIG set 0
Hello Kafka Team,
We have an environment where Kafka Broker can go down for whatever reason.
Hence, we had configured MAX_BLOCK_MS_CONFIG=0 because we wanted to drop
messages when brokers were NOT available.
Now the issue is we get data loss due to METADATA not being available and
get this exception “*Topic <topic> not present in metadata after 0 ms.”. *This
is due to the fast metadata has expired and the next request to send an
event does not have metadata.
Why does Kafka have his design? Why can’t Kafka distinguish between Broker
down vs metadata refresh not available? Is it reasonable to expect
metadata would refresh BEFORE it expires so metadata refresh doesn’t need
before it expires? Have Metadata ready before expires? Any particular
reason send() has wait for metadata refresh vs background thread that
automatically refreshes metadata before it expires, hence send() method
never incur wait().
Let me know what suggestion you have to prevent the application thread from
blocking (MAX_BLOCK_MS_CONFIG) when the Kafka brokers are DOWN vs metadata
is NOT available due to expiration.
Let me know your suggestions and what you think about metadata refresh.
Should Kafka Producer be proactively refreshing metadata intelligently
rather than what the producer does today?
Thanks,
Bhavesh
Re: Metadata Refresh and TimeoutException when MAX_BLOCK_MS_CONFIG set 0
Posted by Bhavesh Mistry <mi...@gmail.com>.
Hi Luke and Kafka Dev Team,
Any interest in making Kafka Producer non-blocking when Broker is down and
when the metadata refresh cache does not have topic details? This seems to
be a missing piece when it comes to Kafka Producer not being able to handle
state when it is really down vs metadata refresh is not available.
I hope there is enough interest to make this producer broker down vs
metadata not available.
Thanks,
Bhavesh
On Mon, Oct 10, 2022 at 4:04 PM Bhavesh Mistry <mi...@gmail.com>
wrote:
> Hi Luke,
>
> Thanks for the pointers.
>
> Sorry for being late I was out.
>
>
>
> I would like to propose the following which might be a little different
> from the Old one:
>
> Kafka Producer must distinguish between *broker down state* vs *metadata
> NOT available* for a given topic.
>
>
>
> Like the boot-strap server option, many applications (like ours) do not
> dynamically create topics and publish/subscribe to predefine topics. So,
> the Kafka producer can have a configuration option to “*predefine-topics*”.
> When a predefine-topic is configured, Metadata is fetched for those
> pre-defined topics before the producer is initialized. Also, these
> pre-defined topics will always guarantee that Metadata will be refreshed
> before it expires meaning (the metadata cache will expire at X time, then
> the producer should automatically fetch metadata refresh request X-(3000)
> ms so the cache will always have the latest mapping of topic partition
> states continue to fetch everyone seconds till it expires cache X-2000 and
> X-1000). This will guarantee the non-blocking behavior for pre-defined
> topics. Blocking behavior is acceptable for topics that are NOT defined
> ahead of time or dynamic topics.
>
>
>
> Another configuration we should have is to *drop-message-on-broker-down*
> (true or false), even if the metadata has expired just DROP the message
> till the broker is online. Do NOT block the application thread which puts
> stuff on the Kafka in-memory queue. Of course, the Kafka producer will
> have to keep track of all brokers and it states the in-memory data
> structure and update it periodically (when send is a success or ping to
> port (IP: port) is a success).
>
>
>
> Luke and others let me know what you think about it.
>
>
> I can write documents if there is interest in the topic.
>
>
> Thanks,
>
>
> Bhavesh
>
>
> On Sun, Sep 25, 2022 at 8:44 PM Luke Chen <sh...@gmail.com> wrote:
>
>> Hi Bhavesh,
>>
>> I understand your point.
>> There was an old KIP with the similar idea which was not accepted by the
>> community in the end.
>> Maybe you can try to bring it back to the community again, or try to
>> propose your own KIP for this idea?
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-286%3A+producer.send%28%29+should+not+block+on+metadata+update
>>
>> Thank you.
>> Luke
>>
>> On Sat, Sep 24, 2022 at 6:36 AM Bhavesh Mistry <
>> mistry.p.bhavesh@gmail.com>
>> wrote:
>>
>> > Hello Kafka Team,
>> >
>> > I would appreciate any insight into how to distinguish between Brocker
>> Down
>> > vs Metadata Refresh not available due to timing issues.
>> >
>> > Thanks,
>> >
>> > Bhavesh
>> >
>> > On Mon, Sep 19, 2022 at 12:50 PM Bhavesh Mistry <
>> > mistry.p.bhavesh@gmail.com>
>> > wrote:
>> >
>> > > Hello Kafka Team,
>> > >
>> > >
>> > >
>> > > We have an environment where Kafka Broker can go down for whatever
>> > reason.
>> > >
>> > >
>> > >
>> > > Hence, we had configured MAX_BLOCK_MS_CONFIG=0 because we wanted to
>> drop
>> > > messages when brokers were NOT available.
>> > >
>> > >
>> > >
>> > > Now the issue is we get data loss due to METADATA not being available
>> and
>> > > get this exception “*Topic <topic> not present in metadata after 0
>> ms.”.
>> > > *This is due to the fast metadata has expired and the next request to
>> > > send an event does not have metadata.
>> > >
>> > >
>> > >
>> > > Why does Kafka have his design? Why can’t Kafka distinguish between
>> > > Broker down vs metadata refresh not available? Is it reasonable to
>> > expect
>> > > metadata would refresh BEFORE it expires so metadata refresh doesn’t
>> need
>> > > before it expires? Have Metadata ready before expires? Any particular
>> > > reason send() has wait for metadata refresh vs background thread that
>> > > automatically refreshes metadata before it expires, hence send()
>> method
>> > > never incur wait().
>> > >
>> > >
>> > > Let me know what suggestion you have to prevent the application thread
>> > > from blocking (MAX_BLOCK_MS_CONFIG) when the Kafka brokers are DOWN vs
>> > > metadata is NOT available due to expiration.
>> > >
>> > >
>> > >
>> > > Let me know your suggestions and what you think about metadata
>> refresh.
>> > > Should Kafka Producer be proactively refreshing metadata intelligently
>> > > rather than what the producer does today?
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > Thanks,
>> > > Bhavesh
>> > >
>> >
>>
>
Re: Metadata Refresh and TimeoutException when MAX_BLOCK_MS_CONFIG set 0
Posted by Bhavesh Mistry <mi...@gmail.com>.
Hi Luke,
Thanks for the pointers.
Sorry for being late I was out.
I would like to propose the following which might be a little different
from the Old one:
Kafka Producer must distinguish between *broker down state* vs *metadata
NOT available* for a given topic.
Like the boot-strap server option, many applications (like ours) do not
dynamically create topics and publish/subscribe to predefine topics. So,
the Kafka producer can have a configuration option to “*predefine-topics*”.
When a predefine-topic is configured, Metadata is fetched for those
pre-defined topics before the producer is initialized. Also, these
pre-defined topics will always guarantee that Metadata will be refreshed
before it expires meaning (the metadata cache will expire at X time, then
the producer should automatically fetch metadata refresh request X-(3000)
ms so the cache will always have the latest mapping of topic partition
states continue to fetch everyone seconds till it expires cache X-2000 and
X-1000). This will guarantee the non-blocking behavior for pre-defined
topics. Blocking behavior is acceptable for topics that are NOT defined
ahead of time or dynamic topics.
Another configuration we should have is to *drop-message-on-broker-down*
(true or false), even if the metadata has expired just DROP the message
till the broker is online. Do NOT block the application thread which puts
stuff on the Kafka in-memory queue. Of course, the Kafka producer will
have to keep track of all brokers and it states the in-memory data
structure and update it periodically (when send is a success or ping to
port (IP: port) is a success).
Luke and others let me know what you think about it.
I can write documents if there is interest in the topic.
Thanks,
Bhavesh
On Sun, Sep 25, 2022 at 8:44 PM Luke Chen <sh...@gmail.com> wrote:
> Hi Bhavesh,
>
> I understand your point.
> There was an old KIP with the similar idea which was not accepted by the
> community in the end.
> Maybe you can try to bring it back to the community again, or try to
> propose your own KIP for this idea?
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-286%3A+producer.send%28%29+should+not+block+on+metadata+update
>
> Thank you.
> Luke
>
> On Sat, Sep 24, 2022 at 6:36 AM Bhavesh Mistry <mistry.p.bhavesh@gmail.com
> >
> wrote:
>
> > Hello Kafka Team,
> >
> > I would appreciate any insight into how to distinguish between Brocker
> Down
> > vs Metadata Refresh not available due to timing issues.
> >
> > Thanks,
> >
> > Bhavesh
> >
> > On Mon, Sep 19, 2022 at 12:50 PM Bhavesh Mistry <
> > mistry.p.bhavesh@gmail.com>
> > wrote:
> >
> > > Hello Kafka Team,
> > >
> > >
> > >
> > > We have an environment where Kafka Broker can go down for whatever
> > reason.
> > >
> > >
> > >
> > > Hence, we had configured MAX_BLOCK_MS_CONFIG=0 because we wanted to
> drop
> > > messages when brokers were NOT available.
> > >
> > >
> > >
> > > Now the issue is we get data loss due to METADATA not being available
> and
> > > get this exception “*Topic <topic> not present in metadata after 0
> ms.”.
> > > *This is due to the fast metadata has expired and the next request to
> > > send an event does not have metadata.
> > >
> > >
> > >
> > > Why does Kafka have his design? Why can’t Kafka distinguish between
> > > Broker down vs metadata refresh not available? Is it reasonable to
> > expect
> > > metadata would refresh BEFORE it expires so metadata refresh doesn’t
> need
> > > before it expires? Have Metadata ready before expires? Any particular
> > > reason send() has wait for metadata refresh vs background thread that
> > > automatically refreshes metadata before it expires, hence send() method
> > > never incur wait().
> > >
> > >
> > > Let me know what suggestion you have to prevent the application thread
> > > from blocking (MAX_BLOCK_MS_CONFIG) when the Kafka brokers are DOWN vs
> > > metadata is NOT available due to expiration.
> > >
> > >
> > >
> > > Let me know your suggestions and what you think about metadata refresh.
> > > Should Kafka Producer be proactively refreshing metadata intelligently
> > > rather than what the producer does today?
> > >
> > >
> > >
> > >
> > >
> > > Thanks,
> > > Bhavesh
> > >
> >
>
Re: Metadata Refresh and TimeoutException when MAX_BLOCK_MS_CONFIG set 0
Posted by Luke Chen <sh...@gmail.com>.
Hi Bhavesh,
I understand your point.
There was an old KIP with the similar idea which was not accepted by the
community in the end.
Maybe you can try to bring it back to the community again, or try to
propose your own KIP for this idea?
https://cwiki.apache.org/confluence/display/KAFKA/KIP-286%3A+producer.send%28%29+should+not+block+on+metadata+update
Thank you.
Luke
On Sat, Sep 24, 2022 at 6:36 AM Bhavesh Mistry <mi...@gmail.com>
wrote:
> Hello Kafka Team,
>
> I would appreciate any insight into how to distinguish between Brocker Down
> vs Metadata Refresh not available due to timing issues.
>
> Thanks,
>
> Bhavesh
>
> On Mon, Sep 19, 2022 at 12:50 PM Bhavesh Mistry <
> mistry.p.bhavesh@gmail.com>
> wrote:
>
> > Hello Kafka Team,
> >
> >
> >
> > We have an environment where Kafka Broker can go down for whatever
> reason.
> >
> >
> >
> > Hence, we had configured MAX_BLOCK_MS_CONFIG=0 because we wanted to drop
> > messages when brokers were NOT available.
> >
> >
> >
> > Now the issue is we get data loss due to METADATA not being available and
> > get this exception “*Topic <topic> not present in metadata after 0 ms.”.
> > *This is due to the fast metadata has expired and the next request to
> > send an event does not have metadata.
> >
> >
> >
> > Why does Kafka have his design? Why can’t Kafka distinguish between
> > Broker down vs metadata refresh not available? Is it reasonable to
> expect
> > metadata would refresh BEFORE it expires so metadata refresh doesn’t need
> > before it expires? Have Metadata ready before expires? Any particular
> > reason send() has wait for metadata refresh vs background thread that
> > automatically refreshes metadata before it expires, hence send() method
> > never incur wait().
> >
> >
> > Let me know what suggestion you have to prevent the application thread
> > from blocking (MAX_BLOCK_MS_CONFIG) when the Kafka brokers are DOWN vs
> > metadata is NOT available due to expiration.
> >
> >
> >
> > Let me know your suggestions and what you think about metadata refresh.
> > Should Kafka Producer be proactively refreshing metadata intelligently
> > rather than what the producer does today?
> >
> >
> >
> >
> >
> > Thanks,
> > Bhavesh
> >
>
Re: Metadata Refresh and TimeoutException when MAX_BLOCK_MS_CONFIG set 0
Posted by Bhavesh Mistry <mi...@gmail.com>.
Hello Kafka Team,
I would appreciate any insight into how to distinguish between Brocker Down
vs Metadata Refresh not available due to timing issues.
Thanks,
Bhavesh
On Mon, Sep 19, 2022 at 12:50 PM Bhavesh Mistry <mi...@gmail.com>
wrote:
> Hello Kafka Team,
>
>
>
> We have an environment where Kafka Broker can go down for whatever reason.
>
>
>
> Hence, we had configured MAX_BLOCK_MS_CONFIG=0 because we wanted to drop
> messages when brokers were NOT available.
>
>
>
> Now the issue is we get data loss due to METADATA not being available and
> get this exception “*Topic <topic> not present in metadata after 0 ms.”.
> *This is due to the fast metadata has expired and the next request to
> send an event does not have metadata.
>
>
>
> Why does Kafka have his design? Why can’t Kafka distinguish between
> Broker down vs metadata refresh not available? Is it reasonable to expect
> metadata would refresh BEFORE it expires so metadata refresh doesn’t need
> before it expires? Have Metadata ready before expires? Any particular
> reason send() has wait for metadata refresh vs background thread that
> automatically refreshes metadata before it expires, hence send() method
> never incur wait().
>
>
> Let me know what suggestion you have to prevent the application thread
> from blocking (MAX_BLOCK_MS_CONFIG) when the Kafka brokers are DOWN vs
> metadata is NOT available due to expiration.
>
>
>
> Let me know your suggestions and what you think about metadata refresh.
> Should Kafka Producer be proactively refreshing metadata intelligently
> rather than what the producer does today?
>
>
>
>
>
> Thanks,
> Bhavesh
>