You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Rendy Bambang Junior <re...@gmail.com> on 2015/05/12 11:48:26 UTC
New Producer Async - Metadata Fetch Timeout
Hi, sorry if my understanding is incorrect.
I am integrating kafka producer with application, when i try to shutdown
all kafka broker (preparing for prod env) I notice that 'send' method is
blocking.
Is new producer fetch metadata not async?
Rendy
Re: New Producer Async - Metadata Fetch Timeout
Posted by Jiangjie Qin <jq...@linkedin.com.INVALID>.
Isn’t the producer part of the application? The metadata is stored in
memory. If the application rebooted (process restarted), all the metadata
will be gone.
Jiangjie (Becket) Qin
On 5/13/15, 9:54 AM, "Mohit Gupta" <su...@gmail.com> wrote:
>I meant the producer. ( i.e. application using the producer api to push
>messages into kafka ) .
>
>On Wed, May 13, 2015 at 10:20 PM, Mayuresh Gharat <
>gharatmayuresh15@gmail.com> wrote:
>
>> By application rebooting, do you mean you bounce the brokers?
>>
>> Thanks,
>>
>> Mayuresh
>>
>> On Wed, May 13, 2015 at 4:06 AM, Mohit Gupta <
>> success.mohit.gupta@gmail.com>
>> wrote:
>>
>> > Thanks Jiangjie. This is helpful.
>> >
>> > Adding to what you have mentioned, I can think of one more scenario
>>which
>> > may not be very rare.
>> > Say, the application is rebooted and the Kafka brokers registered in
>>the
>> > producer are not reachable ( could be due to network issues or those
>> > brokers are actually down ). Since, no metadata is available the send
>> will
>> > block. Right?
>> >
>> > On Wed, May 13, 2015 at 10:51 AM, Jiangjie Qin
>><jqin@linkedin.com.invalid
>> >
>> > wrote:
>> >
>> > >
>> > > Application will not block on each metadata refresh or metadata is
>> > > expired.
>> > > Application will only be blocked when
>> > > 1. It sends the first message to a topic (only for that single
>> message),
>> > or
>> > > 2. The topic has been deleted from broker thus refreshed metadata
>>loses
>> > > the topic info (which is pretty rare).
>> > >
>> > > So I think the async here might mean a little bit different. It
>>means
>> > when
>> > > you send first message to a topic, you wait till you know the topic
>> > exist,
>> > > after that point it is async.
>> > > It is very low chance that your application will block on send. If
>>it
>> is
>> > > then something probably really went wrong and needs immediate
>> attention.
>> > >
>> > > Thanks.
>> > >
>> > > Jiangjie (Becket) Qin
>> > >
>> > > On 5/12/15, 5:08 PM, "Rendy Bambang Junior"
>><re...@gmail.com>
>> > > wrote:
>> > >
>> > > >Thank you for the clarification.
>> > > >
>> > > >I think I agree with Mohit. Sometime blocking on logging is not
>> > acceptable
>> > > >by nature of application who uses kafka.
>> > > >
>> > > >Yes it is not blocking when metadata is still available. But
>> application
>> > > >will be blocked once metada is expired.
>> > > >
>> > > >It might be handled by application, by implementing async call
>>when do
>> > > >send() and manage buffer and async timeout internally, but it makes
>> > async
>> > > >feature in kafka producer has less meaning.
>> > > >
>> > > >Sorry if my understanding is incorrect.
>> > > >
>> > > >Rendy
>> > > >On May 13, 2015 6:59 AM, "Jiangjie Qin" <jq...@linkedin.com.invalid>
>> > > wrote:
>> > > >
>> > > >> Send() will only block if the metadata is *not available* for the
>> > topic.
>> > > >> It won't block if metadata there is stale. The metadata refresh
>>is
>> > async
>> > > >> to send(). However, if you send the message to a topic for the
>>first
>> > > >>time,
>> > > >> send() will trigger a metadata refresh and block until it has
>> metadata
>> > > >>for
>> > > >> that topic.
>> > > >>
>> > > >> Jiangjie (Becket) Qin
>> > > >>
>> > > >> On 5/12/15, 12:58 PM, "Magnus Edenhill" <ma...@edenhill.se>
>>wrote:
>> > > >>
>> > > >> >I completely agree with Mohit, an application should not have to
>> know
>> > > >>or
>> > > >> >care about
>> > > >> >producer implementation internals.
>> > > >> >Given a message and its delivery constraints (produce retry
>>count
>> and
>> > > >> >timeout) the producer
>> > > >> >should hide any temporal failures until the message is
>>succesfully
>> > > >> >delivered, a permanent
>> > > >> >error is encountered or the constraints are hit.
>> > > >> >This should also include internal start up sequencing, such as
>> > metadata
>> > > >> >retrieval.
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> >2015-05-12 21:22 GMT+02:00 Mohit Gupta <
>> > success.mohit.gupta@gmail.com
>> > > >:
>> > > >> >
>> > > >> >> I could not follow the reasoning behind blocking the send
>>method
>> if
>> > > >>the
>> > > >> >> metadata is not up-to-date. Though, I see that it as per
>>design,
>> it
>> > > >> >> requires the metadata to batch the message into appropriate
>> > > >> >>topicPartition
>> > > >> >> queue. Also, if the metadata could not be updated in the
>> specified
>> > > >> >> interval, it throws an exception and the message is not
>>queued to
>> > be
>> > > >> >> retried once the brokers are up.
>> > > >> >>
>> > > >> >> Should it not be that messages are buffered in another queue (
>> > up-to
>> > > >>a
>> > > >> >> limit ) if the brokers are down and retried later?
>> > > >> >> Is it not a general use case to require producer to be
>> asynchronous
>> > > >>in
>> > > >> >>all
>> > > >> >> the scenarios?
>> > > >> >>
>> > > >> >>
>> > > >> >> On Tue, May 12, 2015 at 10:54 PM, Mayuresh Gharat <
>> > > >> >> gharatmayuresh15@gmail.com> wrote:
>> > > >> >>
>> > > >> >> > The way it works I suppose is that, the producer will do
>> > > >> >>fetchMetadata,
>> > > >> >> if
>> > > >> >> > the last fetched metadata is stale (the refresh interval has
>> > > >>expired)
>> > > >> >>or
>> > > >> >> if
>> > > >> >> > it is not able to send data to a particular broker in its
>> current
>> > > >> >> metadata
>> > > >> >> > (This might happen in some cases like if the leader moves).
>> > > >> >> >
>> > > >> >> > It cannot produce without having the right metadata.
>> > > >> >> >
>> > > >> >> > Thanks,
>> > > >> >> >
>> > > >> >> > Mayuresh
>> > > >> >> >
>> > > >> >> > On Tue, May 12, 2015 at 10:09 AM, Jiangjie Qin
>> > > >> >><jqin@linkedin.com.invalid
>> > > >> >> >
>> > > >> >> > wrote:
>> > > >> >> >
>> > > >> >> > > That¹s right. Send() will first try to get metadata of a
>> topic,
>> > > >>that
>> > > >> >> is a
>> > > >> >> > > blocking operation.
>> > > >> >> > >
>> > > >> >> > > On 5/12/15, 2:48 AM, "Rendy Bambang Junior"
>> > > >> >><re...@gmail.com>
>> > > >> >> > > wrote:
>> > > >> >> > >
>> > > >> >> > > >Hi, sorry if my understanding is incorrect.
>> > > >> >> > > >
>> > > >> >> > > >I am integrating kafka producer with application, when i
>>try
>> > to
>> > > >> >> shutdown
>> > > >> >> > > >all kafka broker (preparing for prod env) I notice that
>> 'send'
>> > > >> >>method
>> > > >> >> is
>> > > >> >> > > >blocking.
>> > > >> >> > > >
>> > > >> >> > > >Is new producer fetch metadata not async?
>> > > >> >> > > >
>> > > >> >> > > >Rendy
>> > > >> >> > >
>> > > >> >> > >
>> > > >> >> >
>> > > >> >> >
>> > > >> >> > --
>> > > >> >> > -Regards,
>> > > >> >> > Mayuresh R. Gharat
>> > > >> >> > (862) 250-7125
>> > > >> >> >
>> > > >> >>
>> > > >> >>
>> > > >> >>
>> > > >> >> --
>> > > >> >> Best Regards,
>> > > >> >>
>> > > >> >> Mohit Gupta
>> > > >> >>
>> > > >>
>> > > >>
>> > >
>> > >
>> >
>> >
>> > --
>> > Best Regards,
>> >
>> > Mohit Gupta
>> >
>>
>>
>>
>> --
>> -Regards,
>> Mayuresh R. Gharat
>> (862) 250-7125
>>
>
>
>
>--
>Best Regards,
>
>Mohit Gupta
Re: New Producer Async - Metadata Fetch Timeout
Posted by Mohit Gupta <su...@gmail.com>.
I meant the producer. ( i.e. application using the producer api to push
messages into kafka ) .
On Wed, May 13, 2015 at 10:20 PM, Mayuresh Gharat <
gharatmayuresh15@gmail.com> wrote:
> By application rebooting, do you mean you bounce the brokers?
>
> Thanks,
>
> Mayuresh
>
> On Wed, May 13, 2015 at 4:06 AM, Mohit Gupta <
> success.mohit.gupta@gmail.com>
> wrote:
>
> > Thanks Jiangjie. This is helpful.
> >
> > Adding to what you have mentioned, I can think of one more scenario which
> > may not be very rare.
> > Say, the application is rebooted and the Kafka brokers registered in the
> > producer are not reachable ( could be due to network issues or those
> > brokers are actually down ). Since, no metadata is available the send
> will
> > block. Right?
> >
> > On Wed, May 13, 2015 at 10:51 AM, Jiangjie Qin <jqin@linkedin.com.invalid
> >
> > wrote:
> >
> > >
> > > Application will not block on each metadata refresh or metadata is
> > > expired.
> > > Application will only be blocked when
> > > 1. It sends the first message to a topic (only for that single
> message),
> > or
> > > 2. The topic has been deleted from broker thus refreshed metadata loses
> > > the topic info (which is pretty rare).
> > >
> > > So I think the async here might mean a little bit different. It means
> > when
> > > you send first message to a topic, you wait till you know the topic
> > exist,
> > > after that point it is async.
> > > It is very low chance that your application will block on send. If it
> is
> > > then something probably really went wrong and needs immediate
> attention.
> > >
> > > Thanks.
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > On 5/12/15, 5:08 PM, "Rendy Bambang Junior" <re...@gmail.com>
> > > wrote:
> > >
> > > >Thank you for the clarification.
> > > >
> > > >I think I agree with Mohit. Sometime blocking on logging is not
> > acceptable
> > > >by nature of application who uses kafka.
> > > >
> > > >Yes it is not blocking when metadata is still available. But
> application
> > > >will be blocked once metada is expired.
> > > >
> > > >It might be handled by application, by implementing async call when do
> > > >send() and manage buffer and async timeout internally, but it makes
> > async
> > > >feature in kafka producer has less meaning.
> > > >
> > > >Sorry if my understanding is incorrect.
> > > >
> > > >Rendy
> > > >On May 13, 2015 6:59 AM, "Jiangjie Qin" <jq...@linkedin.com.invalid>
> > > wrote:
> > > >
> > > >> Send() will only block if the metadata is *not available* for the
> > topic.
> > > >> It won't block if metadata there is stale. The metadata refresh is
> > async
> > > >> to send(). However, if you send the message to a topic for the first
> > > >>time,
> > > >> send() will trigger a metadata refresh and block until it has
> metadata
> > > >>for
> > > >> that topic.
> > > >>
> > > >> Jiangjie (Becket) Qin
> > > >>
> > > >> On 5/12/15, 12:58 PM, "Magnus Edenhill" <ma...@edenhill.se> wrote:
> > > >>
> > > >> >I completely agree with Mohit, an application should not have to
> know
> > > >>or
> > > >> >care about
> > > >> >producer implementation internals.
> > > >> >Given a message and its delivery constraints (produce retry count
> and
> > > >> >timeout) the producer
> > > >> >should hide any temporal failures until the message is succesfully
> > > >> >delivered, a permanent
> > > >> >error is encountered or the constraints are hit.
> > > >> >This should also include internal start up sequencing, such as
> > metadata
> > > >> >retrieval.
> > > >> >
> > > >> >
> > > >> >
> > > >> >2015-05-12 21:22 GMT+02:00 Mohit Gupta <
> > success.mohit.gupta@gmail.com
> > > >:
> > > >> >
> > > >> >> I could not follow the reasoning behind blocking the send method
> if
> > > >>the
> > > >> >> metadata is not up-to-date. Though, I see that it as per design,
> it
> > > >> >> requires the metadata to batch the message into appropriate
> > > >> >>topicPartition
> > > >> >> queue. Also, if the metadata could not be updated in the
> specified
> > > >> >> interval, it throws an exception and the message is not queued to
> > be
> > > >> >> retried once the brokers are up.
> > > >> >>
> > > >> >> Should it not be that messages are buffered in another queue (
> > up-to
> > > >>a
> > > >> >> limit ) if the brokers are down and retried later?
> > > >> >> Is it not a general use case to require producer to be
> asynchronous
> > > >>in
> > > >> >>all
> > > >> >> the scenarios?
> > > >> >>
> > > >> >>
> > > >> >> On Tue, May 12, 2015 at 10:54 PM, Mayuresh Gharat <
> > > >> >> gharatmayuresh15@gmail.com> wrote:
> > > >> >>
> > > >> >> > The way it works I suppose is that, the producer will do
> > > >> >>fetchMetadata,
> > > >> >> if
> > > >> >> > the last fetched metadata is stale (the refresh interval has
> > > >>expired)
> > > >> >>or
> > > >> >> if
> > > >> >> > it is not able to send data to a particular broker in its
> current
> > > >> >> metadata
> > > >> >> > (This might happen in some cases like if the leader moves).
> > > >> >> >
> > > >> >> > It cannot produce without having the right metadata.
> > > >> >> >
> > > >> >> > Thanks,
> > > >> >> >
> > > >> >> > Mayuresh
> > > >> >> >
> > > >> >> > On Tue, May 12, 2015 at 10:09 AM, Jiangjie Qin
> > > >> >><jqin@linkedin.com.invalid
> > > >> >> >
> > > >> >> > wrote:
> > > >> >> >
> > > >> >> > > That¹s right. Send() will first try to get metadata of a
> topic,
> > > >>that
> > > >> >> is a
> > > >> >> > > blocking operation.
> > > >> >> > >
> > > >> >> > > On 5/12/15, 2:48 AM, "Rendy Bambang Junior"
> > > >> >><re...@gmail.com>
> > > >> >> > > wrote:
> > > >> >> > >
> > > >> >> > > >Hi, sorry if my understanding is incorrect.
> > > >> >> > > >
> > > >> >> > > >I am integrating kafka producer with application, when i try
> > to
> > > >> >> shutdown
> > > >> >> > > >all kafka broker (preparing for prod env) I notice that
> 'send'
> > > >> >>method
> > > >> >> is
> > > >> >> > > >blocking.
> > > >> >> > > >
> > > >> >> > > >Is new producer fetch metadata not async?
> > > >> >> > > >
> > > >> >> > > >Rendy
> > > >> >> > >
> > > >> >> > >
> > > >> >> >
> > > >> >> >
> > > >> >> > --
> > > >> >> > -Regards,
> > > >> >> > Mayuresh R. Gharat
> > > >> >> > (862) 250-7125
> > > >> >> >
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >> --
> > > >> >> Best Regards,
> > > >> >>
> > > >> >> Mohit Gupta
> > > >> >>
> > > >>
> > > >>
> > >
> > >
> >
> >
> > --
> > Best Regards,
> >
> > Mohit Gupta
> >
>
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>
--
Best Regards,
Mohit Gupta
Re: New Producer Async - Metadata Fetch Timeout
Posted by Mayuresh Gharat <gh...@gmail.com>.
By application rebooting, do you mean you bounce the brokers?
Thanks,
Mayuresh
On Wed, May 13, 2015 at 4:06 AM, Mohit Gupta <su...@gmail.com>
wrote:
> Thanks Jiangjie. This is helpful.
>
> Adding to what you have mentioned, I can think of one more scenario which
> may not be very rare.
> Say, the application is rebooted and the Kafka brokers registered in the
> producer are not reachable ( could be due to network issues or those
> brokers are actually down ). Since, no metadata is available the send will
> block. Right?
>
> On Wed, May 13, 2015 at 10:51 AM, Jiangjie Qin <jq...@linkedin.com.invalid>
> wrote:
>
> >
> > Application will not block on each metadata refresh or metadata is
> > expired.
> > Application will only be blocked when
> > 1. It sends the first message to a topic (only for that single message),
> or
> > 2. The topic has been deleted from broker thus refreshed metadata loses
> > the topic info (which is pretty rare).
> >
> > So I think the async here might mean a little bit different. It means
> when
> > you send first message to a topic, you wait till you know the topic
> exist,
> > after that point it is async.
> > It is very low chance that your application will block on send. If it is
> > then something probably really went wrong and needs immediate attention.
> >
> > Thanks.
> >
> > Jiangjie (Becket) Qin
> >
> > On 5/12/15, 5:08 PM, "Rendy Bambang Junior" <re...@gmail.com>
> > wrote:
> >
> > >Thank you for the clarification.
> > >
> > >I think I agree with Mohit. Sometime blocking on logging is not
> acceptable
> > >by nature of application who uses kafka.
> > >
> > >Yes it is not blocking when metadata is still available. But application
> > >will be blocked once metada is expired.
> > >
> > >It might be handled by application, by implementing async call when do
> > >send() and manage buffer and async timeout internally, but it makes
> async
> > >feature in kafka producer has less meaning.
> > >
> > >Sorry if my understanding is incorrect.
> > >
> > >Rendy
> > >On May 13, 2015 6:59 AM, "Jiangjie Qin" <jq...@linkedin.com.invalid>
> > wrote:
> > >
> > >> Send() will only block if the metadata is *not available* for the
> topic.
> > >> It won't block if metadata there is stale. The metadata refresh is
> async
> > >> to send(). However, if you send the message to a topic for the first
> > >>time,
> > >> send() will trigger a metadata refresh and block until it has metadata
> > >>for
> > >> that topic.
> > >>
> > >> Jiangjie (Becket) Qin
> > >>
> > >> On 5/12/15, 12:58 PM, "Magnus Edenhill" <ma...@edenhill.se> wrote:
> > >>
> > >> >I completely agree with Mohit, an application should not have to know
> > >>or
> > >> >care about
> > >> >producer implementation internals.
> > >> >Given a message and its delivery constraints (produce retry count and
> > >> >timeout) the producer
> > >> >should hide any temporal failures until the message is succesfully
> > >> >delivered, a permanent
> > >> >error is encountered or the constraints are hit.
> > >> >This should also include internal start up sequencing, such as
> metadata
> > >> >retrieval.
> > >> >
> > >> >
> > >> >
> > >> >2015-05-12 21:22 GMT+02:00 Mohit Gupta <
> success.mohit.gupta@gmail.com
> > >:
> > >> >
> > >> >> I could not follow the reasoning behind blocking the send method if
> > >>the
> > >> >> metadata is not up-to-date. Though, I see that it as per design, it
> > >> >> requires the metadata to batch the message into appropriate
> > >> >>topicPartition
> > >> >> queue. Also, if the metadata could not be updated in the specified
> > >> >> interval, it throws an exception and the message is not queued to
> be
> > >> >> retried once the brokers are up.
> > >> >>
> > >> >> Should it not be that messages are buffered in another queue (
> up-to
> > >>a
> > >> >> limit ) if the brokers are down and retried later?
> > >> >> Is it not a general use case to require producer to be asynchronous
> > >>in
> > >> >>all
> > >> >> the scenarios?
> > >> >>
> > >> >>
> > >> >> On Tue, May 12, 2015 at 10:54 PM, Mayuresh Gharat <
> > >> >> gharatmayuresh15@gmail.com> wrote:
> > >> >>
> > >> >> > The way it works I suppose is that, the producer will do
> > >> >>fetchMetadata,
> > >> >> if
> > >> >> > the last fetched metadata is stale (the refresh interval has
> > >>expired)
> > >> >>or
> > >> >> if
> > >> >> > it is not able to send data to a particular broker in its current
> > >> >> metadata
> > >> >> > (This might happen in some cases like if the leader moves).
> > >> >> >
> > >> >> > It cannot produce without having the right metadata.
> > >> >> >
> > >> >> > Thanks,
> > >> >> >
> > >> >> > Mayuresh
> > >> >> >
> > >> >> > On Tue, May 12, 2015 at 10:09 AM, Jiangjie Qin
> > >> >><jqin@linkedin.com.invalid
> > >> >> >
> > >> >> > wrote:
> > >> >> >
> > >> >> > > That¹s right. Send() will first try to get metadata of a topic,
> > >>that
> > >> >> is a
> > >> >> > > blocking operation.
> > >> >> > >
> > >> >> > > On 5/12/15, 2:48 AM, "Rendy Bambang Junior"
> > >> >><re...@gmail.com>
> > >> >> > > wrote:
> > >> >> > >
> > >> >> > > >Hi, sorry if my understanding is incorrect.
> > >> >> > > >
> > >> >> > > >I am integrating kafka producer with application, when i try
> to
> > >> >> shutdown
> > >> >> > > >all kafka broker (preparing for prod env) I notice that 'send'
> > >> >>method
> > >> >> is
> > >> >> > > >blocking.
> > >> >> > > >
> > >> >> > > >Is new producer fetch metadata not async?
> > >> >> > > >
> > >> >> > > >Rendy
> > >> >> > >
> > >> >> > >
> > >> >> >
> > >> >> >
> > >> >> > --
> > >> >> > -Regards,
> > >> >> > Mayuresh R. Gharat
> > >> >> > (862) 250-7125
> > >> >> >
> > >> >>
> > >> >>
> > >> >>
> > >> >> --
> > >> >> Best Regards,
> > >> >>
> > >> >> Mohit Gupta
> > >> >>
> > >>
> > >>
> >
> >
>
>
> --
> Best Regards,
>
> Mohit Gupta
>
--
-Regards,
Mayuresh R. Gharat
(862) 250-7125
Re: New Producer Async - Metadata Fetch Timeout
Posted by Mohit Gupta <su...@gmail.com>.
Thanks Jiangjie. This is helpful.
Adding to what you have mentioned, I can think of one more scenario which
may not be very rare.
Say, the application is rebooted and the Kafka brokers registered in the
producer are not reachable ( could be due to network issues or those
brokers are actually down ). Since, no metadata is available the send will
block. Right?
On Wed, May 13, 2015 at 10:51 AM, Jiangjie Qin <jq...@linkedin.com.invalid>
wrote:
>
> Application will not block on each metadata refresh or metadata is
> expired.
> Application will only be blocked when
> 1. It sends the first message to a topic (only for that single message), or
> 2. The topic has been deleted from broker thus refreshed metadata loses
> the topic info (which is pretty rare).
>
> So I think the async here might mean a little bit different. It means when
> you send first message to a topic, you wait till you know the topic exist,
> after that point it is async.
> It is very low chance that your application will block on send. If it is
> then something probably really went wrong and needs immediate attention.
>
> Thanks.
>
> Jiangjie (Becket) Qin
>
> On 5/12/15, 5:08 PM, "Rendy Bambang Junior" <re...@gmail.com>
> wrote:
>
> >Thank you for the clarification.
> >
> >I think I agree with Mohit. Sometime blocking on logging is not acceptable
> >by nature of application who uses kafka.
> >
> >Yes it is not blocking when metadata is still available. But application
> >will be blocked once metada is expired.
> >
> >It might be handled by application, by implementing async call when do
> >send() and manage buffer and async timeout internally, but it makes async
> >feature in kafka producer has less meaning.
> >
> >Sorry if my understanding is incorrect.
> >
> >Rendy
> >On May 13, 2015 6:59 AM, "Jiangjie Qin" <jq...@linkedin.com.invalid>
> wrote:
> >
> >> Send() will only block if the metadata is *not available* for the topic.
> >> It won't block if metadata there is stale. The metadata refresh is async
> >> to send(). However, if you send the message to a topic for the first
> >>time,
> >> send() will trigger a metadata refresh and block until it has metadata
> >>for
> >> that topic.
> >>
> >> Jiangjie (Becket) Qin
> >>
> >> On 5/12/15, 12:58 PM, "Magnus Edenhill" <ma...@edenhill.se> wrote:
> >>
> >> >I completely agree with Mohit, an application should not have to know
> >>or
> >> >care about
> >> >producer implementation internals.
> >> >Given a message and its delivery constraints (produce retry count and
> >> >timeout) the producer
> >> >should hide any temporal failures until the message is succesfully
> >> >delivered, a permanent
> >> >error is encountered or the constraints are hit.
> >> >This should also include internal start up sequencing, such as metadata
> >> >retrieval.
> >> >
> >> >
> >> >
> >> >2015-05-12 21:22 GMT+02:00 Mohit Gupta <success.mohit.gupta@gmail.com
> >:
> >> >
> >> >> I could not follow the reasoning behind blocking the send method if
> >>the
> >> >> metadata is not up-to-date. Though, I see that it as per design, it
> >> >> requires the metadata to batch the message into appropriate
> >> >>topicPartition
> >> >> queue. Also, if the metadata could not be updated in the specified
> >> >> interval, it throws an exception and the message is not queued to be
> >> >> retried once the brokers are up.
> >> >>
> >> >> Should it not be that messages are buffered in another queue ( up-to
> >>a
> >> >> limit ) if the brokers are down and retried later?
> >> >> Is it not a general use case to require producer to be asynchronous
> >>in
> >> >>all
> >> >> the scenarios?
> >> >>
> >> >>
> >> >> On Tue, May 12, 2015 at 10:54 PM, Mayuresh Gharat <
> >> >> gharatmayuresh15@gmail.com> wrote:
> >> >>
> >> >> > The way it works I suppose is that, the producer will do
> >> >>fetchMetadata,
> >> >> if
> >> >> > the last fetched metadata is stale (the refresh interval has
> >>expired)
> >> >>or
> >> >> if
> >> >> > it is not able to send data to a particular broker in its current
> >> >> metadata
> >> >> > (This might happen in some cases like if the leader moves).
> >> >> >
> >> >> > It cannot produce without having the right metadata.
> >> >> >
> >> >> > Thanks,
> >> >> >
> >> >> > Mayuresh
> >> >> >
> >> >> > On Tue, May 12, 2015 at 10:09 AM, Jiangjie Qin
> >> >><jqin@linkedin.com.invalid
> >> >> >
> >> >> > wrote:
> >> >> >
> >> >> > > That¹s right. Send() will first try to get metadata of a topic,
> >>that
> >> >> is a
> >> >> > > blocking operation.
> >> >> > >
> >> >> > > On 5/12/15, 2:48 AM, "Rendy Bambang Junior"
> >> >><re...@gmail.com>
> >> >> > > wrote:
> >> >> > >
> >> >> > > >Hi, sorry if my understanding is incorrect.
> >> >> > > >
> >> >> > > >I am integrating kafka producer with application, when i try to
> >> >> shutdown
> >> >> > > >all kafka broker (preparing for prod env) I notice that 'send'
> >> >>method
> >> >> is
> >> >> > > >blocking.
> >> >> > > >
> >> >> > > >Is new producer fetch metadata not async?
> >> >> > > >
> >> >> > > >Rendy
> >> >> > >
> >> >> > >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > -Regards,
> >> >> > Mayuresh R. Gharat
> >> >> > (862) 250-7125
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Best Regards,
> >> >>
> >> >> Mohit Gupta
> >> >>
> >>
> >>
>
>
--
Best Regards,
Mohit Gupta
Re: New Producer Async - Metadata Fetch Timeout
Posted by Jiangjie Qin <jq...@linkedin.com.INVALID>.
Application will not block on each metadata refresh or metadata is
expired.
Application will only be blocked when
1. It sends the first message to a topic (only for that single message), or
2. The topic has been deleted from broker thus refreshed metadata loses
the topic info (which is pretty rare).
So I think the async here might mean a little bit different. It means when
you send first message to a topic, you wait till you know the topic exist,
after that point it is async.
It is very low chance that your application will block on send. If it is
then something probably really went wrong and needs immediate attention.
Thanks.
Jiangjie (Becket) Qin
On 5/12/15, 5:08 PM, "Rendy Bambang Junior" <re...@gmail.com>
wrote:
>Thank you for the clarification.
>
>I think I agree with Mohit. Sometime blocking on logging is not acceptable
>by nature of application who uses kafka.
>
>Yes it is not blocking when metadata is still available. But application
>will be blocked once metada is expired.
>
>It might be handled by application, by implementing async call when do
>send() and manage buffer and async timeout internally, but it makes async
>feature in kafka producer has less meaning.
>
>Sorry if my understanding is incorrect.
>
>Rendy
>On May 13, 2015 6:59 AM, "Jiangjie Qin" <jq...@linkedin.com.invalid> wrote:
>
>> Send() will only block if the metadata is *not available* for the topic.
>> It won’t block if metadata there is stale. The metadata refresh is async
>> to send(). However, if you send the message to a topic for the first
>>time,
>> send() will trigger a metadata refresh and block until it has metadata
>>for
>> that topic.
>>
>> Jiangjie (Becket) Qin
>>
>> On 5/12/15, 12:58 PM, "Magnus Edenhill" <ma...@edenhill.se> wrote:
>>
>> >I completely agree with Mohit, an application should not have to know
>>or
>> >care about
>> >producer implementation internals.
>> >Given a message and its delivery constraints (produce retry count and
>> >timeout) the producer
>> >should hide any temporal failures until the message is succesfully
>> >delivered, a permanent
>> >error is encountered or the constraints are hit.
>> >This should also include internal start up sequencing, such as metadata
>> >retrieval.
>> >
>> >
>> >
>> >2015-05-12 21:22 GMT+02:00 Mohit Gupta <su...@gmail.com>:
>> >
>> >> I could not follow the reasoning behind blocking the send method if
>>the
>> >> metadata is not up-to-date. Though, I see that it as per design, it
>> >> requires the metadata to batch the message into appropriate
>> >>topicPartition
>> >> queue. Also, if the metadata could not be updated in the specified
>> >> interval, it throws an exception and the message is not queued to be
>> >> retried once the brokers are up.
>> >>
>> >> Should it not be that messages are buffered in another queue ( up-to
>>a
>> >> limit ) if the brokers are down and retried later?
>> >> Is it not a general use case to require producer to be asynchronous
>>in
>> >>all
>> >> the scenarios?
>> >>
>> >>
>> >> On Tue, May 12, 2015 at 10:54 PM, Mayuresh Gharat <
>> >> gharatmayuresh15@gmail.com> wrote:
>> >>
>> >> > The way it works I suppose is that, the producer will do
>> >>fetchMetadata,
>> >> if
>> >> > the last fetched metadata is stale (the refresh interval has
>>expired)
>> >>or
>> >> if
>> >> > it is not able to send data to a particular broker in its current
>> >> metadata
>> >> > (This might happen in some cases like if the leader moves).
>> >> >
>> >> > It cannot produce without having the right metadata.
>> >> >
>> >> > Thanks,
>> >> >
>> >> > Mayuresh
>> >> >
>> >> > On Tue, May 12, 2015 at 10:09 AM, Jiangjie Qin
>> >><jqin@linkedin.com.invalid
>> >> >
>> >> > wrote:
>> >> >
>> >> > > That¹s right. Send() will first try to get metadata of a topic,
>>that
>> >> is a
>> >> > > blocking operation.
>> >> > >
>> >> > > On 5/12/15, 2:48 AM, "Rendy Bambang Junior"
>> >><re...@gmail.com>
>> >> > > wrote:
>> >> > >
>> >> > > >Hi, sorry if my understanding is incorrect.
>> >> > > >
>> >> > > >I am integrating kafka producer with application, when i try to
>> >> shutdown
>> >> > > >all kafka broker (preparing for prod env) I notice that 'send'
>> >>method
>> >> is
>> >> > > >blocking.
>> >> > > >
>> >> > > >Is new producer fetch metadata not async?
>> >> > > >
>> >> > > >Rendy
>> >> > >
>> >> > >
>> >> >
>> >> >
>> >> > --
>> >> > -Regards,
>> >> > Mayuresh R. Gharat
>> >> > (862) 250-7125
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Best Regards,
>> >>
>> >> Mohit Gupta
>> >>
>>
>>
Re: New Producer Async - Metadata Fetch Timeout
Posted by Rendy Bambang Junior <re...@gmail.com>.
Thank you for the clarification.
I think I agree with Mohit. Sometime blocking on logging is not acceptable
by nature of application who uses kafka.
Yes it is not blocking when metadata is still available. But application
will be blocked once metada is expired.
It might be handled by application, by implementing async call when do
send() and manage buffer and async timeout internally, but it makes async
feature in kafka producer has less meaning.
Sorry if my understanding is incorrect.
Rendy
On May 13, 2015 6:59 AM, "Jiangjie Qin" <jq...@linkedin.com.invalid> wrote:
> Send() will only block if the metadata is *not available* for the topic.
> It won’t block if metadata there is stale. The metadata refresh is async
> to send(). However, if you send the message to a topic for the first time,
> send() will trigger a metadata refresh and block until it has metadata for
> that topic.
>
> Jiangjie (Becket) Qin
>
> On 5/12/15, 12:58 PM, "Magnus Edenhill" <ma...@edenhill.se> wrote:
>
> >I completely agree with Mohit, an application should not have to know or
> >care about
> >producer implementation internals.
> >Given a message and its delivery constraints (produce retry count and
> >timeout) the producer
> >should hide any temporal failures until the message is succesfully
> >delivered, a permanent
> >error is encountered or the constraints are hit.
> >This should also include internal start up sequencing, such as metadata
> >retrieval.
> >
> >
> >
> >2015-05-12 21:22 GMT+02:00 Mohit Gupta <su...@gmail.com>:
> >
> >> I could not follow the reasoning behind blocking the send method if the
> >> metadata is not up-to-date. Though, I see that it as per design, it
> >> requires the metadata to batch the message into appropriate
> >>topicPartition
> >> queue. Also, if the metadata could not be updated in the specified
> >> interval, it throws an exception and the message is not queued to be
> >> retried once the brokers are up.
> >>
> >> Should it not be that messages are buffered in another queue ( up-to a
> >> limit ) if the brokers are down and retried later?
> >> Is it not a general use case to require producer to be asynchronous in
> >>all
> >> the scenarios?
> >>
> >>
> >> On Tue, May 12, 2015 at 10:54 PM, Mayuresh Gharat <
> >> gharatmayuresh15@gmail.com> wrote:
> >>
> >> > The way it works I suppose is that, the producer will do
> >>fetchMetadata,
> >> if
> >> > the last fetched metadata is stale (the refresh interval has expired)
> >>or
> >> if
> >> > it is not able to send data to a particular broker in its current
> >> metadata
> >> > (This might happen in some cases like if the leader moves).
> >> >
> >> > It cannot produce without having the right metadata.
> >> >
> >> > Thanks,
> >> >
> >> > Mayuresh
> >> >
> >> > On Tue, May 12, 2015 at 10:09 AM, Jiangjie Qin
> >><jqin@linkedin.com.invalid
> >> >
> >> > wrote:
> >> >
> >> > > That¹s right. Send() will first try to get metadata of a topic, that
> >> is a
> >> > > blocking operation.
> >> > >
> >> > > On 5/12/15, 2:48 AM, "Rendy Bambang Junior"
> >><re...@gmail.com>
> >> > > wrote:
> >> > >
> >> > > >Hi, sorry if my understanding is incorrect.
> >> > > >
> >> > > >I am integrating kafka producer with application, when i try to
> >> shutdown
> >> > > >all kafka broker (preparing for prod env) I notice that 'send'
> >>method
> >> is
> >> > > >blocking.
> >> > > >
> >> > > >Is new producer fetch metadata not async?
> >> > > >
> >> > > >Rendy
> >> > >
> >> > >
> >> >
> >> >
> >> > --
> >> > -Regards,
> >> > Mayuresh R. Gharat
> >> > (862) 250-7125
> >> >
> >>
> >>
> >>
> >> --
> >> Best Regards,
> >>
> >> Mohit Gupta
> >>
>
>
Re: New Producer Async - Metadata Fetch Timeout
Posted by Jiangjie Qin <jq...@linkedin.com.INVALID>.
Send() will only block if the metadata is *not available* for the topic.
It won’t block if metadata there is stale. The metadata refresh is async
to send(). However, if you send the message to a topic for the first time,
send() will trigger a metadata refresh and block until it has metadata for
that topic.
Jiangjie (Becket) Qin
On 5/12/15, 12:58 PM, "Magnus Edenhill" <ma...@edenhill.se> wrote:
>I completely agree with Mohit, an application should not have to know or
>care about
>producer implementation internals.
>Given a message and its delivery constraints (produce retry count and
>timeout) the producer
>should hide any temporal failures until the message is succesfully
>delivered, a permanent
>error is encountered or the constraints are hit.
>This should also include internal start up sequencing, such as metadata
>retrieval.
>
>
>
>2015-05-12 21:22 GMT+02:00 Mohit Gupta <su...@gmail.com>:
>
>> I could not follow the reasoning behind blocking the send method if the
>> metadata is not up-to-date. Though, I see that it as per design, it
>> requires the metadata to batch the message into appropriate
>>topicPartition
>> queue. Also, if the metadata could not be updated in the specified
>> interval, it throws an exception and the message is not queued to be
>> retried once the brokers are up.
>>
>> Should it not be that messages are buffered in another queue ( up-to a
>> limit ) if the brokers are down and retried later?
>> Is it not a general use case to require producer to be asynchronous in
>>all
>> the scenarios?
>>
>>
>> On Tue, May 12, 2015 at 10:54 PM, Mayuresh Gharat <
>> gharatmayuresh15@gmail.com> wrote:
>>
>> > The way it works I suppose is that, the producer will do
>>fetchMetadata,
>> if
>> > the last fetched metadata is stale (the refresh interval has expired)
>>or
>> if
>> > it is not able to send data to a particular broker in its current
>> metadata
>> > (This might happen in some cases like if the leader moves).
>> >
>> > It cannot produce without having the right metadata.
>> >
>> > Thanks,
>> >
>> > Mayuresh
>> >
>> > On Tue, May 12, 2015 at 10:09 AM, Jiangjie Qin
>><jqin@linkedin.com.invalid
>> >
>> > wrote:
>> >
>> > > That¹s right. Send() will first try to get metadata of a topic, that
>> is a
>> > > blocking operation.
>> > >
>> > > On 5/12/15, 2:48 AM, "Rendy Bambang Junior"
>><re...@gmail.com>
>> > > wrote:
>> > >
>> > > >Hi, sorry if my understanding is incorrect.
>> > > >
>> > > >I am integrating kafka producer with application, when i try to
>> shutdown
>> > > >all kafka broker (preparing for prod env) I notice that 'send'
>>method
>> is
>> > > >blocking.
>> > > >
>> > > >Is new producer fetch metadata not async?
>> > > >
>> > > >Rendy
>> > >
>> > >
>> >
>> >
>> > --
>> > -Regards,
>> > Mayuresh R. Gharat
>> > (862) 250-7125
>> >
>>
>>
>>
>> --
>> Best Regards,
>>
>> Mohit Gupta
>>
Re: New Producer Async - Metadata Fetch Timeout
Posted by Magnus Edenhill <ma...@edenhill.se>.
I completely agree with Mohit, an application should not have to know or
care about
producer implementation internals.
Given a message and its delivery constraints (produce retry count and
timeout) the producer
should hide any temporal failures until the message is succesfully
delivered, a permanent
error is encountered or the constraints are hit.
This should also include internal start up sequencing, such as metadata
retrieval.
2015-05-12 21:22 GMT+02:00 Mohit Gupta <su...@gmail.com>:
> I could not follow the reasoning behind blocking the send method if the
> metadata is not up-to-date. Though, I see that it as per design, it
> requires the metadata to batch the message into appropriate topicPartition
> queue. Also, if the metadata could not be updated in the specified
> interval, it throws an exception and the message is not queued to be
> retried once the brokers are up.
>
> Should it not be that messages are buffered in another queue ( up-to a
> limit ) if the brokers are down and retried later?
> Is it not a general use case to require producer to be asynchronous in all
> the scenarios?
>
>
> On Tue, May 12, 2015 at 10:54 PM, Mayuresh Gharat <
> gharatmayuresh15@gmail.com> wrote:
>
> > The way it works I suppose is that, the producer will do fetchMetadata,
> if
> > the last fetched metadata is stale (the refresh interval has expired) or
> if
> > it is not able to send data to a particular broker in its current
> metadata
> > (This might happen in some cases like if the leader moves).
> >
> > It cannot produce without having the right metadata.
> >
> > Thanks,
> >
> > Mayuresh
> >
> > On Tue, May 12, 2015 at 10:09 AM, Jiangjie Qin <jqin@linkedin.com.invalid
> >
> > wrote:
> >
> > > That¹s right. Send() will first try to get metadata of a topic, that
> is a
> > > blocking operation.
> > >
> > > On 5/12/15, 2:48 AM, "Rendy Bambang Junior" <re...@gmail.com>
> > > wrote:
> > >
> > > >Hi, sorry if my understanding is incorrect.
> > > >
> > > >I am integrating kafka producer with application, when i try to
> shutdown
> > > >all kafka broker (preparing for prod env) I notice that 'send' method
> is
> > > >blocking.
> > > >
> > > >Is new producer fetch metadata not async?
> > > >
> > > >Rendy
> > >
> > >
> >
> >
> > --
> > -Regards,
> > Mayuresh R. Gharat
> > (862) 250-7125
> >
>
>
>
> --
> Best Regards,
>
> Mohit Gupta
>
Re: New Producer Async - Metadata Fetch Timeout
Posted by Mohit Gupta <su...@gmail.com>.
I could not follow the reasoning behind blocking the send method if the
metadata is not up-to-date. Though, I see that it as per design, it
requires the metadata to batch the message into appropriate topicPartition
queue. Also, if the metadata could not be updated in the specified
interval, it throws an exception and the message is not queued to be
retried once the brokers are up.
Should it not be that messages are buffered in another queue ( up-to a
limit ) if the brokers are down and retried later?
Is it not a general use case to require producer to be asynchronous in all
the scenarios?
On Tue, May 12, 2015 at 10:54 PM, Mayuresh Gharat <
gharatmayuresh15@gmail.com> wrote:
> The way it works I suppose is that, the producer will do fetchMetadata, if
> the last fetched metadata is stale (the refresh interval has expired) or if
> it is not able to send data to a particular broker in its current metadata
> (This might happen in some cases like if the leader moves).
>
> It cannot produce without having the right metadata.
>
> Thanks,
>
> Mayuresh
>
> On Tue, May 12, 2015 at 10:09 AM, Jiangjie Qin <jq...@linkedin.com.invalid>
> wrote:
>
> > That¹s right. Send() will first try to get metadata of a topic, that is a
> > blocking operation.
> >
> > On 5/12/15, 2:48 AM, "Rendy Bambang Junior" <re...@gmail.com>
> > wrote:
> >
> > >Hi, sorry if my understanding is incorrect.
> > >
> > >I am integrating kafka producer with application, when i try to shutdown
> > >all kafka broker (preparing for prod env) I notice that 'send' method is
> > >blocking.
> > >
> > >Is new producer fetch metadata not async?
> > >
> > >Rendy
> >
> >
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>
--
Best Regards,
Mohit Gupta
Re: New Producer Async - Metadata Fetch Timeout
Posted by Mayuresh Gharat <gh...@gmail.com>.
The way it works I suppose is that, the producer will do fetchMetadata, if
the last fetched metadata is stale (the refresh interval has expired) or if
it is not able to send data to a particular broker in its current metadata
(This might happen in some cases like if the leader moves).
It cannot produce without having the right metadata.
Thanks,
Mayuresh
On Tue, May 12, 2015 at 10:09 AM, Jiangjie Qin <jq...@linkedin.com.invalid>
wrote:
> That¹s right. Send() will first try to get metadata of a topic, that is a
> blocking operation.
>
> On 5/12/15, 2:48 AM, "Rendy Bambang Junior" <re...@gmail.com>
> wrote:
>
> >Hi, sorry if my understanding is incorrect.
> >
> >I am integrating kafka producer with application, when i try to shutdown
> >all kafka broker (preparing for prod env) I notice that 'send' method is
> >blocking.
> >
> >Is new producer fetch metadata not async?
> >
> >Rendy
>
>
--
-Regards,
Mayuresh R. Gharat
(862) 250-7125
Re: New Producer Async - Metadata Fetch Timeout
Posted by Jiangjie Qin <jq...@linkedin.com.INVALID>.
That¹s right. Send() will first try to get metadata of a topic, that is a
blocking operation.
On 5/12/15, 2:48 AM, "Rendy Bambang Junior" <re...@gmail.com>
wrote:
>Hi, sorry if my understanding is incorrect.
>
>I am integrating kafka producer with application, when i try to shutdown
>all kafka broker (preparing for prod env) I notice that 'send' method is
>blocking.
>
>Is new producer fetch metadata not async?
>
>Rendy