KafkaProducer "send" blocks on first attempt with Kafka server offline

[Paolo Patierno <pp...@live.com>]

Hello,
as documentation says, the KafkaProducer.send() method is asynchronous and it just returns immediately.I found out that it's not so true when the Kafka server it's trying to connect isn't online.Of course it happens only on the first send() method invocation. It means that if the Kafka server isn't reachable when my application starts for the first time, the send() method isn't so asynchronous but it blocks.I know that it's trying to connect to the Kafka server but why it doesn't save the message into the buffer and returns immediately ?Is it a behavior or a bug ?
Thanks,Paolo

Paolo PatiernoSenior Software Engineer
 

Windows Embedded & IoTMicrosoft Azure Advisor 
Twitter : @ppatierno
Linkedin : paolopatierno
Blog : DevExperienceBlog : Embedded101
 		 	   		  

Re: KafkaProducer "send" blocks on first attempt with Kafka server offline

[Steven Wu <st...@gmail.com>]

Oleg, I believe 0.9 producer gave you the control "max.block.ms" now

On Wed, Mar 30, 2016 at 5:31 AM, Oleg Zhurakousky <
ozhurakousky@hortonworks.com> wrote:

> I'll buy both 'back pressure' and 'block' argument, but what does it have
> to do with the Future? Isn't that the main point of the Future - a
> reference to an invocation that may or may not occur some time in the
> future? Isn't that the purpose of the Future.get(..)  to give user a choice
> and chance to wait for completion, and that is where blocking and back
> pressure is handled.
>
> The scary part is that not only send() can block indefinitely, if used
> within users code it creates a deadlock in users code unless user does
> something to avoid that. And what some of us do is invoke send() as
> Callable returning yet another Future. Doesn't that seem awkward? Please
> let me know if there is another way to deal with infinite block.
>
> Furthermore, the problem is similar to the one in consumer API where
> Iterator was mapped to network service and hasNext() is also a blocking
> call, but at least unlike send() there is a workaround property for
> hasNext(). Could at least the same workaround be introduced to send()?
>
> Cheers
> Oleg
>
> > On Mar 29, 2016, at 23:12, Dana Powers <da...@gmail.com> wrote:
> >
> > Somewhat of an aside, but you might note that the kafka producer is
> > intended to block during send() as backpressure on memory allocation.
> > This is admittedly different than blocking on metadata, but it is worth
> > considering that the premise that send() should *never* block because
> > it returns a Future seems fundamentally at odds with the current design.
> >
> > In any event, there is a configuration that you can tweak to set the max
> > time the producer will spend blocking in send(): max.block.ms
> >
> > -Dana
> >
> >
> >> On Tue, Mar 29, 2016 at 7:26 PM, Steven Wu <st...@gmail.com>
> wrote:
> >>
> >> I also agree that returning a Future should never block. I have brought
> >> this up when 0.8.2 was first released for new Java  producer.
> >>
> >> As Oleg said, KafkaProducer can also block if metadata is not fetched.
> This
> >> is probably more often than offline broker, because metadata is loaded
> >> lazily when there is a first send attempt for the topic. In another
> word,
> >> first send to a topic will always block until metadata is fetched or
> timed
> >> out.
> >>
> >> We had to handle this in our wrapper code. Basically, we check if
> metadata
> >> not available, we put msg into a queue and drain the queue from a diff
> >> thread.
> >>
> >> Thanks,
> >> Steven
> >>
> >>
> >> On Tue, Mar 29, 2016 at 4:59 AM, Oleg Zhurakousky <
> >> ozhurakousky@hortonworks.com> wrote:
> >>
> >>> I agree and considering that send(..) method returns Future one would
> >>> argue it must never block, otherwise what’s the point of returning
> Future
> >>> if you remove user’s ability to control how long are they willing to
> wait
> >>> and what to do when certain types of exception arise. Nevertheless it
> >> does
> >>> and it happens in the very first line of code:
> >>> // first make sure the metadata for the topic is available
> >>> waitOnMetadata(record.topic(), this.metadataFetchTimeoutMs);
> >>> So I am curious as well as to what is the motivation for it as we’ve
> seen
> >>> the same symptoms multiple times.
> >>> Cheers
> >>> Oleg
> >>> On Mar 29, 2016, at 4:31 AM, Paolo Patierno <ppatierno@live.com
> <mailto:
> >>> ppatierno@live.com>> wrote:
> >>>
> >>> Hello,
> >>> as documentation says, the KafkaProducer.send() method is asynchronous
> >> and
> >>> it just returns immediately.I found out that it's not so true when the
> >>> Kafka server it's trying to connect isn't online.Of course it happens
> >> only
> >>> on the first send() method invocation. It means that if the Kafka
> server
> >>> isn't reachable when my application starts for the first time, the
> send()
> >>> method isn't so asynchronous but it blocks.I know that it's trying to
> >>> connect to the Kafka server but why it doesn't save the message into
> the
> >>> buffer and returns immediately ?Is it a behavior or a bug ?
> >>> Thanks,Paolo
> >>>
> >>> Paolo PatiernoSenior Software Engineer
> >>>
> >>>
> >>> Windows Embedded & IoTMicrosoft Azure Advisor
> >>> Twitter : @ppatierno
> >>> Linkedin : paolopatierno
> >>> Blog : DevExperienceBlog : Embedded101
> >>
>

Re: KafkaProducer "send" blocks on first attempt with Kafka server offline

[Oleg Zhurakousky <oz...@hortonworks.com>]

I'll buy both 'back pressure' and 'block' argument, but what does it have to do with the Future? Isn't that the main point of the Future - a reference to an invocation that may or may not occur some time in the future? Isn't that the purpose of the Future.get(..)  to give user a choice and chance to wait for completion, and that is where blocking and back pressure is handled.

The scary part is that not only send() can block indefinitely, if used within users code it creates a deadlock in users code unless user does something to avoid that. And what some of us do is invoke send() as Callable returning yet another Future. Doesn't that seem awkward? Please let me know if there is another way to deal with infinite block. 

Furthermore, the problem is similar to the one in consumer API where Iterator was mapped to network service and hasNext() is also a blocking call, but at least unlike send() there is a workaround property for  hasNext(). Could at least the same workaround be introduced to send()?

Cheers 
Oleg

> On Mar 29, 2016, at 23:12, Dana Powers <da...@gmail.com> wrote:
> 
> Somewhat of an aside, but you might note that the kafka producer is
> intended to block during send() as backpressure on memory allocation.
> This is admittedly different than blocking on metadata, but it is worth
> considering that the premise that send() should *never* block because
> it returns a Future seems fundamentally at odds with the current design.
> 
> In any event, there is a configuration that you can tweak to set the max
> time the producer will spend blocking in send(): max.block.ms
> 
> -Dana
> 
> 
>> On Tue, Mar 29, 2016 at 7:26 PM, Steven Wu <st...@gmail.com> wrote:
>> 
>> I also agree that returning a Future should never block. I have brought
>> this up when 0.8.2 was first released for new Java  producer.
>> 
>> As Oleg said, KafkaProducer can also block if metadata is not fetched. This
>> is probably more often than offline broker, because metadata is loaded
>> lazily when there is a first send attempt for the topic. In another word,
>> first send to a topic will always block until metadata is fetched or timed
>> out.
>> 
>> We had to handle this in our wrapper code. Basically, we check if metadata
>> not available, we put msg into a queue and drain the queue from a diff
>> thread.
>> 
>> Thanks,
>> Steven
>> 
>> 
>> On Tue, Mar 29, 2016 at 4:59 AM, Oleg Zhurakousky <
>> ozhurakousky@hortonworks.com> wrote:
>> 
>>> I agree and considering that send(..) method returns Future one would
>>> argue it must never block, otherwise what’s the point of returning Future
>>> if you remove user’s ability to control how long are they willing to wait
>>> and what to do when certain types of exception arise. Nevertheless it
>> does
>>> and it happens in the very first line of code:
>>> // first make sure the metadata for the topic is available
>>> waitOnMetadata(record.topic(), this.metadataFetchTimeoutMs);
>>> So I am curious as well as to what is the motivation for it as we’ve seen
>>> the same symptoms multiple times.
>>> Cheers
>>> Oleg
>>> On Mar 29, 2016, at 4:31 AM, Paolo Patierno <ppatierno@live.com<mailto:
>>> ppatierno@live.com>> wrote:
>>> 
>>> Hello,
>>> as documentation says, the KafkaProducer.send() method is asynchronous
>> and
>>> it just returns immediately.I found out that it's not so true when the
>>> Kafka server it's trying to connect isn't online.Of course it happens
>> only
>>> on the first send() method invocation. It means that if the Kafka server
>>> isn't reachable when my application starts for the first time, the send()
>>> method isn't so asynchronous but it blocks.I know that it's trying to
>>> connect to the Kafka server but why it doesn't save the message into the
>>> buffer and returns immediately ?Is it a behavior or a bug ?
>>> Thanks,Paolo
>>> 
>>> Paolo PatiernoSenior Software Engineer
>>> 
>>> 
>>> Windows Embedded & IoTMicrosoft Azure Advisor
>>> Twitter : @ppatierno
>>> Linkedin : paolopatierno
>>> Blog : DevExperienceBlog : Embedded101
>> 

Re: KafkaProducer "send" blocks on first attempt with Kafka server offline

[Dana Powers <da...@gmail.com>]

Somewhat of an aside, but you might note that the kafka producer is
intended to block during send() as backpressure on memory allocation.
This is admittedly different than blocking on metadata, but it is worth
considering that the premise that send() should *never* block because
it returns a Future seems fundamentally at odds with the current design.

In any event, there is a configuration that you can tweak to set the max
time the producer will spend blocking in send(): max.block.ms

-Dana


On Tue, Mar 29, 2016 at 7:26 PM, Steven Wu <st...@gmail.com> wrote:

> I also agree that returning a Future should never block. I have brought
> this up when 0.8.2 was first released for new Java  producer.
>
> As Oleg said, KafkaProducer can also block if metadata is not fetched. This
> is probably more often than offline broker, because metadata is loaded
> lazily when there is a first send attempt for the topic. In another word,
> first send to a topic will always block until metadata is fetched or timed
> out.
>
> We had to handle this in our wrapper code. Basically, we check if metadata
> not available, we put msg into a queue and drain the queue from a diff
> thread.
>
> Thanks,
> Steven
>
>
> On Tue, Mar 29, 2016 at 4:59 AM, Oleg Zhurakousky <
> ozhurakousky@hortonworks.com> wrote:
>
> > I agree and considering that send(..) method returns Future one would
> > argue it must never block, otherwise what’s the point of returning Future
> > if you remove user’s ability to control how long are they willing to wait
> > and what to do when certain types of exception arise. Nevertheless it
> does
> > and it happens in the very first line of code:
> > // first make sure the metadata for the topic is available
> > waitOnMetadata(record.topic(), this.metadataFetchTimeoutMs);
> > So I am curious as well as to what is the motivation for it as we’ve seen
> > the same symptoms multiple times.
> > Cheers
> > Oleg
> > On Mar 29, 2016, at 4:31 AM, Paolo Patierno <ppatierno@live.com<mailto:
> > ppatierno@live.com>> wrote:
> >
> > Hello,
> > as documentation says, the KafkaProducer.send() method is asynchronous
> and
> > it just returns immediately.I found out that it's not so true when the
> > Kafka server it's trying to connect isn't online.Of course it happens
> only
> > on the first send() method invocation. It means that if the Kafka server
> > isn't reachable when my application starts for the first time, the send()
> > method isn't so asynchronous but it blocks.I know that it's trying to
> > connect to the Kafka server but why it doesn't save the message into the
> > buffer and returns immediately ?Is it a behavior or a bug ?
> > Thanks,Paolo
> >
> > Paolo PatiernoSenior Software Engineer
> >
> >
> > Windows Embedded & IoTMicrosoft Azure Advisor
> > Twitter : @ppatierno
> > Linkedin : paolopatierno
> > Blog : DevExperienceBlog : Embedded101
> >
> >
> >
>

Re: KafkaProducer "send" blocks on first attempt with Kafka server offline

[Steven Wu <st...@gmail.com>]

I also agree that returning a Future should never block. I have brought
this up when 0.8.2 was first released for new Java  producer.

As Oleg said, KafkaProducer can also block if metadata is not fetched. This
is probably more often than offline broker, because metadata is loaded
lazily when there is a first send attempt for the topic. In another word,
first send to a topic will always block until metadata is fetched or timed
out.

We had to handle this in our wrapper code. Basically, we check if metadata
not available, we put msg into a queue and drain the queue from a diff
thread.

Thanks,
Steven


On Tue, Mar 29, 2016 at 4:59 AM, Oleg Zhurakousky <
ozhurakousky@hortonworks.com> wrote:

> I agree and considering that send(..) method returns Future one would
> argue it must never block, otherwise what’s the point of returning Future
> if you remove user’s ability to control how long are they willing to wait
> and what to do when certain types of exception arise. Nevertheless it does
> and it happens in the very first line of code:
> // first make sure the metadata for the topic is available
> waitOnMetadata(record.topic(), this.metadataFetchTimeoutMs);
> So I am curious as well as to what is the motivation for it as we’ve seen
> the same symptoms multiple times.
> Cheers
> Oleg
> On Mar 29, 2016, at 4:31 AM, Paolo Patierno <ppatierno@live.com<mailto:
> ppatierno@live.com>> wrote:
>
> Hello,
> as documentation says, the KafkaProducer.send() method is asynchronous and
> it just returns immediately.I found out that it's not so true when the
> Kafka server it's trying to connect isn't online.Of course it happens only
> on the first send() method invocation. It means that if the Kafka server
> isn't reachable when my application starts for the first time, the send()
> method isn't so asynchronous but it blocks.I know that it's trying to
> connect to the Kafka server but why it doesn't save the message into the
> buffer and returns immediately ?Is it a behavior or a bug ?
> Thanks,Paolo
>
> Paolo PatiernoSenior Software Engineer
>
>
> Windows Embedded & IoTMicrosoft Azure Advisor
> Twitter : @ppatierno
> Linkedin : paolopatierno
> Blog : DevExperienceBlog : Embedded101
>
>
>

Re: KafkaProducer "send" blocks on first attempt with Kafka server offline

[Oleg Zhurakousky <oz...@hortonworks.com>]

I agree and considering that send(..) method returns Future one would argue it must never block, otherwise what’s the point of returning Future if you remove user’s ability to control how long are they willing to wait and what to do when certain types of exception arise. Nevertheless it does and it happens in the very first line of code:
// first make sure the metadata for the topic is available
waitOnMetadata(record.topic(), this.metadataFetchTimeoutMs);
So I am curious as well as to what is the motivation for it as we’ve seen the same symptoms multiple times.
Cheers
Oleg
On Mar 29, 2016, at 4:31 AM, Paolo Patierno <pp...@live.com>> wrote:

Hello,
as documentation says, the KafkaProducer.send() method is asynchronous and it just returns immediately.I found out that it's not so true when the Kafka server it's trying to connect isn't online.Of course it happens only on the first send() method invocation. It means that if the Kafka server isn't reachable when my application starts for the first time, the send() method isn't so asynchronous but it blocks.I know that it's trying to connect to the Kafka server but why it doesn't save the message into the buffer and returns immediately ?Is it a behavior or a bug ?
Thanks,Paolo

Paolo PatiernoSenior Software Engineer


Windows Embedded & IoTMicrosoft Azure Advisor
Twitter : @ppatierno
Linkedin : paolopatierno
Blog : DevExperienceBlog : Embedded101