Flush Messages in KafkaProducer Buffer

[Muqtafi Akhmad <mu...@traveloka.com>]

Hello guys,

I am using KafkaProducer (org.apache.kafka.clients.producer.KafkaProducer)
to send messages to Kafka broker. I set BUFFER_MEMORY_CONFIG to some MBs.
If there is a case where we need to shutdown kafka producer when there are
messages in producer's buffer, is there any way to flush these messages to
kafka to prevent event loss? will be calling the close() method do the
trick?

Thank you,

-- 
Muqtafi Akhmad
Software Engineer
Traveloka

Re: Flush Messages in KafkaProducer Buffer

[Guozhang Wang <wa...@gmail.com>]

0.9.0 producers are not compatible with 0.8.2 brokers, so you need to
upgrade the brokers as well. You can find the upgrade guidance here:

http://kafka.apache.org/documentation.html#upgrade

Key point is to first upgrade the in-between broker protocol first before
upgrading the broker and producer versions.

When exception is thrown it usually means the failure is not recoverable
(e.g. brokers is not available for a long time), otherwise it will be
automatically handled by the retrying mechanism as long as the closing
timeout is long enough. If you still want to keep the messages that are
failed to send while closing, you would then need to buffer them yourselves.

There are some discussions in adding persistent buffering on producer for
long-running failures such as broker un-availability:

https://issues.apache.org/jira/browse/KAFKA-1955

Guozhang

On Wed, Nov 25, 2015 at 7:59 PM, Muqtafi Akhmad <mu...@traveloka.com>
wrote:

> Hello Gwen,
>
> Currently I am using 0.8.2, is it recommended to upgrade to 0.9.0? does it
> will backward compatible to 0.8.2 brokers?
> Is there any way to 'recover' buffered messages in case there is exception
> when calling the close() method?
>
> Thank you,
>
> On Thu, Nov 26, 2015 at 8:26 AM, Gwen Shapira <gw...@confluent.io> wrote:
>
> > In 0.9.0, close() has a timeout parameter that allows specifying how long
> > to wait for the in-flight messages to complete (definition of complete
> > depends on value of "acks" parameter).
> >
> > On Wed, Nov 25, 2015 at 3:58 AM, Muqtafi Akhmad <mu...@traveloka.com>
> > wrote:
> >
> > > Hello guys,
> > >
> > > I am using KafkaProducer
> > (org.apache.kafka.clients.producer.KafkaProducer)
> > > to send messages to Kafka broker. I set BUFFER_MEMORY_CONFIG to some
> MBs.
> > > If there is a case where we need to shutdown kafka producer when there
> > are
> > > messages in producer's buffer, is there any way to flush these messages
> > to
> > > kafka to prevent event loss? will be calling the close() method do the
> > > trick?
> > >
> > > Thank you,
> > >
> > > --
> > > Muqtafi Akhmad
> > > Software Engineer
> > > Traveloka
> > >
> >
>
>
>
> --
> Muqtafi Akhmad
> Software Engineer
> Traveloka
>



-- 
-- Guozhang

Re: Flush Messages in KafkaProducer Buffer

[Muqtafi Akhmad <mu...@traveloka.com>]

Hello Gwen,

Currently I am using 0.8.2, is it recommended to upgrade to 0.9.0? does it
will backward compatible to 0.8.2 brokers?
Is there any way to 'recover' buffered messages in case there is exception
when calling the close() method?

Thank you,

On Thu, Nov 26, 2015 at 8:26 AM, Gwen Shapira <gw...@confluent.io> wrote:

> In 0.9.0, close() has a timeout parameter that allows specifying how long
> to wait for the in-flight messages to complete (definition of complete
> depends on value of "acks" parameter).
>
> On Wed, Nov 25, 2015 at 3:58 AM, Muqtafi Akhmad <mu...@traveloka.com>
> wrote:
>
> > Hello guys,
> >
> > I am using KafkaProducer
> (org.apache.kafka.clients.producer.KafkaProducer)
> > to send messages to Kafka broker. I set BUFFER_MEMORY_CONFIG to some MBs.
> > If there is a case where we need to shutdown kafka producer when there
> are
> > messages in producer's buffer, is there any way to flush these messages
> to
> > kafka to prevent event loss? will be calling the close() method do the
> > trick?
> >
> > Thank you,
> >
> > --
> > Muqtafi Akhmad
> > Software Engineer
> > Traveloka
> >
>



-- 
Muqtafi Akhmad
Software Engineer
Traveloka

Re: Flush Messages in KafkaProducer Buffer

[Gwen Shapira <gw...@confluent.io>]

In 0.9.0, close() has a timeout parameter that allows specifying how long
to wait for the in-flight messages to complete (definition of complete
depends on value of "acks" parameter).

On Wed, Nov 25, 2015 at 3:58 AM, Muqtafi Akhmad <mu...@traveloka.com>
wrote:

> Hello guys,
>
> I am using KafkaProducer (org.apache.kafka.clients.producer.KafkaProducer)
> to send messages to Kafka broker. I set BUFFER_MEMORY_CONFIG to some MBs.
> If there is a case where we need to shutdown kafka producer when there are
> messages in producer's buffer, is there any way to flush these messages to
> kafka to prevent event loss? will be calling the close() method do the
> trick?
>
> Thank you,
>
> --
> Muqtafi Akhmad
> Software Engineer
> Traveloka
>