Lost message in asynchronous kafka producer

[Agostino Calamita <ag...@gmail.com>]

Hi,
I have 2 brokers with a topic with replication factor = 2.
Brokers are configured with min.insync.replicas=2.

I use a producer in asynchronous mode to send 100000 messages.

After some seconds after producer start, I stop one broker.

On producer side I got no exceptions, so for producer application is all
OK, but in broker log message I see
 ERROR [Replica Manager on Broker 0]: Error processing append operation on
partition diameternew-10 (kafka.server.ReplicaManager)

So I lost all messages after broker shutdown.

Is there a way to catch as soon as possible this kind of exeption ?

I know that I can use

     m_kafkaProducer.send(prMessage).get();

or

    Future _future = m_kafkaProducer.send(prMessage);

or callback in send method, but they are very slow.


Thank.

Re: Lost message in asynchronous kafka producer

[Damian Guy <da...@gmail.com>]

You need to use one of the options you've already outlined. Without this
you are just doing fire-and-forget.

On Fri, 16 Sep 2016 at 09:25 Agostino Calamita <ag...@gmail.com>
wrote:

> Hi,
> I have 2 brokers with a topic with replication factor = 2.
> Brokers are configured with min.insync.replicas=2.
>
> I use a producer in asynchronous mode to send 100000 messages.
>
> After some seconds after producer start, I stop one broker.
>
> On producer side I got no exceptions, so for producer application is all
> OK, but in broker log message I see
>  ERROR [Replica Manager on Broker 0]: Error processing append operation on
> partition diameternew-10 (kafka.server.ReplicaManager)
>
> So I lost all messages after broker shutdown.
>
> Is there a way to catch as soon as possible this kind of exeption ?
>
> I know that I can use
>
>      m_kafkaProducer.send(prMessage).get();
>
> or
>
>     Future _future = m_kafkaProducer.send(prMessage);
>
> or callback in send method, but they are very slow.
>
>
> Thank.
>