You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by xiaoyu wang <xi...@gmail.com> on 2012/08/24 02:19:57 UTC
How to minimize the number of message loss when broker goes down.
Hello,
We are using sync produce to push messages to kafka brokers. It will stop
once it receives an IOException: connection reset by peer. It seems when a
broker goes down, we lose some messages. I have reduced
"log.flush.interval" to 1, still see > 200 message loss.
I also reduced the batch.size on producer side to 10, but the message loss
is about the same.
So, what's the best way to minimize message loss on broker down?
Thanks,
-Xiaoyu
Re: How to minimize the number of message loss when broker goes down.
Posted by Taylor Gautier <tg...@tagged.com>.
Agreed! Once and only once is very difficult if not impossible. 2PC
makes the window smaller still (but does not 100% eliminate it) at a
significant performance cost.
On Fri, Aug 24, 2012 at 9:53 AM, Jun Rao <ju...@gmail.com> wrote:
> Taylor,
>
> Yes, with ack, if you get an exception, it means that the messages may or
> may not be successfully received by the broker. Producer can choose to
> resend (which could introduce dups) or ignore (which could introduce loss).
> Guaranteeing exactly once delivery is a bit hard.
>
> Thanks,
>
> Jun
>
> On Fri, Aug 24, 2012 at 9:05 AM, Taylor Gautier <tg...@tagged.com>
> wrote:
>
> > Re-sending could lead to duplicated messages however - consider the case
> > that the broker has committed the message and sent the ack but the ack is
> > either sitting in the send buffer of the broker or recv buffer of the
> > producer.
> >
> > Just saying that you will trade off a relatively small amount of loss
> for a
> > probably smaller amount of duplication using only a simple ack.
> >
> >
> > On Fri, Aug 24, 2012 at 6:49 AM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > Xiaoyu,
> > >
> > > In 0.7, we have this problem that the producer doesn't receive any ack.
> > So,
> > > syncProducer.send is considered successful as soon as the messages are
> in
> > > the socket buffer. If the broker goes down before the socket buffer is
> > > flushed, those supposedly successful messages are lost. What's worse is
> > > that the producer doesn't know this since it doesn't wait for a
> response.
> > > Such lost messages should be small. However, I am not sure how to
> > > reduce/prevent it. This issue will be addressed in 0.8, in which the
> > > producer will receive an ack. If a broker goes down in the middle of a
> > > send, the producer will get an exception and can resend.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Thu, Aug 23, 2012 at 5:19 PM, xiaoyu wang <xi...@gmail.com>
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > We are using sync produce to push messages to kafka brokers. It will
> > stop
> > > > once it receives an IOException: connection reset by peer. It seems
> > when
> > > a
> > > > broker goes down, we lose some messages. I have reduced
> > > > "log.flush.interval" to 1, still see > 200 message loss.
> > > >
> > > > I also reduced the batch.size on producer side to 10, but the message
> > > loss
> > > > is about the same.
> > > >
> > > > So, what's the best way to minimize message loss on broker down?
> > > >
> > > > Thanks,
> > > >
> > > > -Xiaoyu
> > > >
> > >
> >
>
Re: How to minimize the number of message loss when broker goes down.
Posted by Jun Rao <ju...@gmail.com>.
Taylor,
Yes, with ack, if you get an exception, it means that the messages may or
may not be successfully received by the broker. Producer can choose to
resend (which could introduce dups) or ignore (which could introduce loss).
Guaranteeing exactly once delivery is a bit hard.
Thanks,
Jun
On Fri, Aug 24, 2012 at 9:05 AM, Taylor Gautier <tg...@tagged.com> wrote:
> Re-sending could lead to duplicated messages however - consider the case
> that the broker has committed the message and sent the ack but the ack is
> either sitting in the send buffer of the broker or recv buffer of the
> producer.
>
> Just saying that you will trade off a relatively small amount of loss for a
> probably smaller amount of duplication using only a simple ack.
>
>
> On Fri, Aug 24, 2012 at 6:49 AM, Jun Rao <ju...@gmail.com> wrote:
>
> > Xiaoyu,
> >
> > In 0.7, we have this problem that the producer doesn't receive any ack.
> So,
> > syncProducer.send is considered successful as soon as the messages are in
> > the socket buffer. If the broker goes down before the socket buffer is
> > flushed, those supposedly successful messages are lost. What's worse is
> > that the producer doesn't know this since it doesn't wait for a response.
> > Such lost messages should be small. However, I am not sure how to
> > reduce/prevent it. This issue will be addressed in 0.8, in which the
> > producer will receive an ack. If a broker goes down in the middle of a
> > send, the producer will get an exception and can resend.
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Aug 23, 2012 at 5:19 PM, xiaoyu wang <xi...@gmail.com>
> > wrote:
> >
> > > Hello,
> > >
> > > We are using sync produce to push messages to kafka brokers. It will
> stop
> > > once it receives an IOException: connection reset by peer. It seems
> when
> > a
> > > broker goes down, we lose some messages. I have reduced
> > > "log.flush.interval" to 1, still see > 200 message loss.
> > >
> > > I also reduced the batch.size on producer side to 10, but the message
> > loss
> > > is about the same.
> > >
> > > So, what's the best way to minimize message loss on broker down?
> > >
> > > Thanks,
> > >
> > > -Xiaoyu
> > >
> >
>
Re: How to minimize the number of message loss when broker goes down.
Posted by Joe Stein <cr...@gmail.com>.
we reconcile our data using hadoop throughout the day to "heal" what we are
live streaming (especially for revenue generating data).
so our setup is basically (seriously high level)
server > kafka > aggregate > persist insert (happens in seconds) && server
> log file > hadoop map/reduce > re-persist update/overwrite (happens every
20-180 minutes)
so we get best of both worlds and audit our trade offs => real time
analytics of our data processing and (within the time we SLA on for
reporting) have 100% accurate data.
one thing I have been meaning to-do is to keep track of the variance of
those diffs, to-be-dos
On Fri, Aug 24, 2012 at 12:05 PM, Taylor Gautier <tg...@tagged.com>wrote:
> Re-sending could lead to duplicated messages however - consider the case
> that the broker has committed the message and sent the ack but the ack is
> either sitting in the send buffer of the broker or recv buffer of the
> producer.
>
> Just saying that you will trade off a relatively small amount of loss for a
> probably smaller amount of duplication using only a simple ack.
>
>
> On Fri, Aug 24, 2012 at 6:49 AM, Jun Rao <ju...@gmail.com> wrote:
>
> > Xiaoyu,
> >
> > In 0.7, we have this problem that the producer doesn't receive any ack.
> So,
> > syncProducer.send is considered successful as soon as the messages are in
> > the socket buffer. If the broker goes down before the socket buffer is
> > flushed, those supposedly successful messages are lost. What's worse is
> > that the producer doesn't know this since it doesn't wait for a response.
> > Such lost messages should be small. However, I am not sure how to
> > reduce/prevent it. This issue will be addressed in 0.8, in which the
> > producer will receive an ack. If a broker goes down in the middle of a
> > send, the producer will get an exception and can resend.
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Aug 23, 2012 at 5:19 PM, xiaoyu wang <xi...@gmail.com>
> > wrote:
> >
> > > Hello,
> > >
> > > We are using sync produce to push messages to kafka brokers. It will
> stop
> > > once it receives an IOException: connection reset by peer. It seems
> when
> > a
> > > broker goes down, we lose some messages. I have reduced
> > > "log.flush.interval" to 1, still see > 200 message loss.
> > >
> > > I also reduced the batch.size on producer side to 10, but the message
> > loss
> > > is about the same.
> > >
> > > So, what's the best way to minimize message loss on broker down?
> > >
> > > Thanks,
> > >
> > > -Xiaoyu
> > >
> >
>
--
/*
Joe Stein
http://www.linkedin.com/in/charmalloc
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
*/
Re: How to minimize the number of message loss when broker goes down.
Posted by Taylor Gautier <tg...@tagged.com>.
Re-sending could lead to duplicated messages however - consider the case
that the broker has committed the message and sent the ack but the ack is
either sitting in the send buffer of the broker or recv buffer of the
producer.
Just saying that you will trade off a relatively small amount of loss for a
probably smaller amount of duplication using only a simple ack.
On Fri, Aug 24, 2012 at 6:49 AM, Jun Rao <ju...@gmail.com> wrote:
> Xiaoyu,
>
> In 0.7, we have this problem that the producer doesn't receive any ack. So,
> syncProducer.send is considered successful as soon as the messages are in
> the socket buffer. If the broker goes down before the socket buffer is
> flushed, those supposedly successful messages are lost. What's worse is
> that the producer doesn't know this since it doesn't wait for a response.
> Such lost messages should be small. However, I am not sure how to
> reduce/prevent it. This issue will be addressed in 0.8, in which the
> producer will receive an ack. If a broker goes down in the middle of a
> send, the producer will get an exception and can resend.
>
> Thanks,
>
> Jun
>
> On Thu, Aug 23, 2012 at 5:19 PM, xiaoyu wang <xi...@gmail.com>
> wrote:
>
> > Hello,
> >
> > We are using sync produce to push messages to kafka brokers. It will stop
> > once it receives an IOException: connection reset by peer. It seems when
> a
> > broker goes down, we lose some messages. I have reduced
> > "log.flush.interval" to 1, still see > 200 message loss.
> >
> > I also reduced the batch.size on producer side to 10, but the message
> loss
> > is about the same.
> >
> > So, what's the best way to minimize message loss on broker down?
> >
> > Thanks,
> >
> > -Xiaoyu
> >
>
Re: How to minimize the number of message loss when broker goes down.
Posted by Jun Rao <ju...@gmail.com>.
Xiaoyu,
In 0.7, we have this problem that the producer doesn't receive any ack. So,
syncProducer.send is considered successful as soon as the messages are in
the socket buffer. If the broker goes down before the socket buffer is
flushed, those supposedly successful messages are lost. What's worse is
that the producer doesn't know this since it doesn't wait for a response.
Such lost messages should be small. However, I am not sure how to
reduce/prevent it. This issue will be addressed in 0.8, in which the
producer will receive an ack. If a broker goes down in the middle of a
send, the producer will get an exception and can resend.
Thanks,
Jun
On Thu, Aug 23, 2012 at 5:19 PM, xiaoyu wang <xi...@gmail.com> wrote:
> Hello,
>
> We are using sync produce to push messages to kafka brokers. It will stop
> once it receives an IOException: connection reset by peer. It seems when a
> broker goes down, we lose some messages. I have reduced
> "log.flush.interval" to 1, still see > 200 message loss.
>
> I also reduced the batch.size on producer side to 10, but the message loss
> is about the same.
>
> So, what's the best way to minimize message loss on broker down?
>
> Thanks,
>
> -Xiaoyu
>