You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Libo Yu <yu...@hotmail.com> on 2014/06/05 15:20:32 UTC
RE: question about synchronous producer
I want to know why there will be message loss when brokers are down for too long.
I've noticed message loss when brokers are restarted during publishing. It is a sync producer with request.required.acks set to 1.
Libo
> Date: Thu, 29 May 2014 20:11:48 -0700
> Subject: Re: question about synchronous producer
> From: wangguoz@gmail.com
> To: users@kafka.apache.org
>
> Libo,
>
> That is correct. You may want to increase the retry.backoff.ms in this
> case. In practice, if the brokers are down for too long, then data loss is
> usually inevitable.
>
> Guozhang
>
>
> On Thu, May 29, 2014 at 2:55 PM, Libo Yu <yu...@hotmail.com> wrote:
>
> > Hi team,
> >
> > Assume I am using a synchronous producer and it has the following default
> > properties:
> >
> > message.send.max.retries
> > 3
> > retry.backoff.ms
> > 100
> >
> > I use java api Producer.send(message) to send a message.
> > While send() is being called, if the brokers are shutdown, what happens?
> > send() will retry 3 times with a 100ms interval and fail silently?
> > If I don't want to lose any message when the brokers are back online, what
> > should I do? Thanks.
> >
> > Libo
> >
> >
> >
> >
>
>
>
>
> --
> -- Guozhang
Re: question about synchronous producer
Posted by Guozhang Wang <wa...@gmail.com>.
I think I know what is the problem now. With ack=1 the server will respond
to the producer once the messages are appended to the log, BUT appending to
the log does not guarantee the log is flushed to disk, it may still be in
the page cache of the broker. Now if broker fails and recovers, messages
not flushed yet will be lost although the producer has got the ACK from the
broker.
With ack=-1 the server will not respond to the producer until all of the
replicas have gotten this message in their logs, again not necessarily
flushed to disk. So with all brokers down there is still a chance to lose
data, but smaller compared with ack=1.
Guozhang
On Sat, Jun 7, 2014 at 6:57 PM, Libo Yu <yu...@hotmail.com> wrote:
> Yes. I can hardly believe there is message loss in such a case so I've
> checked
> my test code very carefully. Unfortunately, I cannot provide any log
> because
> of the company policy. This is a big risk for our operation as all our
> servers
> at one data center must be rebooted regularly at the same time by system
> admins.
> So rolling start is not an option for us.
>
> > Date: Sat, 7 Jun 2014 16:02:35 -0700
> > Subject: Re: question about synchronous producer
> > From: wangguoz@gmail.com
> > To: users@kafka.apache.org
> >
> > I see. So previously you run the test as ack=1?
> >
> > Guozhang
> >
> >
> > On Sat, Jun 7, 2014 at 7:24 AM, Libo Yu <yu...@hotmail.com> wrote:
> >
> > > Hi Guozhang,
> > >
> > > The issue is not constantly to reproduce but fairly easy to reproduce.
> > > It seems there is some kind of failure that has not been captured by
> the
> > > Kafka code and no exception has been thrown.
> > >
> > > I did a new test with request.required.acks set to -1. The number of
> lost
> > > message dropped significantly but there were still message loss.
> > >
> > > > Date: Fri, 6 Jun 2014 08:12:17 -0700
> > > > Subject: Re: question about synchronous producer
> > > > From: wangguoz@gmail.com
> > > > To: users@kafka.apache.org
> > > >
> > > > Libo,
> > > >
> > > > I have double checked the code. With sync producers all failures
> should
> > > be
> > > > either thrown as exceptions or logged in warning/error log entries.
> > > >
> > > > Guozhang
> > > >
> > > >
> > > > On Thu, Jun 5, 2014 at 6:38 PM, Libo Yu <yu...@hotmail.com> wrote:
> > > >
> > > > > Not really. The issue was reported by a client. I added a lot of
> > > logging
> > > > > to make sure no exception was thrown
> > > > > from send() when the message was lost. It is not hard to reproduce.
> > > This
> > > > > is a critical issue for operation. It may
> > > > > not be possible for brokers and producers to be restarted at the
> same
> > > time.
> > > > >
> > > > > > Date: Thu, 5 Jun 2014 16:53:29 -0700
> > > > > > Subject: Re: question about synchronous producer
> > > > > > From: wangguoz@gmail.com
> > > > > > To: users@kafka.apache.org
> > > > > >
> > > > > > Libo, did you see any exception/error entries on the producer
> log?
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > >
> > > > > > On Thu, Jun 5, 2014 at 10:33 AM, Libo Yu <yu...@hotmail.com>
> > > wrote:
> > > > > >
> > > > > > > Yes. I used three sync producers with request.required.acks=1.
> I
> > > let
> > > > > them
> > > > > > > publish 2k short messages and in the process I restart all
> > > zookeeper
> > > > > and
> > > > > > > kafka processes ( 3 hosts in a cluster). Normally there will be
> > > message
> > > > > > > loss after 3 restarts. After 3 restarts, I use a consumer to
> > > retrieve
> > > > > the
> > > > > > > messages and do the verification.
> > > > > > >
> > > > > > > > Date: Thu, 5 Jun 2014 10:15:18 -0700
> > > > > > > > Subject: Re: question about synchronous producer
> > > > > > > > From: wangguoz@gmail.com
> > > > > > > > To: users@kafka.apache.org
> > > > > > > >
> > > > > > > > Libo,
> > > > > > > >
> > > > > > > > For clarification, you can use sync producer to reproduce
> this
> > > issue?
> > > > > > > >
> > > > > > > > Guozhang
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Jun 5, 2014 at 10:03 AM, Libo Yu <
> yu_libo@hotmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > When all the brokers are down the producer should retry
> for a
> > > few
> > > > > > > times
> > > > > > > > > and throw FailedToSendMessageException. And user code can
> > > catch the
> > > > > > > > > exception and retry after a backoff. However, in my tests,
> no
> > > > > > > exception was
> > > > > > > > > caught and the message was lost silently. My broker is
> 0.8.1.1
> > > and
> > > > > my
> > > > > > > > > client is 0.8.0. It is fairly easy to reproduce. Any
> insight on
> > > > > this
> > > > > > > issue?
> > > > > > > > >
> > > > > > > > > Libo
> > > > > > > > >
> > > > > > > > > > Date: Thu, 5 Jun 2014 09:05:27 -0700
> > > > > > > > > > Subject: Re: question about synchronous producer
> > > > > > > > > > From: wangguoz@gmail.com
> > > > > > > > > > To: users@kafka.apache.org
> > > > > > > > > >
> > > > > > > > > > When the producer exhausted all the retries it will drop
> the
> > > > > message
> > > > > > > on
> > > > > > > > > the
> > > > > > > > > > floor. So when the broker is down for too long there
> will be
> > > data
> > > > > > > loss.
> > > > > > > > > >
> > > > > > > > > > Guozhang
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Jun 5, 2014 at 6:20 AM, Libo Yu <
> yu_libo@hotmail.com
> > > >
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I want to know why there will be message loss when
> brokers
> > > are
> > > > > > > down for
> > > > > > > > > > > too long.
> > > > > > > > > > > I've noticed message loss when brokers are restarted
> during
> > > > > > > > > publishing. It
> > > > > > > > > > > is a sync producer with request.required.acks set to 1.
> > > > > > > > > > >
> > > > > > > > > > > Libo
> > > > > > > > > > >
> > > > > > > > > > > > Date: Thu, 29 May 2014 20:11:48 -0700
> > > > > > > > > > > > Subject: Re: question about synchronous producer
> > > > > > > > > > > > From: wangguoz@gmail.com
> > > > > > > > > > > > To: users@kafka.apache.org
> > > > > > > > > > > >
> > > > > > > > > > > > Libo,
> > > > > > > > > > > >
> > > > > > > > > > > > That is correct. You may want to increase the
> > > > > retry.backoff.ms
> > > > > > > in
> > > > > > > > > this
> > > > > > > > > > > > case. In practice, if the brokers are down for too
> long,
> > > then
> > > > > > > data
> > > > > > > > > loss
> > > > > > > > > > > is
> > > > > > > > > > > > usually inevitable.
> > > > > > > > > > > >
> > > > > > > > > > > > Guozhang
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, May 29, 2014 at 2:55 PM, Libo Yu <
> > > > > yu_libo@hotmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi team,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Assume I am using a synchronous producer and it
> has the
> > > > > > > following
> > > > > > > > > > > default
> > > > > > > > > > > > > properties:
> > > > > > > > > > > > >
> > > > > > > > > > > > > message.send.max.retries
> > > > > > > > > > > > > 3
> > > > > > > > > > > > > retry.backoff.ms
> > > > > > > > > > > > > 100
> > > > > > > > > > > > >
> > > > > > > > > > > > > I use java api Producer.send(message) to send a
> > > message.
> > > > > > > > > > > > > While send() is being called, if the brokers are
> > > shutdown,
> > > > > what
> > > > > > > > > > > happens?
> > > > > > > > > > > > > send() will retry 3 times with a 100ms interval and
> > > fail
> > > > > > > silently?
> > > > > > > > > > > > > If I don't want to lose any message when the
> brokers
> > > are
> > > > > back
> > > > > > > > > online,
> > > > > > > > > > > what
> > > > > > > > > > > > > should I do? Thanks.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Libo
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > -- Guozhang
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > -- Guozhang
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > -- Guozhang
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -- Guozhang
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
>
>
--
-- Guozhang
RE: question about synchronous producer
Posted by Libo Yu <yu...@hotmail.com>.
Yes. I can hardly believe there is message loss in such a case so I've checked
my test code very carefully. Unfortunately, I cannot provide any log because
of the company policy. This is a big risk for our operation as all our servers
at one data center must be rebooted regularly at the same time by system admins.
So rolling start is not an option for us.
> Date: Sat, 7 Jun 2014 16:02:35 -0700
> Subject: Re: question about synchronous producer
> From: wangguoz@gmail.com
> To: users@kafka.apache.org
>
> I see. So previously you run the test as ack=1?
>
> Guozhang
>
>
> On Sat, Jun 7, 2014 at 7:24 AM, Libo Yu <yu...@hotmail.com> wrote:
>
> > Hi Guozhang,
> >
> > The issue is not constantly to reproduce but fairly easy to reproduce.
> > It seems there is some kind of failure that has not been captured by the
> > Kafka code and no exception has been thrown.
> >
> > I did a new test with request.required.acks set to -1. The number of lost
> > message dropped significantly but there were still message loss.
> >
> > > Date: Fri, 6 Jun 2014 08:12:17 -0700
> > > Subject: Re: question about synchronous producer
> > > From: wangguoz@gmail.com
> > > To: users@kafka.apache.org
> > >
> > > Libo,
> > >
> > > I have double checked the code. With sync producers all failures should
> > be
> > > either thrown as exceptions or logged in warning/error log entries.
> > >
> > > Guozhang
> > >
> > >
> > > On Thu, Jun 5, 2014 at 6:38 PM, Libo Yu <yu...@hotmail.com> wrote:
> > >
> > > > Not really. The issue was reported by a client. I added a lot of
> > logging
> > > > to make sure no exception was thrown
> > > > from send() when the message was lost. It is not hard to reproduce.
> > This
> > > > is a critical issue for operation. It may
> > > > not be possible for brokers and producers to be restarted at the same
> > time.
> > > >
> > > > > Date: Thu, 5 Jun 2014 16:53:29 -0700
> > > > > Subject: Re: question about synchronous producer
> > > > > From: wangguoz@gmail.com
> > > > > To: users@kafka.apache.org
> > > > >
> > > > > Libo, did you see any exception/error entries on the producer log?
> > > > >
> > > > > Guozhang
> > > > >
> > > > >
> > > > > On Thu, Jun 5, 2014 at 10:33 AM, Libo Yu <yu...@hotmail.com>
> > wrote:
> > > > >
> > > > > > Yes. I used three sync producers with request.required.acks=1. I
> > let
> > > > them
> > > > > > publish 2k short messages and in the process I restart all
> > zookeeper
> > > > and
> > > > > > kafka processes ( 3 hosts in a cluster). Normally there will be
> > message
> > > > > > loss after 3 restarts. After 3 restarts, I use a consumer to
> > retrieve
> > > > the
> > > > > > messages and do the verification.
> > > > > >
> > > > > > > Date: Thu, 5 Jun 2014 10:15:18 -0700
> > > > > > > Subject: Re: question about synchronous producer
> > > > > > > From: wangguoz@gmail.com
> > > > > > > To: users@kafka.apache.org
> > > > > > >
> > > > > > > Libo,
> > > > > > >
> > > > > > > For clarification, you can use sync producer to reproduce this
> > issue?
> > > > > > >
> > > > > > > Guozhang
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jun 5, 2014 at 10:03 AM, Libo Yu <yu...@hotmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > When all the brokers are down the producer should retry for a
> > few
> > > > > > times
> > > > > > > > and throw FailedToSendMessageException. And user code can
> > catch the
> > > > > > > > exception and retry after a backoff. However, in my tests, no
> > > > > > exception was
> > > > > > > > caught and the message was lost silently. My broker is 0.8.1.1
> > and
> > > > my
> > > > > > > > client is 0.8.0. It is fairly easy to reproduce. Any insight on
> > > > this
> > > > > > issue?
> > > > > > > >
> > > > > > > > Libo
> > > > > > > >
> > > > > > > > > Date: Thu, 5 Jun 2014 09:05:27 -0700
> > > > > > > > > Subject: Re: question about synchronous producer
> > > > > > > > > From: wangguoz@gmail.com
> > > > > > > > > To: users@kafka.apache.org
> > > > > > > > >
> > > > > > > > > When the producer exhausted all the retries it will drop the
> > > > message
> > > > > > on
> > > > > > > > the
> > > > > > > > > floor. So when the broker is down for too long there will be
> > data
> > > > > > loss.
> > > > > > > > >
> > > > > > > > > Guozhang
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Jun 5, 2014 at 6:20 AM, Libo Yu <yu_libo@hotmail.com
> > >
> > > > wrote:
> > > > > > > > >
> > > > > > > > > > I want to know why there will be message loss when brokers
> > are
> > > > > > down for
> > > > > > > > > > too long.
> > > > > > > > > > I've noticed message loss when brokers are restarted during
> > > > > > > > publishing. It
> > > > > > > > > > is a sync producer with request.required.acks set to 1.
> > > > > > > > > >
> > > > > > > > > > Libo
> > > > > > > > > >
> > > > > > > > > > > Date: Thu, 29 May 2014 20:11:48 -0700
> > > > > > > > > > > Subject: Re: question about synchronous producer
> > > > > > > > > > > From: wangguoz@gmail.com
> > > > > > > > > > > To: users@kafka.apache.org
> > > > > > > > > > >
> > > > > > > > > > > Libo,
> > > > > > > > > > >
> > > > > > > > > > > That is correct. You may want to increase the
> > > > retry.backoff.ms
> > > > > > in
> > > > > > > > this
> > > > > > > > > > > case. In practice, if the brokers are down for too long,
> > then
> > > > > > data
> > > > > > > > loss
> > > > > > > > > > is
> > > > > > > > > > > usually inevitable.
> > > > > > > > > > >
> > > > > > > > > > > Guozhang
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Thu, May 29, 2014 at 2:55 PM, Libo Yu <
> > > > yu_libo@hotmail.com>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi team,
> > > > > > > > > > > >
> > > > > > > > > > > > Assume I am using a synchronous producer and it has the
> > > > > > following
> > > > > > > > > > default
> > > > > > > > > > > > properties:
> > > > > > > > > > > >
> > > > > > > > > > > > message.send.max.retries
> > > > > > > > > > > > 3
> > > > > > > > > > > > retry.backoff.ms
> > > > > > > > > > > > 100
> > > > > > > > > > > >
> > > > > > > > > > > > I use java api Producer.send(message) to send a
> > message.
> > > > > > > > > > > > While send() is being called, if the brokers are
> > shutdown,
> > > > what
> > > > > > > > > > happens?
> > > > > > > > > > > > send() will retry 3 times with a 100ms interval and
> > fail
> > > > > > silently?
> > > > > > > > > > > > If I don't want to lose any message when the brokers
> > are
> > > > back
> > > > > > > > online,
> > > > > > > > > > what
> > > > > > > > > > > > should I do? Thanks.
> > > > > > > > > > > >
> > > > > > > > > > > > Libo
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > -- Guozhang
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > -- Guozhang
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > -- Guozhang
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> >
> >
>
>
>
> --
> -- Guozhang
Re: question about synchronous producer
Posted by Guozhang Wang <wa...@gmail.com>.
I see. So previously you run the test as ack=1?
Guozhang
On Sat, Jun 7, 2014 at 7:24 AM, Libo Yu <yu...@hotmail.com> wrote:
> Hi Guozhang,
>
> The issue is not constantly to reproduce but fairly easy to reproduce.
> It seems there is some kind of failure that has not been captured by the
> Kafka code and no exception has been thrown.
>
> I did a new test with request.required.acks set to -1. The number of lost
> message dropped significantly but there were still message loss.
>
> > Date: Fri, 6 Jun 2014 08:12:17 -0700
> > Subject: Re: question about synchronous producer
> > From: wangguoz@gmail.com
> > To: users@kafka.apache.org
> >
> > Libo,
> >
> > I have double checked the code. With sync producers all failures should
> be
> > either thrown as exceptions or logged in warning/error log entries.
> >
> > Guozhang
> >
> >
> > On Thu, Jun 5, 2014 at 6:38 PM, Libo Yu <yu...@hotmail.com> wrote:
> >
> > > Not really. The issue was reported by a client. I added a lot of
> logging
> > > to make sure no exception was thrown
> > > from send() when the message was lost. It is not hard to reproduce.
> This
> > > is a critical issue for operation. It may
> > > not be possible for brokers and producers to be restarted at the same
> time.
> > >
> > > > Date: Thu, 5 Jun 2014 16:53:29 -0700
> > > > Subject: Re: question about synchronous producer
> > > > From: wangguoz@gmail.com
> > > > To: users@kafka.apache.org
> > > >
> > > > Libo, did you see any exception/error entries on the producer log?
> > > >
> > > > Guozhang
> > > >
> > > >
> > > > On Thu, Jun 5, 2014 at 10:33 AM, Libo Yu <yu...@hotmail.com>
> wrote:
> > > >
> > > > > Yes. I used three sync producers with request.required.acks=1. I
> let
> > > them
> > > > > publish 2k short messages and in the process I restart all
> zookeeper
> > > and
> > > > > kafka processes ( 3 hosts in a cluster). Normally there will be
> message
> > > > > loss after 3 restarts. After 3 restarts, I use a consumer to
> retrieve
> > > the
> > > > > messages and do the verification.
> > > > >
> > > > > > Date: Thu, 5 Jun 2014 10:15:18 -0700
> > > > > > Subject: Re: question about synchronous producer
> > > > > > From: wangguoz@gmail.com
> > > > > > To: users@kafka.apache.org
> > > > > >
> > > > > > Libo,
> > > > > >
> > > > > > For clarification, you can use sync producer to reproduce this
> issue?
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > >
> > > > > > On Thu, Jun 5, 2014 at 10:03 AM, Libo Yu <yu...@hotmail.com>
> > > wrote:
> > > > > >
> > > > > > > When all the brokers are down the producer should retry for a
> few
> > > > > times
> > > > > > > and throw FailedToSendMessageException. And user code can
> catch the
> > > > > > > exception and retry after a backoff. However, in my tests, no
> > > > > exception was
> > > > > > > caught and the message was lost silently. My broker is 0.8.1.1
> and
> > > my
> > > > > > > client is 0.8.0. It is fairly easy to reproduce. Any insight on
> > > this
> > > > > issue?
> > > > > > >
> > > > > > > Libo
> > > > > > >
> > > > > > > > Date: Thu, 5 Jun 2014 09:05:27 -0700
> > > > > > > > Subject: Re: question about synchronous producer
> > > > > > > > From: wangguoz@gmail.com
> > > > > > > > To: users@kafka.apache.org
> > > > > > > >
> > > > > > > > When the producer exhausted all the retries it will drop the
> > > message
> > > > > on
> > > > > > > the
> > > > > > > > floor. So when the broker is down for too long there will be
> data
> > > > > loss.
> > > > > > > >
> > > > > > > > Guozhang
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Jun 5, 2014 at 6:20 AM, Libo Yu <yu_libo@hotmail.com
> >
> > > wrote:
> > > > > > > >
> > > > > > > > > I want to know why there will be message loss when brokers
> are
> > > > > down for
> > > > > > > > > too long.
> > > > > > > > > I've noticed message loss when brokers are restarted during
> > > > > > > publishing. It
> > > > > > > > > is a sync producer with request.required.acks set to 1.
> > > > > > > > >
> > > > > > > > > Libo
> > > > > > > > >
> > > > > > > > > > Date: Thu, 29 May 2014 20:11:48 -0700
> > > > > > > > > > Subject: Re: question about synchronous producer
> > > > > > > > > > From: wangguoz@gmail.com
> > > > > > > > > > To: users@kafka.apache.org
> > > > > > > > > >
> > > > > > > > > > Libo,
> > > > > > > > > >
> > > > > > > > > > That is correct. You may want to increase the
> > > retry.backoff.ms
> > > > > in
> > > > > > > this
> > > > > > > > > > case. In practice, if the brokers are down for too long,
> then
> > > > > data
> > > > > > > loss
> > > > > > > > > is
> > > > > > > > > > usually inevitable.
> > > > > > > > > >
> > > > > > > > > > Guozhang
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, May 29, 2014 at 2:55 PM, Libo Yu <
> > > yu_libo@hotmail.com>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi team,
> > > > > > > > > > >
> > > > > > > > > > > Assume I am using a synchronous producer and it has the
> > > > > following
> > > > > > > > > default
> > > > > > > > > > > properties:
> > > > > > > > > > >
> > > > > > > > > > > message.send.max.retries
> > > > > > > > > > > 3
> > > > > > > > > > > retry.backoff.ms
> > > > > > > > > > > 100
> > > > > > > > > > >
> > > > > > > > > > > I use java api Producer.send(message) to send a
> message.
> > > > > > > > > > > While send() is being called, if the brokers are
> shutdown,
> > > what
> > > > > > > > > happens?
> > > > > > > > > > > send() will retry 3 times with a 100ms interval and
> fail
> > > > > silently?
> > > > > > > > > > > If I don't want to lose any message when the brokers
> are
> > > back
> > > > > > > online,
> > > > > > > > > what
> > > > > > > > > > > should I do? Thanks.
> > > > > > > > > > >
> > > > > > > > > > > Libo
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > -- Guozhang
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > -- Guozhang
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -- Guozhang
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
>
>
--
-- Guozhang
RE: question about synchronous producer
Posted by Libo Yu <yu...@hotmail.com>.
Hi Guozhang,
The issue is not constantly to reproduce but fairly easy to reproduce.
It seems there is some kind of failure that has not been captured by the
Kafka code and no exception has been thrown.
I did a new test with request.required.acks set to -1. The number of lost
message dropped significantly but there were still message loss.
> Date: Fri, 6 Jun 2014 08:12:17 -0700
> Subject: Re: question about synchronous producer
> From: wangguoz@gmail.com
> To: users@kafka.apache.org
>
> Libo,
>
> I have double checked the code. With sync producers all failures should be
> either thrown as exceptions or logged in warning/error log entries.
>
> Guozhang
>
>
> On Thu, Jun 5, 2014 at 6:38 PM, Libo Yu <yu...@hotmail.com> wrote:
>
> > Not really. The issue was reported by a client. I added a lot of logging
> > to make sure no exception was thrown
> > from send() when the message was lost. It is not hard to reproduce. This
> > is a critical issue for operation. It may
> > not be possible for brokers and producers to be restarted at the same time.
> >
> > > Date: Thu, 5 Jun 2014 16:53:29 -0700
> > > Subject: Re: question about synchronous producer
> > > From: wangguoz@gmail.com
> > > To: users@kafka.apache.org
> > >
> > > Libo, did you see any exception/error entries on the producer log?
> > >
> > > Guozhang
> > >
> > >
> > > On Thu, Jun 5, 2014 at 10:33 AM, Libo Yu <yu...@hotmail.com> wrote:
> > >
> > > > Yes. I used three sync producers with request.required.acks=1. I let
> > them
> > > > publish 2k short messages and in the process I restart all zookeeper
> > and
> > > > kafka processes ( 3 hosts in a cluster). Normally there will be message
> > > > loss after 3 restarts. After 3 restarts, I use a consumer to retrieve
> > the
> > > > messages and do the verification.
> > > >
> > > > > Date: Thu, 5 Jun 2014 10:15:18 -0700
> > > > > Subject: Re: question about synchronous producer
> > > > > From: wangguoz@gmail.com
> > > > > To: users@kafka.apache.org
> > > > >
> > > > > Libo,
> > > > >
> > > > > For clarification, you can use sync producer to reproduce this issue?
> > > > >
> > > > > Guozhang
> > > > >
> > > > >
> > > > > On Thu, Jun 5, 2014 at 10:03 AM, Libo Yu <yu...@hotmail.com>
> > wrote:
> > > > >
> > > > > > When all the brokers are down the producer should retry for a few
> > > > times
> > > > > > and throw FailedToSendMessageException. And user code can catch the
> > > > > > exception and retry after a backoff. However, in my tests, no
> > > > exception was
> > > > > > caught and the message was lost silently. My broker is 0.8.1.1 and
> > my
> > > > > > client is 0.8.0. It is fairly easy to reproduce. Any insight on
> > this
> > > > issue?
> > > > > >
> > > > > > Libo
> > > > > >
> > > > > > > Date: Thu, 5 Jun 2014 09:05:27 -0700
> > > > > > > Subject: Re: question about synchronous producer
> > > > > > > From: wangguoz@gmail.com
> > > > > > > To: users@kafka.apache.org
> > > > > > >
> > > > > > > When the producer exhausted all the retries it will drop the
> > message
> > > > on
> > > > > > the
> > > > > > > floor. So when the broker is down for too long there will be data
> > > > loss.
> > > > > > >
> > > > > > > Guozhang
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jun 5, 2014 at 6:20 AM, Libo Yu <yu...@hotmail.com>
> > wrote:
> > > > > > >
> > > > > > > > I want to know why there will be message loss when brokers are
> > > > down for
> > > > > > > > too long.
> > > > > > > > I've noticed message loss when brokers are restarted during
> > > > > > publishing. It
> > > > > > > > is a sync producer with request.required.acks set to 1.
> > > > > > > >
> > > > > > > > Libo
> > > > > > > >
> > > > > > > > > Date: Thu, 29 May 2014 20:11:48 -0700
> > > > > > > > > Subject: Re: question about synchronous producer
> > > > > > > > > From: wangguoz@gmail.com
> > > > > > > > > To: users@kafka.apache.org
> > > > > > > > >
> > > > > > > > > Libo,
> > > > > > > > >
> > > > > > > > > That is correct. You may want to increase the
> > retry.backoff.ms
> > > > in
> > > > > > this
> > > > > > > > > case. In practice, if the brokers are down for too long, then
> > > > data
> > > > > > loss
> > > > > > > > is
> > > > > > > > > usually inevitable.
> > > > > > > > >
> > > > > > > > > Guozhang
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, May 29, 2014 at 2:55 PM, Libo Yu <
> > yu_libo@hotmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi team,
> > > > > > > > > >
> > > > > > > > > > Assume I am using a synchronous producer and it has the
> > > > following
> > > > > > > > default
> > > > > > > > > > properties:
> > > > > > > > > >
> > > > > > > > > > message.send.max.retries
> > > > > > > > > > 3
> > > > > > > > > > retry.backoff.ms
> > > > > > > > > > 100
> > > > > > > > > >
> > > > > > > > > > I use java api Producer.send(message) to send a message.
> > > > > > > > > > While send() is being called, if the brokers are shutdown,
> > what
> > > > > > > > happens?
> > > > > > > > > > send() will retry 3 times with a 100ms interval and fail
> > > > silently?
> > > > > > > > > > If I don't want to lose any message when the brokers are
> > back
> > > > > > online,
> > > > > > > > what
> > > > > > > > > > should I do? Thanks.
> > > > > > > > > >
> > > > > > > > > > Libo
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > -- Guozhang
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > -- Guozhang
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> >
> >
>
>
>
> --
> -- Guozhang
Re: question about synchronous producer
Posted by Guozhang Wang <wa...@gmail.com>.
Libo,
I have double checked the code. With sync producers all failures should be
either thrown as exceptions or logged in warning/error log entries.
Guozhang
On Thu, Jun 5, 2014 at 6:38 PM, Libo Yu <yu...@hotmail.com> wrote:
> Not really. The issue was reported by a client. I added a lot of logging
> to make sure no exception was thrown
> from send() when the message was lost. It is not hard to reproduce. This
> is a critical issue for operation. It may
> not be possible for brokers and producers to be restarted at the same time.
>
> > Date: Thu, 5 Jun 2014 16:53:29 -0700
> > Subject: Re: question about synchronous producer
> > From: wangguoz@gmail.com
> > To: users@kafka.apache.org
> >
> > Libo, did you see any exception/error entries on the producer log?
> >
> > Guozhang
> >
> >
> > On Thu, Jun 5, 2014 at 10:33 AM, Libo Yu <yu...@hotmail.com> wrote:
> >
> > > Yes. I used three sync producers with request.required.acks=1. I let
> them
> > > publish 2k short messages and in the process I restart all zookeeper
> and
> > > kafka processes ( 3 hosts in a cluster). Normally there will be message
> > > loss after 3 restarts. After 3 restarts, I use a consumer to retrieve
> the
> > > messages and do the verification.
> > >
> > > > Date: Thu, 5 Jun 2014 10:15:18 -0700
> > > > Subject: Re: question about synchronous producer
> > > > From: wangguoz@gmail.com
> > > > To: users@kafka.apache.org
> > > >
> > > > Libo,
> > > >
> > > > For clarification, you can use sync producer to reproduce this issue?
> > > >
> > > > Guozhang
> > > >
> > > >
> > > > On Thu, Jun 5, 2014 at 10:03 AM, Libo Yu <yu...@hotmail.com>
> wrote:
> > > >
> > > > > When all the brokers are down the producer should retry for a few
> > > times
> > > > > and throw FailedToSendMessageException. And user code can catch the
> > > > > exception and retry after a backoff. However, in my tests, no
> > > exception was
> > > > > caught and the message was lost silently. My broker is 0.8.1.1 and
> my
> > > > > client is 0.8.0. It is fairly easy to reproduce. Any insight on
> this
> > > issue?
> > > > >
> > > > > Libo
> > > > >
> > > > > > Date: Thu, 5 Jun 2014 09:05:27 -0700
> > > > > > Subject: Re: question about synchronous producer
> > > > > > From: wangguoz@gmail.com
> > > > > > To: users@kafka.apache.org
> > > > > >
> > > > > > When the producer exhausted all the retries it will drop the
> message
> > > on
> > > > > the
> > > > > > floor. So when the broker is down for too long there will be data
> > > loss.
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > >
> > > > > > On Thu, Jun 5, 2014 at 6:20 AM, Libo Yu <yu...@hotmail.com>
> wrote:
> > > > > >
> > > > > > > I want to know why there will be message loss when brokers are
> > > down for
> > > > > > > too long.
> > > > > > > I've noticed message loss when brokers are restarted during
> > > > > publishing. It
> > > > > > > is a sync producer with request.required.acks set to 1.
> > > > > > >
> > > > > > > Libo
> > > > > > >
> > > > > > > > Date: Thu, 29 May 2014 20:11:48 -0700
> > > > > > > > Subject: Re: question about synchronous producer
> > > > > > > > From: wangguoz@gmail.com
> > > > > > > > To: users@kafka.apache.org
> > > > > > > >
> > > > > > > > Libo,
> > > > > > > >
> > > > > > > > That is correct. You may want to increase the
> retry.backoff.ms
> > > in
> > > > > this
> > > > > > > > case. In practice, if the brokers are down for too long, then
> > > data
> > > > > loss
> > > > > > > is
> > > > > > > > usually inevitable.
> > > > > > > >
> > > > > > > > Guozhang
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, May 29, 2014 at 2:55 PM, Libo Yu <
> yu_libo@hotmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi team,
> > > > > > > > >
> > > > > > > > > Assume I am using a synchronous producer and it has the
> > > following
> > > > > > > default
> > > > > > > > > properties:
> > > > > > > > >
> > > > > > > > > message.send.max.retries
> > > > > > > > > 3
> > > > > > > > > retry.backoff.ms
> > > > > > > > > 100
> > > > > > > > >
> > > > > > > > > I use java api Producer.send(message) to send a message.
> > > > > > > > > While send() is being called, if the brokers are shutdown,
> what
> > > > > > > happens?
> > > > > > > > > send() will retry 3 times with a 100ms interval and fail
> > > silently?
> > > > > > > > > If I don't want to lose any message when the brokers are
> back
> > > > > online,
> > > > > > > what
> > > > > > > > > should I do? Thanks.
> > > > > > > > >
> > > > > > > > > Libo
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > -- Guozhang
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -- Guozhang
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
>
>
--
-- Guozhang
RE: question about synchronous producer
Posted by Libo Yu <yu...@hotmail.com>.
Not really. The issue was reported by a client. I added a lot of logging to make sure no exception was thrown
from send() when the message was lost. It is not hard to reproduce. This is a critical issue for operation. It may
not be possible for brokers and producers to be restarted at the same time.
> Date: Thu, 5 Jun 2014 16:53:29 -0700
> Subject: Re: question about synchronous producer
> From: wangguoz@gmail.com
> To: users@kafka.apache.org
>
> Libo, did you see any exception/error entries on the producer log?
>
> Guozhang
>
>
> On Thu, Jun 5, 2014 at 10:33 AM, Libo Yu <yu...@hotmail.com> wrote:
>
> > Yes. I used three sync producers with request.required.acks=1. I let them
> > publish 2k short messages and in the process I restart all zookeeper and
> > kafka processes ( 3 hosts in a cluster). Normally there will be message
> > loss after 3 restarts. After 3 restarts, I use a consumer to retrieve the
> > messages and do the verification.
> >
> > > Date: Thu, 5 Jun 2014 10:15:18 -0700
> > > Subject: Re: question about synchronous producer
> > > From: wangguoz@gmail.com
> > > To: users@kafka.apache.org
> > >
> > > Libo,
> > >
> > > For clarification, you can use sync producer to reproduce this issue?
> > >
> > > Guozhang
> > >
> > >
> > > On Thu, Jun 5, 2014 at 10:03 AM, Libo Yu <yu...@hotmail.com> wrote:
> > >
> > > > When all the brokers are down the producer should retry for a few
> > times
> > > > and throw FailedToSendMessageException. And user code can catch the
> > > > exception and retry after a backoff. However, in my tests, no
> > exception was
> > > > caught and the message was lost silently. My broker is 0.8.1.1 and my
> > > > client is 0.8.0. It is fairly easy to reproduce. Any insight on this
> > issue?
> > > >
> > > > Libo
> > > >
> > > > > Date: Thu, 5 Jun 2014 09:05:27 -0700
> > > > > Subject: Re: question about synchronous producer
> > > > > From: wangguoz@gmail.com
> > > > > To: users@kafka.apache.org
> > > > >
> > > > > When the producer exhausted all the retries it will drop the message
> > on
> > > > the
> > > > > floor. So when the broker is down for too long there will be data
> > loss.
> > > > >
> > > > > Guozhang
> > > > >
> > > > >
> > > > > On Thu, Jun 5, 2014 at 6:20 AM, Libo Yu <yu...@hotmail.com> wrote:
> > > > >
> > > > > > I want to know why there will be message loss when brokers are
> > down for
> > > > > > too long.
> > > > > > I've noticed message loss when brokers are restarted during
> > > > publishing. It
> > > > > > is a sync producer with request.required.acks set to 1.
> > > > > >
> > > > > > Libo
> > > > > >
> > > > > > > Date: Thu, 29 May 2014 20:11:48 -0700
> > > > > > > Subject: Re: question about synchronous producer
> > > > > > > From: wangguoz@gmail.com
> > > > > > > To: users@kafka.apache.org
> > > > > > >
> > > > > > > Libo,
> > > > > > >
> > > > > > > That is correct. You may want to increase the retry.backoff.ms
> > in
> > > > this
> > > > > > > case. In practice, if the brokers are down for too long, then
> > data
> > > > loss
> > > > > > is
> > > > > > > usually inevitable.
> > > > > > >
> > > > > > > Guozhang
> > > > > > >
> > > > > > >
> > > > > > > On Thu, May 29, 2014 at 2:55 PM, Libo Yu <yu...@hotmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > Hi team,
> > > > > > > >
> > > > > > > > Assume I am using a synchronous producer and it has the
> > following
> > > > > > default
> > > > > > > > properties:
> > > > > > > >
> > > > > > > > message.send.max.retries
> > > > > > > > 3
> > > > > > > > retry.backoff.ms
> > > > > > > > 100
> > > > > > > >
> > > > > > > > I use java api Producer.send(message) to send a message.
> > > > > > > > While send() is being called, if the brokers are shutdown, what
> > > > > > happens?
> > > > > > > > send() will retry 3 times with a 100ms interval and fail
> > silently?
> > > > > > > > If I don't want to lose any message when the brokers are back
> > > > online,
> > > > > > what
> > > > > > > > should I do? Thanks.
> > > > > > > >
> > > > > > > > Libo
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > -- Guozhang
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> >
> >
>
>
>
> --
> -- Guozhang
Re: question about synchronous producer
Posted by Guozhang Wang <wa...@gmail.com>.
Libo, did you see any exception/error entries on the producer log?
Guozhang
On Thu, Jun 5, 2014 at 10:33 AM, Libo Yu <yu...@hotmail.com> wrote:
> Yes. I used three sync producers with request.required.acks=1. I let them
> publish 2k short messages and in the process I restart all zookeeper and
> kafka processes ( 3 hosts in a cluster). Normally there will be message
> loss after 3 restarts. After 3 restarts, I use a consumer to retrieve the
> messages and do the verification.
>
> > Date: Thu, 5 Jun 2014 10:15:18 -0700
> > Subject: Re: question about synchronous producer
> > From: wangguoz@gmail.com
> > To: users@kafka.apache.org
> >
> > Libo,
> >
> > For clarification, you can use sync producer to reproduce this issue?
> >
> > Guozhang
> >
> >
> > On Thu, Jun 5, 2014 at 10:03 AM, Libo Yu <yu...@hotmail.com> wrote:
> >
> > > When all the brokers are down the producer should retry for a few
> times
> > > and throw FailedToSendMessageException. And user code can catch the
> > > exception and retry after a backoff. However, in my tests, no
> exception was
> > > caught and the message was lost silently. My broker is 0.8.1.1 and my
> > > client is 0.8.0. It is fairly easy to reproduce. Any insight on this
> issue?
> > >
> > > Libo
> > >
> > > > Date: Thu, 5 Jun 2014 09:05:27 -0700
> > > > Subject: Re: question about synchronous producer
> > > > From: wangguoz@gmail.com
> > > > To: users@kafka.apache.org
> > > >
> > > > When the producer exhausted all the retries it will drop the message
> on
> > > the
> > > > floor. So when the broker is down for too long there will be data
> loss.
> > > >
> > > > Guozhang
> > > >
> > > >
> > > > On Thu, Jun 5, 2014 at 6:20 AM, Libo Yu <yu...@hotmail.com> wrote:
> > > >
> > > > > I want to know why there will be message loss when brokers are
> down for
> > > > > too long.
> > > > > I've noticed message loss when brokers are restarted during
> > > publishing. It
> > > > > is a sync producer with request.required.acks set to 1.
> > > > >
> > > > > Libo
> > > > >
> > > > > > Date: Thu, 29 May 2014 20:11:48 -0700
> > > > > > Subject: Re: question about synchronous producer
> > > > > > From: wangguoz@gmail.com
> > > > > > To: users@kafka.apache.org
> > > > > >
> > > > > > Libo,
> > > > > >
> > > > > > That is correct. You may want to increase the retry.backoff.ms
> in
> > > this
> > > > > > case. In practice, if the brokers are down for too long, then
> data
> > > loss
> > > > > is
> > > > > > usually inevitable.
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > >
> > > > > > On Thu, May 29, 2014 at 2:55 PM, Libo Yu <yu...@hotmail.com>
> > > wrote:
> > > > > >
> > > > > > > Hi team,
> > > > > > >
> > > > > > > Assume I am using a synchronous producer and it has the
> following
> > > > > default
> > > > > > > properties:
> > > > > > >
> > > > > > > message.send.max.retries
> > > > > > > 3
> > > > > > > retry.backoff.ms
> > > > > > > 100
> > > > > > >
> > > > > > > I use java api Producer.send(message) to send a message.
> > > > > > > While send() is being called, if the brokers are shutdown, what
> > > > > happens?
> > > > > > > send() will retry 3 times with a 100ms interval and fail
> silently?
> > > > > > > If I don't want to lose any message when the brokers are back
> > > online,
> > > > > what
> > > > > > > should I do? Thanks.
> > > > > > >
> > > > > > > Libo
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -- Guozhang
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
>
>
--
-- Guozhang
RE: question about synchronous producer
Posted by Libo Yu <yu...@hotmail.com>.
Yes. I used three sync producers with request.required.acks=1. I let them publish 2k short messages and in the process I restart all zookeeper and kafka processes ( 3 hosts in a cluster). Normally there will be message loss after 3 restarts. After 3 restarts, I use a consumer to retrieve the messages and do the verification.
> Date: Thu, 5 Jun 2014 10:15:18 -0700
> Subject: Re: question about synchronous producer
> From: wangguoz@gmail.com
> To: users@kafka.apache.org
>
> Libo,
>
> For clarification, you can use sync producer to reproduce this issue?
>
> Guozhang
>
>
> On Thu, Jun 5, 2014 at 10:03 AM, Libo Yu <yu...@hotmail.com> wrote:
>
> > When all the brokers are down the producer should retry for a few times
> > and throw FailedToSendMessageException. And user code can catch the
> > exception and retry after a backoff. However, in my tests, no exception was
> > caught and the message was lost silently. My broker is 0.8.1.1 and my
> > client is 0.8.0. It is fairly easy to reproduce. Any insight on this issue?
> >
> > Libo
> >
> > > Date: Thu, 5 Jun 2014 09:05:27 -0700
> > > Subject: Re: question about synchronous producer
> > > From: wangguoz@gmail.com
> > > To: users@kafka.apache.org
> > >
> > > When the producer exhausted all the retries it will drop the message on
> > the
> > > floor. So when the broker is down for too long there will be data loss.
> > >
> > > Guozhang
> > >
> > >
> > > On Thu, Jun 5, 2014 at 6:20 AM, Libo Yu <yu...@hotmail.com> wrote:
> > >
> > > > I want to know why there will be message loss when brokers are down for
> > > > too long.
> > > > I've noticed message loss when brokers are restarted during
> > publishing. It
> > > > is a sync producer with request.required.acks set to 1.
> > > >
> > > > Libo
> > > >
> > > > > Date: Thu, 29 May 2014 20:11:48 -0700
> > > > > Subject: Re: question about synchronous producer
> > > > > From: wangguoz@gmail.com
> > > > > To: users@kafka.apache.org
> > > > >
> > > > > Libo,
> > > > >
> > > > > That is correct. You may want to increase the retry.backoff.ms in
> > this
> > > > > case. In practice, if the brokers are down for too long, then data
> > loss
> > > > is
> > > > > usually inevitable.
> > > > >
> > > > > Guozhang
> > > > >
> > > > >
> > > > > On Thu, May 29, 2014 at 2:55 PM, Libo Yu <yu...@hotmail.com>
> > wrote:
> > > > >
> > > > > > Hi team,
> > > > > >
> > > > > > Assume I am using a synchronous producer and it has the following
> > > > default
> > > > > > properties:
> > > > > >
> > > > > > message.send.max.retries
> > > > > > 3
> > > > > > retry.backoff.ms
> > > > > > 100
> > > > > >
> > > > > > I use java api Producer.send(message) to send a message.
> > > > > > While send() is being called, if the brokers are shutdown, what
> > > > happens?
> > > > > > send() will retry 3 times with a 100ms interval and fail silently?
> > > > > > If I don't want to lose any message when the brokers are back
> > online,
> > > > what
> > > > > > should I do? Thanks.
> > > > > >
> > > > > > Libo
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> >
> >
>
>
>
> --
> -- Guozhang
Re: question about synchronous producer
Posted by Guozhang Wang <wa...@gmail.com>.
Libo,
For clarification, you can use sync producer to reproduce this issue?
Guozhang
On Thu, Jun 5, 2014 at 10:03 AM, Libo Yu <yu...@hotmail.com> wrote:
> When all the brokers are down the producer should retry for a few times
> and throw FailedToSendMessageException. And user code can catch the
> exception and retry after a backoff. However, in my tests, no exception was
> caught and the message was lost silently. My broker is 0.8.1.1 and my
> client is 0.8.0. It is fairly easy to reproduce. Any insight on this issue?
>
> Libo
>
> > Date: Thu, 5 Jun 2014 09:05:27 -0700
> > Subject: Re: question about synchronous producer
> > From: wangguoz@gmail.com
> > To: users@kafka.apache.org
> >
> > When the producer exhausted all the retries it will drop the message on
> the
> > floor. So when the broker is down for too long there will be data loss.
> >
> > Guozhang
> >
> >
> > On Thu, Jun 5, 2014 at 6:20 AM, Libo Yu <yu...@hotmail.com> wrote:
> >
> > > I want to know why there will be message loss when brokers are down for
> > > too long.
> > > I've noticed message loss when brokers are restarted during
> publishing. It
> > > is a sync producer with request.required.acks set to 1.
> > >
> > > Libo
> > >
> > > > Date: Thu, 29 May 2014 20:11:48 -0700
> > > > Subject: Re: question about synchronous producer
> > > > From: wangguoz@gmail.com
> > > > To: users@kafka.apache.org
> > > >
> > > > Libo,
> > > >
> > > > That is correct. You may want to increase the retry.backoff.ms in
> this
> > > > case. In practice, if the brokers are down for too long, then data
> loss
> > > is
> > > > usually inevitable.
> > > >
> > > > Guozhang
> > > >
> > > >
> > > > On Thu, May 29, 2014 at 2:55 PM, Libo Yu <yu...@hotmail.com>
> wrote:
> > > >
> > > > > Hi team,
> > > > >
> > > > > Assume I am using a synchronous producer and it has the following
> > > default
> > > > > properties:
> > > > >
> > > > > message.send.max.retries
> > > > > 3
> > > > > retry.backoff.ms
> > > > > 100
> > > > >
> > > > > I use java api Producer.send(message) to send a message.
> > > > > While send() is being called, if the brokers are shutdown, what
> > > happens?
> > > > > send() will retry 3 times with a 100ms interval and fail silently?
> > > > > If I don't want to lose any message when the brokers are back
> online,
> > > what
> > > > > should I do? Thanks.
> > > > >
> > > > > Libo
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
>
>
--
-- Guozhang
RE: question about synchronous producer
Posted by Libo Yu <yu...@hotmail.com>.
When all the brokers are down the producer should retry for a few times and throw FailedToSendMessageException. And user code can catch the exception and retry after a backoff. However, in my tests, no exception was caught and the message was lost silently. My broker is 0.8.1.1 and my client is 0.8.0. It is fairly easy to reproduce. Any insight on this issue?
Libo
> Date: Thu, 5 Jun 2014 09:05:27 -0700
> Subject: Re: question about synchronous producer
> From: wangguoz@gmail.com
> To: users@kafka.apache.org
>
> When the producer exhausted all the retries it will drop the message on the
> floor. So when the broker is down for too long there will be data loss.
>
> Guozhang
>
>
> On Thu, Jun 5, 2014 at 6:20 AM, Libo Yu <yu...@hotmail.com> wrote:
>
> > I want to know why there will be message loss when brokers are down for
> > too long.
> > I've noticed message loss when brokers are restarted during publishing. It
> > is a sync producer with request.required.acks set to 1.
> >
> > Libo
> >
> > > Date: Thu, 29 May 2014 20:11:48 -0700
> > > Subject: Re: question about synchronous producer
> > > From: wangguoz@gmail.com
> > > To: users@kafka.apache.org
> > >
> > > Libo,
> > >
> > > That is correct. You may want to increase the retry.backoff.ms in this
> > > case. In practice, if the brokers are down for too long, then data loss
> > is
> > > usually inevitable.
> > >
> > > Guozhang
> > >
> > >
> > > On Thu, May 29, 2014 at 2:55 PM, Libo Yu <yu...@hotmail.com> wrote:
> > >
> > > > Hi team,
> > > >
> > > > Assume I am using a synchronous producer and it has the following
> > default
> > > > properties:
> > > >
> > > > message.send.max.retries
> > > > 3
> > > > retry.backoff.ms
> > > > 100
> > > >
> > > > I use java api Producer.send(message) to send a message.
> > > > While send() is being called, if the brokers are shutdown, what
> > happens?
> > > > send() will retry 3 times with a 100ms interval and fail silently?
> > > > If I don't want to lose any message when the brokers are back online,
> > what
> > > > should I do? Thanks.
> > > >
> > > > Libo
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> >
> >
>
>
>
> --
> -- Guozhang
Re: question about synchronous producer
Posted by Guozhang Wang <wa...@gmail.com>.
When the producer exhausted all the retries it will drop the message on the
floor. So when the broker is down for too long there will be data loss.
Guozhang
On Thu, Jun 5, 2014 at 6:20 AM, Libo Yu <yu...@hotmail.com> wrote:
> I want to know why there will be message loss when brokers are down for
> too long.
> I've noticed message loss when brokers are restarted during publishing. It
> is a sync producer with request.required.acks set to 1.
>
> Libo
>
> > Date: Thu, 29 May 2014 20:11:48 -0700
> > Subject: Re: question about synchronous producer
> > From: wangguoz@gmail.com
> > To: users@kafka.apache.org
> >
> > Libo,
> >
> > That is correct. You may want to increase the retry.backoff.ms in this
> > case. In practice, if the brokers are down for too long, then data loss
> is
> > usually inevitable.
> >
> > Guozhang
> >
> >
> > On Thu, May 29, 2014 at 2:55 PM, Libo Yu <yu...@hotmail.com> wrote:
> >
> > > Hi team,
> > >
> > > Assume I am using a synchronous producer and it has the following
> default
> > > properties:
> > >
> > > message.send.max.retries
> > > 3
> > > retry.backoff.ms
> > > 100
> > >
> > > I use java api Producer.send(message) to send a message.
> > > While send() is being called, if the brokers are shutdown, what
> happens?
> > > send() will retry 3 times with a 100ms interval and fail silently?
> > > If I don't want to lose any message when the brokers are back online,
> what
> > > should I do? Thanks.
> > >
> > > Libo
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> > --
> > -- Guozhang
>
>
--
-- Guozhang