You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Tarun Kumar <ag...@gmail.com> on 2012/06/25 17:12:42 UTC
Sync producer question
Hi,
I have a couple of questions related to Sync Producer.
1. From what i know (please correct me if i am wrong), Sync producer send
is a blocking call. Until acknowledgement is received from broker, next
message is not sent by producer.
2. what if ack is lost. Is there any kind of timeout till when producer
waits for ack. If yes, does it try to resend same message?
If yes, duplicate message might reach to broker (due to previous message's
ack timeout). How does kafka avoids that?
3. What are different possibilities of duplicate messages being stored at
broker?
Thanks,
Tarun
Re: Sync producer question
Posted by Jun Rao <ju...@gmail.com>.
Our external API is really Producer, which allows you to send messages
either synchronously (ie, immediately) or asynchronously (with batching).
Under the cover, either sync or async producer is used.
Thanks,
Jun
On Mon, Jun 25, 2012 at 8:24 AM, Tarun Kumar <ag...@gmail.com>wrote:
> Thanks for quick reply Jun.
>
> What's the difference between sync and async producer in 0.7 then?
>
>
> On Mon, Jun 25, 2012 at 8:52 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > Tarun,
> >
> > In 0.7, there is no producer side ack.
> >
> > In 0.8, sync producer will wait for an ack and is blocking. If the ack is
> > lost because a broker is down, sync producer will timeout and it's up to
> > the caller to decide what to do. If the caller chooses to resend, it's
> > possible for duplicated messages to show up in the broker.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Jun 25, 2012 at 8:12 AM, Tarun Kumar <agrawal.tarun23@gmail.com
> > >wrote:
> >
> > > Hi,
> > >
> > > I have a couple of questions related to Sync Producer.
> > >
> > > 1. From what i know (please correct me if i am wrong), Sync producer
> send
> > > is a blocking call. Until acknowledgement is received from broker, next
> > > message is not sent by producer.
> > >
> > > 2. what if ack is lost. Is there any kind of timeout till when producer
> > > waits for ack. If yes, does it try to resend same message?
> > > If yes, duplicate message might reach to broker (due to previous
> > message's
> > > ack timeout). How does kafka avoids that?
> > >
> > > 3. What are different possibilities of duplicate messages being stored
> at
> > > broker?
> > >
> > > Thanks,
> > > Tarun
> > >
> >
>
Re: Sync producer question
Posted by Tarun Kumar <ag...@gmail.com>.
Thanks for quick reply Jun.
What's the difference between sync and async producer in 0.7 then?
On Mon, Jun 25, 2012 at 8:52 PM, Jun Rao <ju...@gmail.com> wrote:
> Tarun,
>
> In 0.7, there is no producer side ack.
>
> In 0.8, sync producer will wait for an ack and is blocking. If the ack is
> lost because a broker is down, sync producer will timeout and it's up to
> the caller to decide what to do. If the caller chooses to resend, it's
> possible for duplicated messages to show up in the broker.
>
> Thanks,
>
> Jun
>
> On Mon, Jun 25, 2012 at 8:12 AM, Tarun Kumar <agrawal.tarun23@gmail.com
> >wrote:
>
> > Hi,
> >
> > I have a couple of questions related to Sync Producer.
> >
> > 1. From what i know (please correct me if i am wrong), Sync producer send
> > is a blocking call. Until acknowledgement is received from broker, next
> > message is not sent by producer.
> >
> > 2. what if ack is lost. Is there any kind of timeout till when producer
> > waits for ack. If yes, does it try to resend same message?
> > If yes, duplicate message might reach to broker (due to previous
> message's
> > ack timeout). How does kafka avoids that?
> >
> > 3. What are different possibilities of duplicate messages being stored at
> > broker?
> >
> > Thanks,
> > Tarun
> >
>
Re: Sync producer question
Posted by Jun Rao <ju...@gmail.com>.
In 0.7, without producer ack, there is no easy way to ensure produced data
reaches the server. For example, send.writeCompletely(channel) returns when
the data is in socket buffer. If a broker goes down before producer's
socket buffer is flushed, the data is lost and producer won't get any
exceptions.
Thanks,
Jun
On Mon, Jun 25, 2012 at 11:49 PM, R S <my...@gmail.com> wrote:
> Hi Jun ,
>
> In the case mentioned below , in 0.7 if send (in SyncProducer) call is
> complete without any exception does that mean message
> has successfully reached the broker .
>
> from the code i see this
> private def send(send: BoundedByteBufferSend) {
> ..............
>
> try {
> send.writeCompletely(channel)
> } catch {
> case e : java.io.IOException =>
> // no way to tell if write succeeded. Disconnect and re-throw
> exception to let client handle retry
> disconnect()
> throw e
> case e2 =>
> throw e2
> }
>
> Is there a way to know if the last offset has changed or not from Producer
> side to make sure we do not send the new message again?
>
> -rks
>
> On Mon, Jun 25, 2012 at 8:52 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > Tarun,
> >
> > In 0.7, there is no producer side ack.
> >
> > In 0.8, sync producer will wait for an ack and is blocking. If the ack is
> > lost because a broker is down, sync producer will timeout and it's up to
> > the caller to decide what to do. If the caller chooses to resend, it's
> > possible for duplicated messages to show up in the broker.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Jun 25, 2012 at 8:12 AM, Tarun Kumar <agrawal.tarun23@gmail.com
> > >wrote:
> >
> > > Hi,
> > >
> > > I have a couple of questions related to Sync Producer.
> > >
> > > 1. From what i know (please correct me if i am wrong), Sync producer
> send
> > > is a blocking call. Until acknowledgement is received from broker, next
> > > message is not sent by producer.
> > >
> > > 2. what if ack is lost. Is there any kind of timeout till when producer
> > > waits for ack. If yes, does it try to resend same message?
> > > If yes, duplicate message might reach to broker (due to previous
> > message's
> > > ack timeout). How does kafka avoids that?
> > >
> > > 3. What are different possibilities of duplicate messages being stored
> at
> > > broker?
> > >
> > > Thanks,
> > > Tarun
> > >
> >
>
Re: Sync producer question
Posted by R S <my...@gmail.com>.
Hi Jun ,
In the case mentioned below , in 0.7 if send (in SyncProducer) call is
complete without any exception does that mean message
has successfully reached the broker .
from the code i see this
private def send(send: BoundedByteBufferSend) {
..............
try {
send.writeCompletely(channel)
} catch {
case e : java.io.IOException =>
// no way to tell if write succeeded. Disconnect and re-throw
exception to let client handle retry
disconnect()
throw e
case e2 =>
throw e2
}
Is there a way to know if the last offset has changed or not from Producer
side to make sure we do not send the new message again?
-rks
On Mon, Jun 25, 2012 at 8:52 PM, Jun Rao <ju...@gmail.com> wrote:
> Tarun,
>
> In 0.7, there is no producer side ack.
>
> In 0.8, sync producer will wait for an ack and is blocking. If the ack is
> lost because a broker is down, sync producer will timeout and it's up to
> the caller to decide what to do. If the caller chooses to resend, it's
> possible for duplicated messages to show up in the broker.
>
> Thanks,
>
> Jun
>
> On Mon, Jun 25, 2012 at 8:12 AM, Tarun Kumar <agrawal.tarun23@gmail.com
> >wrote:
>
> > Hi,
> >
> > I have a couple of questions related to Sync Producer.
> >
> > 1. From what i know (please correct me if i am wrong), Sync producer send
> > is a blocking call. Until acknowledgement is received from broker, next
> > message is not sent by producer.
> >
> > 2. what if ack is lost. Is there any kind of timeout till when producer
> > waits for ack. If yes, does it try to resend same message?
> > If yes, duplicate message might reach to broker (due to previous
> message's
> > ack timeout). How does kafka avoids that?
> >
> > 3. What are different possibilities of duplicate messages being stored at
> > broker?
> >
> > Thanks,
> > Tarun
> >
>
Re: Sync producer question
Posted by Jun Rao <ju...@gmail.com>.
Tarun,
In 0.7, there is no producer side ack.
In 0.8, sync producer will wait for an ack and is blocking. If the ack is
lost because a broker is down, sync producer will timeout and it's up to
the caller to decide what to do. If the caller chooses to resend, it's
possible for duplicated messages to show up in the broker.
Thanks,
Jun
On Mon, Jun 25, 2012 at 8:12 AM, Tarun Kumar <ag...@gmail.com>wrote:
> Hi,
>
> I have a couple of questions related to Sync Producer.
>
> 1. From what i know (please correct me if i am wrong), Sync producer send
> is a blocking call. Until acknowledgement is received from broker, next
> message is not sent by producer.
>
> 2. what if ack is lost. Is there any kind of timeout till when producer
> waits for ack. If yes, does it try to resend same message?
> If yes, duplicate message might reach to broker (due to previous message's
> ack timeout). How does kafka avoids that?
>
> 3. What are different possibilities of duplicate messages being stored at
> broker?
>
> Thanks,
> Tarun
>