You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Franco Giacosa <fg...@gmail.com> on 2016/01/30 13:18:14 UTC
at-least-once delivery
Hi,
The at-least-once delivery is generated in part by the network fails and
the retries (that may generate duplicates) right?
In the event of a duplicated (there was an error but the first message
landed ok on the partition P1) the producer will recalculate the partition
on the retry? is this done automatically?
If in the retry the partition doesn't change and there is only 1 Producer,
will the duplicated be written next to the original? I mean if I poll()
they will come one after the other?
Re: at-least-once delivery
Posted by Gwen Shapira <gw...@confluent.io>.
MAX_INT is a good value if you want to just block until the buffer has some
space (and never get an exception).
On Tue, Feb 2, 2016 at 8:08 AM, Franco Giacosa <fg...@gmail.com> wrote:
> Thanks for the information James, the slides are really good.
>
> One question, in the new producer the property block.on.buffer.full (in the
> slides put this value in TRUE is a good practice, I image that this will
> avoid a buffer overflow) is deprecated, and instead the use of
> max.block.ms,
> which block for X amount of ms, can someone tell me what a good value will
> be for this property in order to mimic the behaviour of
> block.on.buffer.full?
>
> Thanks
>
> 2016-01-31 6:09 GMT+01:00 James Cheng <jc...@tivo.com>:
>
> >
> > > On Jan 30, 2016, at 4:21 AM, Franco Giacosa <fg...@gmail.com>
> wrote:
> > >
> > > Sorry, this solved my questions: "Setting a value greater than zero
> will
> > > cause the client to resend any record whose send fails with a
> potentially
> > > transient error. Note that this retry is no different than if the
> client
> > > resent the record upon receiving the error. Allowing retries will
> > > potentially change the ordering of records because if two records are
> > sent
> > > to a single partition, and the first fails and is retried but the
> second
> > > succeeds, then the second record may appear first."
> > >
> >
> > Franco,
> >
> > Also, you can avoid the message reordering issue in that description by
> > setting max.in.flight.requests.per.connector to 1.
> >
> > This slide deck has good guidelines on the types of things you are
> talking
> > about:
> >
> >
> http://www.slideshare.net/JiangjieQin/no-data-loss-pipeline-with-apache-kafka-49753844
> >
> > -James
> >
> > > 2016-01-30 13:18 GMT+01:00 Franco Giacosa <fg...@gmail.com>:
> > >
> > >> Hi,
> > >>
> > >> The at-least-once delivery is generated in part by the network fails
> and
> > >> the retries (that may generate duplicates) right?
> > >>
> > >> In the event of a duplicated (there was an error but the first message
> > >> landed ok on the partition P1) the producer will recalculate the
> > partition
> > >> on the retry? is this done automatically?
> > >>
> > >> If in the retry the partition doesn't change and there is only 1
> > Producer,
> > >> will the duplicated be written next to the original? I mean if I
> poll()
> > >> they will come one after the other?
> > >>
> > >>
> > >>
> >
> >
> > ________________________________
> >
> > This email and any attachments may contain confidential and privileged
> > material for the sole use of the intended recipient. Any review, copying,
> > or distribution of this email (or any attachments) by others is
> prohibited.
> > If you are not the intended recipient, please contact the sender
> > immediately and permanently delete this email and any attachments. No
> > employee or agent of TiVo Inc. is authorized to conclude any binding
> > agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> > Inc. may only be made by a signed written agreement.
> >
>
Re: at-least-once delivery
Posted by Franco Giacosa <fg...@gmail.com>.
Thanks for the information James, the slides are really good.
One question, in the new producer the property block.on.buffer.full (in the
slides put this value in TRUE is a good practice, I image that this will
avoid a buffer overflow) is deprecated, and instead the use of max.block.ms,
which block for X amount of ms, can someone tell me what a good value will
be for this property in order to mimic the behaviour of
block.on.buffer.full?
Thanks
2016-01-31 6:09 GMT+01:00 James Cheng <jc...@tivo.com>:
>
> > On Jan 30, 2016, at 4:21 AM, Franco Giacosa <fg...@gmail.com> wrote:
> >
> > Sorry, this solved my questions: "Setting a value greater than zero will
> > cause the client to resend any record whose send fails with a potentially
> > transient error. Note that this retry is no different than if the client
> > resent the record upon receiving the error. Allowing retries will
> > potentially change the ordering of records because if two records are
> sent
> > to a single partition, and the first fails and is retried but the second
> > succeeds, then the second record may appear first."
> >
>
> Franco,
>
> Also, you can avoid the message reordering issue in that description by
> setting max.in.flight.requests.per.connector to 1.
>
> This slide deck has good guidelines on the types of things you are talking
> about:
>
> http://www.slideshare.net/JiangjieQin/no-data-loss-pipeline-with-apache-kafka-49753844
>
> -James
>
> > 2016-01-30 13:18 GMT+01:00 Franco Giacosa <fg...@gmail.com>:
> >
> >> Hi,
> >>
> >> The at-least-once delivery is generated in part by the network fails and
> >> the retries (that may generate duplicates) right?
> >>
> >> In the event of a duplicated (there was an error but the first message
> >> landed ok on the partition P1) the producer will recalculate the
> partition
> >> on the retry? is this done automatically?
> >>
> >> If in the retry the partition doesn't change and there is only 1
> Producer,
> >> will the duplicated be written next to the original? I mean if I poll()
> >> they will come one after the other?
> >>
> >>
> >>
>
>
> ________________________________
>
> This email and any attachments may contain confidential and privileged
> material for the sole use of the intended recipient. Any review, copying,
> or distribution of this email (or any attachments) by others is prohibited.
> If you are not the intended recipient, please contact the sender
> immediately and permanently delete this email and any attachments. No
> employee or agent of TiVo Inc. is authorized to conclude any binding
> agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> Inc. may only be made by a signed written agreement.
>
Re: at-least-once delivery
Posted by James Cheng <jc...@tivo.com>.
> On Jan 30, 2016, at 4:21 AM, Franco Giacosa <fg...@gmail.com> wrote:
>
> Sorry, this solved my questions: "Setting a value greater than zero will
> cause the client to resend any record whose send fails with a potentially
> transient error. Note that this retry is no different than if the client
> resent the record upon receiving the error. Allowing retries will
> potentially change the ordering of records because if two records are sent
> to a single partition, and the first fails and is retried but the second
> succeeds, then the second record may appear first."
>
Franco,
Also, you can avoid the message reordering issue in that description by setting max.in.flight.requests.per.connector to 1.
This slide deck has good guidelines on the types of things you are talking about:
http://www.slideshare.net/JiangjieQin/no-data-loss-pipeline-with-apache-kafka-49753844
-James
> 2016-01-30 13:18 GMT+01:00 Franco Giacosa <fg...@gmail.com>:
>
>> Hi,
>>
>> The at-least-once delivery is generated in part by the network fails and
>> the retries (that may generate duplicates) right?
>>
>> In the event of a duplicated (there was an error but the first message
>> landed ok on the partition P1) the producer will recalculate the partition
>> on the retry? is this done automatically?
>>
>> If in the retry the partition doesn't change and there is only 1 Producer,
>> will the duplicated be written next to the original? I mean if I poll()
>> they will come one after the other?
>>
>>
>>
________________________________
This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of TiVo Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo Inc. may only be made by a signed written agreement.
Re: at-least-once delivery
Posted by Franco Giacosa <fg...@gmail.com>.
Sorry, this solved my questions: "Setting a value greater than zero will
cause the client to resend any record whose send fails with a potentially
transient error. Note that this retry is no different than if the client
resent the record upon receiving the error. Allowing retries will
potentially change the ordering of records because if two records are sent
to a single partition, and the first fails and is retried but the second
succeeds, then the second record may appear first."
2016-01-30 13:18 GMT+01:00 Franco Giacosa <fg...@gmail.com>:
> Hi,
>
> The at-least-once delivery is generated in part by the network fails and
> the retries (that may generate duplicates) right?
>
> In the event of a duplicated (there was an error but the first message
> landed ok on the partition P1) the producer will recalculate the partition
> on the retry? is this done automatically?
>
> If in the retry the partition doesn't change and there is only 1 Producer,
> will the duplicated be written next to the original? I mean if I poll()
> they will come one after the other?
>
>
>