Re: at-least-once delivery

[Franco Giacosa <fg...@gmail.com>]

Thanks for the information James, the slides are really good.

One question, in the new producer the property block.on.buffer.full (in the
slides put this value in TRUE is a good practice, I image that this will
avoid a buffer overflow) is deprecated, and instead the use of max.block.ms,
which block for X amount of ms, can someone tell me what a good value will
be for this property in order to mimic the behaviour of
block.on.buffer.full?

Thanks

2016-01-31 6:09 GMT+01:00 James Cheng <jc...@tivo.com>:

>
> > On Jan 30, 2016, at 4:21 AM, Franco Giacosa <fg...@gmail.com> wrote:
> >
> > Sorry, this solved my questions: "Setting a value greater than zero will
> > cause the client to resend any record whose send fails with a potentially
> > transient error. Note that this retry is no different than if the client
> > resent the record upon receiving the error. Allowing retries will
> > potentially change the ordering of records because if two records are
> sent
> > to a single partition, and the first fails and is retried but the second
> > succeeds, then the second record may appear first."
> >
>
> Franco,
>
> Also, you can avoid the message reordering issue in that description by
> setting max.in.flight.requests.per.connector to 1.
>
> This slide deck has good guidelines on the types of things you are talking
> about:
>
> http://www.slideshare.net/JiangjieQin/no-data-loss-pipeline-with-apache-kafka-49753844
>
> -James
>
> > 2016-01-30 13:18 GMT+01:00 Franco Giacosa <fg...@gmail.com>:
> >
> >> Hi,
> >>
> >> The at-least-once delivery is generated in part by the network fails and
> >> the retries (that may generate duplicates) right?
> >>
> >> In the event of a duplicated (there was an error but the first message
> >> landed ok on the partition P1) the producer will recalculate the
> partition
> >> on the retry? is this done automatically?
> >>
> >> If in the retry the partition doesn't change and there is only 1
> Producer,
> >> will the duplicated be written next to the original? I mean if I poll()
> >> they will come one after the other?
> >>
> >>
> >>
>
>
> ________________________________
>
> This email and any attachments may contain confidential and privileged
> material for the sole use of the intended recipient. Any review, copying,
> or distribution of this email (or any attachments) by others is prohibited.
> If you are not the intended recipient, please contact the sender
> immediately and permanently delete this email and any attachments. No
> employee or agent of TiVo Inc. is authorized to conclude any binding
> agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> Inc. may only be made by a signed written agreement.
>

Re: at-least-once delivery

[Gwen Shapira <gw...@confluent.io>]

MAX_INT is a good value if you want to just block until the buffer has some
space (and never get an exception).

On Tue, Feb 2, 2016 at 8:08 AM, Franco Giacosa <fg...@gmail.com> wrote:

> Thanks for the information James, the slides are really good.
>
> One question, in the new producer the property block.on.buffer.full (in the
> slides put this value in TRUE is a good practice, I image that this will
> avoid a buffer overflow) is deprecated, and instead the use of
> max.block.ms,
> which block for X amount of ms, can someone tell me what a good value will
> be for this property in order to mimic the behaviour of
> block.on.buffer.full?
>
> Thanks
>
> 2016-01-31 6:09 GMT+01:00 James Cheng <jc...@tivo.com>:
>
> >
> > > On Jan 30, 2016, at 4:21 AM, Franco Giacosa <fg...@gmail.com>
> wrote:
> > >
> > > Sorry, this solved my questions: "Setting a value greater than zero
> will
> > > cause the client to resend any record whose send fails with a
> potentially
> > > transient error. Note that this retry is no different than if the
> client
> > > resent the record upon receiving the error. Allowing retries will
> > > potentially change the ordering of records because if two records are
> > sent
> > > to a single partition, and the first fails and is retried but the
> second
> > > succeeds, then the second record may appear first."
> > >
> >
> > Franco,
> >
> > Also, you can avoid the message reordering issue in that description by
> > setting max.in.flight.requests.per.connector to 1.
> >
> > This slide deck has good guidelines on the types of things you are
> talking
> > about:
> >
> >
> http://www.slideshare.net/JiangjieQin/no-data-loss-pipeline-with-apache-kafka-49753844
> >
> > -James
> >
> > > 2016-01-30 13:18 GMT+01:00 Franco Giacosa <fg...@gmail.com>:
> > >
> > >> Hi,
> > >>
> > >> The at-least-once delivery is generated in part by the network fails
> and
> > >> the retries (that may generate duplicates) right?
> > >>
> > >> In the event of a duplicated (there was an error but the first message
> > >> landed ok on the partition P1) the producer will recalculate the
> > partition
> > >> on the retry? is this done automatically?
> > >>
> > >> If in the retry the partition doesn't change and there is only 1
> > Producer,
> > >> will the duplicated be written next to the original? I mean if I
> poll()
> > >> they will come one after the other?
> > >>
> > >>
> > >>
> >
> >
> > ________________________________
> >
> > This email and any attachments may contain confidential and privileged
> > material for the sole use of the intended recipient. Any review, copying,
> > or distribution of this email (or any attachments) by others is
> prohibited.
> > If you are not the intended recipient, please contact the sender
> > immediately and permanently delete this email and any attachments. No
> > employee or agent of TiVo Inc. is authorized to conclude any binding
> > agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> > Inc. may only be made by a signed written agreement.
> >
>