You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Péter Sinóros-Szabó <pe...@transferwise.com.INVALID> on 2020/04/22 08:39:16 UTC
MirrorMaker2 ordering guarantees
Hey,
so KIP-382 mentions that:
"Partitioning and order of records is preserved between source and remote
topics."
is the ordering of messages (I guess only in a partition) something that is
actually implemented in 2.4 (or in 2.5)?
Or do I need to set `max.in.flight.requests.per.connection=1` ?
Thanks,
Peter
Re: MirrorMaker2 ordering guarantees
Posted by Péter Sinóros-Szabó <pe...@transferwise.com.INVALID>.
Hey,
so I was just about to set `max.in.flight.requests.per.connection=1` when I
found in the logs that it is already set for the "destination" producers :)
Cool.
Duplicates are fine for me, we have dedup checks or immutable operations on
the consumers.
Have a nice day!
Peter
On Mon, 27 Apr 2020 at 19:45, Ryanne Dolan <ry...@gmail.com> wrote:
> > Could the `max.in.flight.requests.per.connection=1` parameter help to
> > prevent the "slightly out-of-order records"?
>
> Yes that helps, but dupes are still possible when MM2 restarts or
> rebalances, since it will restart at the latest commit. If you are
> replicating something like CDC or changelogs, then dupes might be fine
> (downstream state will be eventually consistent). That's a common pattern
> with MM1 as well.
>
> Ryanne
>
> On Mon, Apr 27, 2020 at 4:47 AM Péter Sinóros-Szabó
> <pe...@transferwise.com.invalid> wrote:
>
> > Hey Ryanne,
> >
> > Is there any documentation where I can read more about this "slightly
> > out-of-order records"?
> > It would help very much to see how we can use MM2 in our systems.
> >
> > Thanks,
> > Peter
> >
> > On Thu, 23 Apr 2020 at 08:56, Péter Sinóros-Szabó <
> > peter.sinoros-szabo@transferwise.com> wrote:
> >
> > > Hey Ryanna,
> > >
> > > Could the `max.in.flight.requests.per.connection=1` parameter help to
> > > prevent the "slightly out-of-order records"?
> > > Or is there any workaround for that? Duplicates are fine for me, but
> I'd
> > > like to have the same order of messages too.
> > > Can you please add some more detail about why those "slightly
> > out-of-order
> > > records" may happen?
> > >
> > > Thanks,
> > > Peter
> > >
> > > On Wed, 22 Apr 2020 at 20:16, Ryanne Dolan <ry...@gmail.com>
> > wrote:
> > >
> > >> Hey Peter, Connect will need to support transactions before we can
> > >> guarantee the order of records in remote topics. We can guarantee that
> > no
> > >> records are dropped or skipped, even during consumer
> failover/migration
> > >> etc, but we can still have duplicates and slightly out-of-order
> records
> > in
> > >> the downstream remote topics, for now.
> > >>
> > >> Ryanne
> > >>
> > >> On Wed, Apr 22, 2020 at 3:39 AM Péter Sinóros-Szabó
> > >> <pe...@transferwise.com.invalid> wrote:
> > >>
> > >> > Hey,
> > >> >
> > >> > so KIP-382 mentions that:
> > >> > "Partitioning and order of records is preserved between source and
> > >> remote
> > >> > topics."
> > >> > is the ordering of messages (I guess only in a partition) something
> > >> that is
> > >> > actually implemented in 2.4 (or in 2.5)?
> > >> >
> > >> > Or do I need to set `max.in.flight.requests.per.connection=1` ?
> > >> >
> > >> > Thanks,
> > >> > Peter
> > >> >
> > >>
> > >
> >
>
Re: MirrorMaker2 ordering guarantees
Posted by Ryanne Dolan <ry...@gmail.com>.
> Could the `max.in.flight.requests.per.connection=1` parameter help to
> prevent the "slightly out-of-order records"?
Yes that helps, but dupes are still possible when MM2 restarts or
rebalances, since it will restart at the latest commit. If you are
replicating something like CDC or changelogs, then dupes might be fine
(downstream state will be eventually consistent). That's a common pattern
with MM1 as well.
Ryanne
On Mon, Apr 27, 2020 at 4:47 AM Péter Sinóros-Szabó
<pe...@transferwise.com.invalid> wrote:
> Hey Ryanne,
>
> Is there any documentation where I can read more about this "slightly
> out-of-order records"?
> It would help very much to see how we can use MM2 in our systems.
>
> Thanks,
> Peter
>
> On Thu, 23 Apr 2020 at 08:56, Péter Sinóros-Szabó <
> peter.sinoros-szabo@transferwise.com> wrote:
>
> > Hey Ryanna,
> >
> > Could the `max.in.flight.requests.per.connection=1` parameter help to
> > prevent the "slightly out-of-order records"?
> > Or is there any workaround for that? Duplicates are fine for me, but I'd
> > like to have the same order of messages too.
> > Can you please add some more detail about why those "slightly
> out-of-order
> > records" may happen?
> >
> > Thanks,
> > Peter
> >
> > On Wed, 22 Apr 2020 at 20:16, Ryanne Dolan <ry...@gmail.com>
> wrote:
> >
> >> Hey Peter, Connect will need to support transactions before we can
> >> guarantee the order of records in remote topics. We can guarantee that
> no
> >> records are dropped or skipped, even during consumer failover/migration
> >> etc, but we can still have duplicates and slightly out-of-order records
> in
> >> the downstream remote topics, for now.
> >>
> >> Ryanne
> >>
> >> On Wed, Apr 22, 2020 at 3:39 AM Péter Sinóros-Szabó
> >> <pe...@transferwise.com.invalid> wrote:
> >>
> >> > Hey,
> >> >
> >> > so KIP-382 mentions that:
> >> > "Partitioning and order of records is preserved between source and
> >> remote
> >> > topics."
> >> > is the ordering of messages (I guess only in a partition) something
> >> that is
> >> > actually implemented in 2.4 (or in 2.5)?
> >> >
> >> > Or do I need to set `max.in.flight.requests.per.connection=1` ?
> >> >
> >> > Thanks,
> >> > Peter
> >> >
> >>
> >
>
Re: MirrorMaker2 ordering guarantees
Posted by Péter Sinóros-Szabó <pe...@transferwise.com.INVALID>.
Hey Ryanne,
Is there any documentation where I can read more about this "slightly
out-of-order records"?
It would help very much to see how we can use MM2 in our systems.
Thanks,
Peter
On Thu, 23 Apr 2020 at 08:56, Péter Sinóros-Szabó <
peter.sinoros-szabo@transferwise.com> wrote:
> Hey Ryanna,
>
> Could the `max.in.flight.requests.per.connection=1` parameter help to
> prevent the "slightly out-of-order records"?
> Or is there any workaround for that? Duplicates are fine for me, but I'd
> like to have the same order of messages too.
> Can you please add some more detail about why those "slightly out-of-order
> records" may happen?
>
> Thanks,
> Peter
>
> On Wed, 22 Apr 2020 at 20:16, Ryanne Dolan <ry...@gmail.com> wrote:
>
>> Hey Peter, Connect will need to support transactions before we can
>> guarantee the order of records in remote topics. We can guarantee that no
>> records are dropped or skipped, even during consumer failover/migration
>> etc, but we can still have duplicates and slightly out-of-order records in
>> the downstream remote topics, for now.
>>
>> Ryanne
>>
>> On Wed, Apr 22, 2020 at 3:39 AM Péter Sinóros-Szabó
>> <pe...@transferwise.com.invalid> wrote:
>>
>> > Hey,
>> >
>> > so KIP-382 mentions that:
>> > "Partitioning and order of records is preserved between source and
>> remote
>> > topics."
>> > is the ordering of messages (I guess only in a partition) something
>> that is
>> > actually implemented in 2.4 (or in 2.5)?
>> >
>> > Or do I need to set `max.in.flight.requests.per.connection=1` ?
>> >
>> > Thanks,
>> > Peter
>> >
>>
>
Re: MirrorMaker2 ordering guarantees
Posted by Péter Sinóros-Szabó <pe...@transferwise.com.INVALID>.
Hey Ryanna,
Could the `max.in.flight.requests.per.connection=1` parameter help to
prevent the "slightly out-of-order records"?
Or is there any workaround for that? Duplicates are fine for me, but I'd
like to have the same order of messages too.
Can you please add some more detail about why those "slightly out-of-order
records" may happen?
Thanks,
Peter
On Wed, 22 Apr 2020 at 20:16, Ryanne Dolan <ry...@gmail.com> wrote:
> Hey Peter, Connect will need to support transactions before we can
> guarantee the order of records in remote topics. We can guarantee that no
> records are dropped or skipped, even during consumer failover/migration
> etc, but we can still have duplicates and slightly out-of-order records in
> the downstream remote topics, for now.
>
> Ryanne
>
> On Wed, Apr 22, 2020 at 3:39 AM Péter Sinóros-Szabó
> <pe...@transferwise.com.invalid> wrote:
>
> > Hey,
> >
> > so KIP-382 mentions that:
> > "Partitioning and order of records is preserved between source and remote
> > topics."
> > is the ordering of messages (I guess only in a partition) something that
> is
> > actually implemented in 2.4 (or in 2.5)?
> >
> > Or do I need to set `max.in.flight.requests.per.connection=1` ?
> >
> > Thanks,
> > Peter
> >
>
Re: MirrorMaker2 ordering guarantees
Posted by Ryanne Dolan <ry...@gmail.com>.
Hey Peter, Connect will need to support transactions before we can
guarantee the order of records in remote topics. We can guarantee that no
records are dropped or skipped, even during consumer failover/migration
etc, but we can still have duplicates and slightly out-of-order records in
the downstream remote topics, for now.
Ryanne
On Wed, Apr 22, 2020 at 3:39 AM Péter Sinóros-Szabó
<pe...@transferwise.com.invalid> wrote:
> Hey,
>
> so KIP-382 mentions that:
> "Partitioning and order of records is preserved between source and remote
> topics."
> is the ordering of messages (I guess only in a partition) something that is
> actually implemented in 2.4 (or in 2.5)?
>
> Or do I need to set `max.in.flight.requests.per.connection=1` ?
>
> Thanks,
> Peter
>