You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@ignite.apache.org by Maksim Timonin <ti...@apache.org> on 2022/05/16 08:52:20 UTC

[DISCUSSION] IEP-89: PITR

Dear Igniters,

I just published IEP-89 [1] that proposes a new feature to Ignite - (point
in time recovery) PITR. I propose to implement the Consistent Cut algorithm
for this, actually I achieved a working PoC for Ignite. And based on my
research I wrote this IEP.

Let's start a discussion here. Any questions or comments are welcomed.

Thanks!


[1]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211884314

Re: [DISCUSSION] IEP-89: PITR

Posted by Maksim Timonin <ti...@apache.org>.

Hi Igniters!

There are some updates for IEP-89 [1]. It proposes to use an incremental
snapshot for PITR.

Incremental snapshots is a lightweight alternative for Ignite full
snapshot. Incremental snapshots are created by a non-blocking Consistent
Cut algorithm, and the incremental snapshot contains a collection of WAL
segments that will be applied to the full snapshot.

IEP contains some open questions:
1. How to handle rebalance data.
2. Whether to force creation of full snapshot after restoring an
incremental snapshot.

Any opinions, questions or comments are welcomed.

Thanks!

[1]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211884314


On Sun, May 29, 2022 at 7:38 PM Roman Puchkovskiy <
roman.puchkovskiy@gmail.com> wrote:

> Maksim,
>
> Thank you for the added clarifications, the document has become a lot
> clearer to me.
>
> пт, 27 мая 2022 г. в 18:55, Maksim Timonin <ti...@apache.org>:
> >
> > Hi Roman,
> >
> > Thanks for your comments! Yes, at the beginning I considered using a
> single class for two types of records. But it is confusing, as you're
> correctly noticed. I reworked IEP, now there are 2 different classes, and
> both contain a list of transactions to include. Also I added info about
> when those WAL records are actually written to 'WAL records' sections.
> Please, have a look.
> >
> > Thanks,
> > Maksim
> >
> > On Thu, May 26, 2022 at 2:54 PM Roman Puchkovskiy <
> roman.puchkovskiy@gmail.com> wrote:
> >>
> >> Hi Maksim.
> >>
> >> Do I understand correctly that 'consistent cut start' message always
> >> contains IDs of transactions to include, while 'consistent cut finish'
> >> message always contains IDs of transactions to exclude from the
> >> consistent cut? (At least, this is the impression I got from the
> >> example of parsing the WAL and the accompanying figure). If this is
> >> the case, then it looks like the `include` and `check` fields are
> >> mutually exclusive in ConsistentCutRecord. Would it make sense to
> >> replace it with two classes, like ConsistentCutStartRecord(cutVer,
> >> include) and ConsistentCutFinishRecord(cutVer, exclude)?
> >>
> >> Also, it seems that it could be beneficial to have a separate section
> >> explaining when the corresponding records are written to WAL, to make
> >> this information easier to find. Or, maybe, this could be added to the
> >> current 'WAL records' section.
> >>
> >> пн, 16 мая 2022 г. в 12:52, Maksim Timonin <ti...@apache.org>:
> >> >
> >> > Dear Igniters,
> >> >
> >> > I just published IEP-89 [1] that proposes a new feature to Ignite -
> (point
> >> > in time recovery) PITR. I propose to implement the Consistent Cut
> algorithm
> >> > for this, actually I achieved a working PoC for Ignite. And based on
> my
> >> > research I wrote this IEP.
> >> >
> >> > Let's start a discussion here. Any questions or comments are welcomed.
> >> >
> >> > Thanks!
> >> >
> >> >
> >> > [1]
> >> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211884314
>

Re: [DISCUSSION] IEP-89: PITR

Posted by Maksim Timonin <ti...@apache.org>.

Hi Roman,

Thanks for your comments! Yes, at the beginning I considered using a single
class for two types of records. But it is confusing, as you're correctly
noticed. I reworked IEP, now there are 2 different classes, and both
contain a list of transactions to include. Also I added info about when
those WAL records are actually written to 'WAL records' sections. Please,
have a look.

Thanks,
Maksim

On Thu, May 26, 2022 at 2:54 PM Roman Puchkovskiy <
roman.puchkovskiy@gmail.com> wrote:

> Hi Maksim.
>
> Do I understand correctly that 'consistent cut start' message always
> contains IDs of transactions to include, while 'consistent cut finish'
> message always contains IDs of transactions to exclude from the
> consistent cut? (At least, this is the impression I got from the
> example of parsing the WAL and the accompanying figure). If this is
> the case, then it looks like the `include` and `check` fields are
> mutually exclusive in ConsistentCutRecord. Would it make sense to
> replace it with two classes, like ConsistentCutStartRecord(cutVer,
> include) and ConsistentCutFinishRecord(cutVer, exclude)?
>
> Also, it seems that it could be beneficial to have a separate section
> explaining when the corresponding records are written to WAL, to make
> this information easier to find. Or, maybe, this could be added to the
> current 'WAL records' section.
>
> пн, 16 мая 2022 г. в 12:52, Maksim Timonin <ti...@apache.org>:
> >
> > Dear Igniters,
> >
> > I just published IEP-89 [1] that proposes a new feature to Ignite -
> (point
> > in time recovery) PITR. I propose to implement the Consistent Cut
> algorithm
> > for this, actually I achieved a working PoC for Ignite. And based on my
> > research I wrote this IEP.
> >
> > Let's start a discussion here. Any questions or comments are welcomed.
> >
> > Thanks!
> >
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211884314
>

Re: [DISCUSSION] IEP-89: PITR

Posted by Roman Puchkovskiy <ro...@gmail.com>.

Hi Maksim.

Do I understand correctly that 'consistent cut start' message always
contains IDs of transactions to include, while 'consistent cut finish'
message always contains IDs of transactions to exclude from the
consistent cut? (At least, this is the impression I got from the
example of parsing the WAL and the accompanying figure). If this is
the case, then it looks like the `include` and `check` fields are
mutually exclusive in ConsistentCutRecord. Would it make sense to
replace it with two classes, like ConsistentCutStartRecord(cutVer,
include) and ConsistentCutFinishRecord(cutVer, exclude)?

Also, it seems that it could be beneficial to have a separate section
explaining when the corresponding records are written to WAL, to make
this information easier to find. Or, maybe, this could be added to the
current 'WAL records' section.

пн, 16 мая 2022 г. в 12:52, Maksim Timonin <ti...@apache.org>:
>
> Dear Igniters,
>
> I just published IEP-89 [1] that proposes a new feature to Ignite - (point
> in time recovery) PITR. I propose to implement the Consistent Cut algorithm
> for this, actually I achieved a working PoC for Ignite. And based on my
> research I wrote this IEP.
>
> Let's start a discussion here. Any questions or comments are welcomed.
>
> Thanks!
>
>
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211884314