Re: [DISCUSSION] IEP-89: PITR

[Maksim Timonin <ti...@apache.org>]

Hi Igniters!

There are some updates for IEP-89 [1]. It proposes to use an incremental
snapshot for PITR.

Incremental snapshots is a lightweight alternative for Ignite full
snapshot. Incremental snapshots are created by a non-blocking Consistent
Cut algorithm, and the incremental snapshot contains a collection of WAL
segments that will be applied to the full snapshot.

IEP contains some open questions:
1. How to handle rebalance data.
2. Whether to force creation of full snapshot after restoring an
incremental snapshot.

Any opinions, questions or comments are welcomed.

Thanks!

[1]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211884314


On Sun, May 29, 2022 at 7:38 PM Roman Puchkovskiy <
roman.puchkovskiy@gmail.com> wrote:

> Maksim,
>
> Thank you for the added clarifications, the document has become a lot
> clearer to me.
>
> пт, 27 мая 2022 г. в 18:55, Maksim Timonin <ti...@apache.org>:
> >
> > Hi Roman,
> >
> > Thanks for your comments! Yes, at the beginning I considered using a
> single class for two types of records. But it is confusing, as you're
> correctly noticed. I reworked IEP, now there are 2 different classes, and
> both contain a list of transactions to include. Also I added info about
> when those WAL records are actually written to 'WAL records' sections.
> Please, have a look.
> >
> > Thanks,
> > Maksim
> >
> > On Thu, May 26, 2022 at 2:54 PM Roman Puchkovskiy <
> roman.puchkovskiy@gmail.com> wrote:
> >>
> >> Hi Maksim.
> >>
> >> Do I understand correctly that 'consistent cut start' message always
> >> contains IDs of transactions to include, while 'consistent cut finish'
> >> message always contains IDs of transactions to exclude from the
> >> consistent cut? (At least, this is the impression I got from the
> >> example of parsing the WAL and the accompanying figure). If this is
> >> the case, then it looks like the `include` and `check` fields are
> >> mutually exclusive in ConsistentCutRecord. Would it make sense to
> >> replace it with two classes, like ConsistentCutStartRecord(cutVer,
> >> include) and ConsistentCutFinishRecord(cutVer, exclude)?
> >>
> >> Also, it seems that it could be beneficial to have a separate section
> >> explaining when the corresponding records are written to WAL, to make
> >> this information easier to find. Or, maybe, this could be added to the
> >> current 'WAL records' section.
> >>
> >> пн, 16 мая 2022 г. в 12:52, Maksim Timonin <ti...@apache.org>:
> >> >
> >> > Dear Igniters,
> >> >
> >> > I just published IEP-89 [1] that proposes a new feature to Ignite -
> (point
> >> > in time recovery) PITR. I propose to implement the Consistent Cut
> algorithm
> >> > for this, actually I achieved a working PoC for Ignite. And based on
> my
> >> > research I wrote this IEP.
> >> >
> >> > Let's start a discussion here. Any questions or comments are welcomed.
> >> >
> >> > Thanks!
> >> >
> >> >
> >> > [1]
> >> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211884314
>