WAL cleanup

[Surinder Mehra <re...@gmail.com>]

Hi,
I have a question about WAL file cleanup.
1. A cache update appends updates to WAl file and writes it to RAM. WAL can
rollover older than 10 files to WalArchive.
2. So checkpoint moves data from RAM to Disk

As per docs link below, after checkpoint, WAl files created before it can
be deleted. I also saw WAL cleanup logs in ignite nodes.
So I just want to know more about the cleanup process. Who does the
cleanup, checkpointing thread or separate cleaner process ?

Can someone please elaborate on it or share a link to docs which explain it.

https://ignite.apache.org/docs/latest/persistence/native-persistence#:~:text=After%20a%20checkpoint%20is%20passed,before%20that%20point%20in%20time
.

Re:WAL cleanup

[ткаленко кирилл <tk...@yandex.ru>]

Checkpoints happen in the following cases: on timeout, but padding one of the persistent data region by 0.75 (many dirty pages) if segments without a breakpoint are greater than 0.25% (i.e. 3 out of 10).

If all 10 segments are filled, then it will be expected (wait) for the first of the segments to go to the archive.

Transferring the segment to the archive is possible only after the checkpoint, since we have accurately written the data to disk.

The archive is an optimization that can be disabled.

Segments may be useful for rebalancing instead of a full data transfer, only the difference will be transferred.

In theory, exceeding the maximum is possible, but already lower than it was before.

Re: WAL cleanup

[Surinder Mehra <re...@gmail.com>]

Ok thanks, just more query.

"If all 10 segments are filled, then it will be expected (wait) for the
first of the segments to go to the archive."

Does it mean application writes will wait until one of the wal segment is
made free(that means atleast checkpoint  must complete so segment file can
move to wal archive) ?

On Fri, 22 Apr 2022, 18:06 ткаленко кирилл, <tk...@yandex.ru> wrote:

> Set the maximum to 50% of the estimated value. If it is possible to use 60
> GB for the archive, set it to 30 GB.
>

Re:WAL cleanup

[ткаленко кирилл <tk...@yandex.ru>]

Set the maximum to 50% of the estimated value. If it is possible to use 60 GB for the archive, set it to 30 GB.

Re: WAL cleanup

[Surinder Mehra <re...@gmail.com>]

Sorry for typo in last thread point 5, I meant to type "the ones which
moves segments from WAL to WAL Archive"

On Fri, 22 Apr 2022, 17:13 Surinder Mehra, <re...@gmail.com> wrote:

> Ok thanks for clarifying. O have follow up queries
> 1. Wal has only 10 segments and default checkpoint frequency is 3 minutes
> What if application writes fill up all 10 segments before next checkpoint
> interval, would it trigger another checkpoint followed by moving of segments
>
> 2. Does it always move segments to archive after checkpoint? What if
> checkpoint is taking time and no space left in WAL. i.e. wal reached its
> max size(10*segmentsize)
>
> 3. If point 2 is true, why do we need walarchive. Since data has been
> flushed to disk with checkpoint. And any data not checkpointed is present
> in WAL, why do we need archive directory.
>
> 4. I have seen WAL size fixed(10G with segment size as 1G) to max size and
> WAL archive increases to 60GB. Does it mean cleaner thread wasn't frequent
> enough?
> 5. Are cleaner threads and the ones which love segments have any relation
> to checkpoint process or performance impact. If so, are they tunable by
> user ?
>
>
> On Fri, 22 Apr 2022, 16:52 ткаленко кирилл, <tk...@yandex.ru> wrote:
>
>> Cleaning takes place in a separate thread.
>> Transfer of segments occurs in a separate thread after the checkpoint.
>>
>>

Re: WAL cleanup

[Surinder Mehra <re...@gmail.com>]

Ok thanks for clarifying. O have follow up queries
1. Wal has only 10 segments and default checkpoint frequency is 3 minutes
What if application writes fill up all 10 segments before next checkpoint
interval, would it trigger another checkpoint followed by moving of segments

2. Does it always move segments to archive after checkpoint? What if
checkpoint is taking time and no space left in WAL. i.e. wal reached its
max size(10*segmentsize)

3. If point 2 is true, why do we need walarchive. Since data has been
flushed to disk with checkpoint. And any data not checkpointed is present
in WAL, why do we need archive directory.

4. I have seen WAL size fixed(10G with segment size as 1G) to max size and
WAL archive increases to 60GB. Does it mean cleaner thread wasn't frequent
enough?
5. Are cleaner threads and the ones which love segments have any relation
to checkpoint process or performance impact. If so, are they tunable by
user ?


On Fri, 22 Apr 2022, 16:52 ткаленко кирилл, <tk...@yandex.ru> wrote:

> Cleaning takes place in a separate thread.
> Transfer of segments occurs in a separate thread after the checkpoint.
>
>

Re:WAL cleanup

[ткаленко кирилл <tk...@yandex.ru>]

Cleaning takes place in a separate thread.
Transfer of segments occurs in a separate thread after the checkpoint.