You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Raymond Wilson <ra...@trimble.com> on 2018/10/01 19:20:59 UTC
Clarifying aspects of write ahead logs
Hi,
I’ve been reading through this page (
https://apacheignite.readme.io/docs/write-ahead-log) on WAL files in Ignite.
I’d like to understand what the purpose of the WAL archive is. I understand
from the documentation that the WAL archive contains data that has been
checkpointed, and so safely committed to the cache partition files.
It seems to be optional in that it is possible to not have one by setting
the WAL active and archive paths to be the same location. The documentation
recommends doing this if ingest volumes mean time spent copying WAL files
to the archive can cause node freezes.
Is the WAL archive used in recovery operations when an Ignite node
restarts? Is there a downside to not having WAL segments archived?
Thanks,
Raymond.
Re: Clarifying aspects of write ahead logs
Posted by Dmitriy Pavlov <dp...@gmail.com>.
Hi Raymond,
No archiver mode means there is no cyclic enumeration of files in the work
directory, enumeration of segments is plain and absolute instead.
And all walHistSize rules applied to default mode applies to no-archiver
mode, as well. So there is not possible to lose any data, and no additional
checkpoints will be forced.
Sincerely,
Dmitriy Pavlov
ср, 3 окт. 2018 г. в 23:49, Raymond Wilson <ra...@trimble.com>:
> HI Dmitriy,
>
>
>
> Thanks for clarifying.
>
>
>
> If archiving is turned off, what happens when all the WAL segments are
> filled before a checkpoint occurs? Is a checkpoint forced at that point, or
> does Ignite increase the number of segment files used above workSegments,
> or is there a potential for data loss?
>
>
>
> Raymond.
>
>
>
> *From:* Dmitriy Pavlov <dp...@gmail.com>
> *Sent:* Tuesday, October 2, 2018 8:42 AM
> *To:* user@ignite.apache.org
> *Subject:* Re: Clarifying aspects of write ahead logs
>
>
>
> Hi Raymond,
>
>
>
> WAL Archive can be used for recovery during node restart. It can happen if
> WAL data during one checkpoint is longer than segmentSize*workSegments
> (640Mb by default).
>
>
>
> The downside of not having archiver is following: Space allocation for
> a file may require significant time in some file systems. If we archive one
> WAL segment (asynchronously) we can later reuse this segment file in the
> work directory for a new segment records. The file is now preallocated,
> there will be no performance penalty for step by step increasing of its
> volume.
>
>
>
> Sincerely,
>
> Dmitriy Pavlov
>
>
>
> You can find scheme of WAL rotation here:
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-WALstructure
>
>
>
>
> пн, 1 окт. 2018 г. в 22:21, Raymond Wilson <ra...@trimble.com>:
>
> Hi,
>
>
>
> I’ve been reading through this page (
> https://apacheignite.readme.io/docs/write-ahead-log) on WAL files in
> Ignite.
>
>
>
> I’d like to understand what the purpose of the WAL archive is. I
> understand from the documentation that the WAL archive contains data that
> has been checkpointed, and so safely committed to the cache partition files.
>
>
>
> It seems to be optional in that it is possible to not have one by setting
> the WAL active and archive paths to be the same location. The documentation
> recommends doing this if ingest volumes mean time spent copying WAL files
> to the archive can cause node freezes.
>
>
>
> Is the WAL archive used in recovery operations when an Ignite node
> restarts? Is there a downside to not having WAL segments archived?
>
>
>
> Thanks,
>
> Raymond.
>
>
>
>
RE: Clarifying aspects of write ahead logs
Posted by Raymond Wilson <ra...@trimble.com>.
HI Dmitriy,
Thanks for clarifying.
If archiving is turned off, what happens when all the WAL segments are
filled before a checkpoint occurs? Is a checkpoint forced at that point, or
does Ignite increase the number of segment files used above workSegments,
or is there a potential for data loss?
Raymond.
*From:* Dmitriy Pavlov <dp...@gmail.com>
*Sent:* Tuesday, October 2, 2018 8:42 AM
*To:* user@ignite.apache.org
*Subject:* Re: Clarifying aspects of write ahead logs
Hi Raymond,
WAL Archive can be used for recovery during node restart. It can happen if
WAL data during one checkpoint is longer than segmentSize*workSegments
(640Mb by default).
The downside of not having archiver is following: Space allocation for
a file may require significant time in some file systems. If we archive one
WAL segment (asynchronously) we can later reuse this segment file in the
work directory for a new segment records. The file is now preallocated,
there will be no performance penalty for step by step increasing of its
volume.
Sincerely,
Dmitriy Pavlov
You can find scheme of WAL rotation here:
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-WALstructure
пн, 1 окт. 2018 г. в 22:21, Raymond Wilson <ra...@trimble.com>:
Hi,
I’ve been reading through this page (
https://apacheignite.readme.io/docs/write-ahead-log) on WAL files in Ignite.
I’d like to understand what the purpose of the WAL archive is. I understand
from the documentation that the WAL archive contains data that has been
checkpointed, and so safely committed to the cache partition files.
It seems to be optional in that it is possible to not have one by setting
the WAL active and archive paths to be the same location. The documentation
recommends doing this if ingest volumes mean time spent copying WAL files
to the archive can cause node freezes.
Is the WAL archive used in recovery operations when an Ignite node
restarts? Is there a downside to not having WAL segments archived?
Thanks,
Raymond.
Re: Clarifying aspects of write ahead logs
Posted by Dmitriy Pavlov <dp...@gmail.com>.
Hi Raymond,
WAL Archive can be used for recovery during node restart. It can happen if
WAL data during one checkpoint is longer than segmentSize*workSegments
(640Mb by default).
The downside of not having archiver is following: Space allocation for
a file may require significant time in some file systems. If we archive one
WAL segment (asynchronously) we can later reuse this segment file in the
work directory for a new segment records. The file is now preallocated,
there will be no performance penalty for step by step increasing of its
volume.
Sincerely,
Dmitriy Pavlov
You can find scheme of WAL rotation here:
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-WALstructure
пн, 1 окт. 2018 г. в 22:21, Raymond Wilson <ra...@trimble.com>:
> Hi,
>
>
>
> I’ve been reading through this page (
> https://apacheignite.readme.io/docs/write-ahead-log) on WAL files in
> Ignite.
>
>
>
> I’d like to understand what the purpose of the WAL archive is. I
> understand from the documentation that the WAL archive contains data that
> has been checkpointed, and so safely committed to the cache partition files.
>
>
>
> It seems to be optional in that it is possible to not have one by setting
> the WAL active and archive paths to be the same location. The documentation
> recommends doing this if ingest volumes mean time spent copying WAL files
> to the archive can cause node freezes.
>
>
>
> Is the WAL archive used in recovery operations when an Ignite node
> restarts? Is there a downside to not having WAL segments archived?
>
>
>
> Thanks,
>
> Raymond.
>
>
>