[jira] [Commented] (IGNITE-10563) Allow manual fsync for WAL

["Stanislav Lukyanov (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IGNITE-10563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17168436#comment-17168436 ] 

Stanislav Lukyanov commented on IGNITE-10563:
---------------------------------------------

Have been looking into this recently.

It seems that most DBs don't really provide a way to flush data on disk manually and just rely on the fsync frequency settings.
The only DB I found that has a manual fsync is MongoDB - https://docs.mongodb.com/manual/reference/command/fsync/.

One could achieve manual fsync on the system level as well by calling `sync(1)` command https://ss64.com/bash/sync.html on every server. This will not work for BACKGROUND with IGNITE_WAL_MMAP=false though as in this case the data is on a Java queue and not a system buffer.

Another topic is checkpointing. Would we want to have a separate command just for WAL fsync, or maybe we just have a command for forcing a checkpoint (which calls fsync on WAL anyway)?

> Allow manual fsync for WAL
> --------------------------
>
>                 Key: IGNITE-10563
>                 URL: https://issues.apache.org/jira/browse/IGNITE-10563
>             Project: Ignite
>          Issue Type: Improvement
>            Reporter: Stanislav Lukyanov
>            Priority: Major
>
> When walMode is set to LOG_ONLY or BACKGROUND there is a gap between successful return of cache write operations and actual of the data to the persistent memory. This gap is, while usually low, generally can be of any length and depends on low-level system parameters (e.g. sysctl memory and IO settings on Linux).
> On the other hand, there are situations when a user may want to ensure that at certain points all writes have been propagated to the disk.
> For example, say a large batch of data is being inserted into Ignite from an upstream system. After finishing the batch we want to ensure that all of the data is secure, and after that we remove the batch from the upstream storage. In other words, we want
> {code}
> map = upstream.getData();
> cache.putAll(map);
> cache.fsync(); // <-------
> upstream.removeAll(map);
> {code}
> Currently there is no easy way to ensure that certain write (or all writes until a point in time) has been flushed to a device. We can only rely on settings like WAL sync interval, checkpoint timeout, etc.
> It would be nice to have a way to manually call fsync() for WAL to have a strong guarantee that all previous updates have been fully written on disk.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)