Adding Persistent Memory Support for Ignite

["Mammo, Mulugeta" <mu...@intel.com>]

Hi all,



Ignite, when persistence mode is enabled, stores data and indexes on disk. To minimize the latency of disks, several tuning options can be applied. Setting the page size of a memory region to match the page size of the underlying storage, using a separate disk for the WAL, and using production-level SSDs are just a few of them [ https://apacheignite.readme.io/docs/durable-memory-tuning#section-native-persistence-related-tuning ].

A persistent memory store with low latency and high capacity offers a viable alternative to disks. In light of this, we are proposing to make use of our Low Level Persistent Library (LLPL), https://github.com/pmem/pcj/tree/master/LLPL, to offer a persistent memory storage for Ignite.

At this point, we envision two distinct implementation options:

1.      Data and indexes will continue to be stored in the off-heap memory but the disk will be replaced by a persistent memory. Since persistence memory in this option is not a file system, the logic currently offered by WAL file and the partition files would have to be implemented from scratch.

2.      In this option, we eliminate the current check-point process and the WAL file. We will use a memory region defined by LLPL to store data and indexes. There will be no off-heap memory. DRAM will be exclusively used to store hot cache entries just like the on-heap cache is in the current implementation.

In both cases, there are more details and subtleties that have to handled - e.g. the atomic and transactional guarantees offered. More clarifications will be given as we go along. And, feel free to provide your thoughts.

Thanks,
Mulugeta

RE: Adding Persistent Memory Support for Ignite

["Mammo, Mulugeta" <mu...@intel.com>]

Hi Dmitriy,

We're still working on the documentation of LLPL. You may forward me your questions in the meantime and I'll try to answer them.

Thanks,
Mulugeta

-----Original Message-----
From: Dmitriy Setrakyan [mailto:dsetrakyan@apache.org] 
Sent: Thursday, November 16, 2017 1:26 PM
To: dev@ignite.apache.org
Subject: Re: Adding Persistent Memory Support for Ignite

Hi Mulugeta,

Where can I find documentation about LLPL to understand how memory and persistence are handled there?

D.

On Thu, Nov 16, 2017 at 7:16 AM, Mammo, Mulugeta <mu...@intel.com>
wrote:

> Hi all,
>
>
>
> Ignite, when persistence mode is enabled, stores data and indexes on disk.
> To minimize the latency of disks, several tuning options can be applied.
> Setting the page size of a memory region to match the page size of the 
> underlying storage, using a separate disk for the WAL, and using 
> production-level SSDs are just a few of them [ 
> https://apacheignite.readme.io/docs/durable-memory-tuning#
> section-native-persistence-related-tuning ].
>
> A persistent memory store with low latency and high capacity offers a 
> viable alternative to disks. In light of this, we are proposing to 
> make use of our Low Level Persistent Library (LLPL), 
> https://github.com/pmem/pcj/ tree/master/LLPL, to offer a persistent memory storage for Ignite.
>
> At this point, we envision two distinct implementation options:
>
> 1.      Data and indexes will continue to be stored in the off-heap memory
> but the disk will be replaced by a persistent memory. Since 
> persistence memory in this option is not a file system, the logic 
> currently offered by WAL file and the partition files would have to be implemented from scratch.
>
> 2.      In this option, we eliminate the current check-point process and
> the WAL file. We will use a memory region defined by LLPL to store 
> data and indexes. There will be no off-heap memory. DRAM will be 
> exclusively used to store hot cache entries just like the on-heap 
> cache is in the current implementation.
>
> In both cases, there are more details and subtleties that have to 
> handled
> - e.g. the atomic and transactional guarantees offered. More 
> clarifications will be given as we go along. And, feel free to provide your thoughts.
>
> Thanks,
> Mulugeta
>
>

Re: Adding Persistent Memory Support for Ignite

[Dmitriy Setrakyan <ds...@apache.org>]

Hi Mulugeta,

Where can I find documentation about LLPL to understand how memory and
persistence are handled there?

D.

On Thu, Nov 16, 2017 at 7:16 AM, Mammo, Mulugeta <mu...@intel.com>
wrote:

> Hi all,
>
>
>
> Ignite, when persistence mode is enabled, stores data and indexes on disk.
> To minimize the latency of disks, several tuning options can be applied.
> Setting the page size of a memory region to match the page size of the
> underlying storage, using a separate disk for the WAL, and using
> production-level SSDs are just a few of them [
> https://apacheignite.readme.io/docs/durable-memory-tuning#
> section-native-persistence-related-tuning ].
>
> A persistent memory store with low latency and high capacity offers a
> viable alternative to disks. In light of this, we are proposing to make use
> of our Low Level Persistent Library (LLPL), https://github.com/pmem/pcj/
> tree/master/LLPL, to offer a persistent memory storage for Ignite.
>
> At this point, we envision two distinct implementation options:
>
> 1.      Data and indexes will continue to be stored in the off-heap memory
> but the disk will be replaced by a persistent memory. Since persistence
> memory in this option is not a file system, the logic currently offered by
> WAL file and the partition files would have to be implemented from scratch.
>
> 2.      In this option, we eliminate the current check-point process and
> the WAL file. We will use a memory region defined by LLPL to store data and
> indexes. There will be no off-heap memory. DRAM will be exclusively used to
> store hot cache entries just like the on-heap cache is in the current
> implementation.
>
> In both cases, there are more details and subtleties that have to handled
> - e.g. the atomic and transactional guarantees offered. More clarifications
> will be given as we go along. And, feel free to provide your thoughts.
>
> Thanks,
> Mulugeta
>
>