You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@ignite.apache.org by al...@apache.org on 2021/07/02 13:31:46 UTC
[ignite] branch ignite-2.11 updated: IGNITE-15048 Documentation:
Page replacement algorithms - Fixes #9216.
This is an automated email from the ASF dual-hosted git repository.
alexpl pushed a commit to branch ignite-2.11
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/ignite-2.11 by this push:
new d74fea2 IGNITE-15048 Documentation: Page replacement algorithms - Fixes #9216.
d74fea2 is described below
commit d74fea2e811426bb2249b52c294a3360bf73a6ef
Author: Aleksey Plekhanov <pl...@gmail.com>
AuthorDate: Fri Jul 2 16:20:00 2021 +0300
IGNITE-15048 Documentation: Page replacement algorithms - Fixes #9216.
Signed-off-by: Aleksey Plekhanov <pl...@gmail.com>
(cherry picked from commit 468c41f8410289ec14a6326ae7af05c1d5aa7055)
---
docs/_data/toc.yaml | 2 +
.../ignite/snippets/ReplacementPolicies.java | 65 +++++++++++++++
.../memory-configuration/eviction-policies.adoc | 2 +-
.../memory-configuration/replacement-policies.adoc | 96 ++++++++++++++++++++++
4 files changed, 164 insertions(+), 1 deletion(-)
diff --git a/docs/_data/toc.yaml b/docs/_data/toc.yaml
index 7ede037..0edcf6e 100644
--- a/docs/_data/toc.yaml
+++ b/docs/_data/toc.yaml
@@ -129,6 +129,8 @@
url: memory-configuration/data-regions
- title: Eviction Policies
url: memory-configuration/eviction-policies
+ - title: Replacement Policies
+ url: memory-configuration/replacement-policies
- title: Configuring Persistence
items:
- title: Ignite Persistence
diff --git a/docs/_docs/code-snippets/java/src/main/java/org/apache/ignite/snippets/ReplacementPolicies.java b/docs/_docs/code-snippets/java/src/main/java/org/apache/ignite/snippets/ReplacementPolicies.java
new file mode 100644
index 0000000..b8cae76
--- /dev/null
+++ b/docs/_docs/code-snippets/java/src/main/java/org/apache/ignite/snippets/ReplacementPolicies.java
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ignite.snippets;
+
+import org.apache.ignite.Ignite;
+import org.apache.ignite.Ignition;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.configuration.PageReplacementMode;
+
+public class ReplacementPolicies {
+ public static void main(String[] args) {
+ segmentedLRU();
+ }
+
+ public static void segmentedLRU() {
+ //tag::segmentedLRU[]
+ // Node configuration.
+ IgniteConfiguration cfg = new IgniteConfiguration();
+
+ // Memory configuration.
+ DataStorageConfiguration storageCfg = new DataStorageConfiguration();
+
+ // Creating a new data region.
+ DataRegionConfiguration regionCfg = new DataRegionConfiguration();
+
+ // Region name.
+ regionCfg.setName("persistent_data_region");
+
+ // Enabling persistence.
+ regionCfg.setPersistenceEnabled(true);
+
+ // 20 GB max size (RAM).
+ regionCfg.setMaxSize(20L * 1024 * 1024 * 1024);
+
+ // Enabling SEGMENTED_LRU page replacement for this region.
+ regionCfg.setPageReplacementMode(PageReplacementMode.SEGMENTED_LRU);
+
+ // Setting the data region configuration.
+ storageCfg.setDataRegionConfigurations(regionCfg);
+
+ // Applying the new configuration.
+ cfg.setDataStorageConfiguration(storageCfg);
+ //end::segmentedLRU[]
+
+ try (Ignite ignite = Ignition.start(new IgniteConfiguration().setDataStorageConfiguration(storageCfg))) {
+
+ }
+ }
+}
diff --git a/docs/_docs/memory-configuration/eviction-policies.adoc b/docs/_docs/memory-configuration/eviction-policies.adoc
index 38921ef..9aee23c 100644
--- a/docs/_docs/memory-configuration/eviction-policies.adoc
+++ b/docs/_docs/memory-configuration/eviction-policies.adoc
@@ -27,7 +27,7 @@ Eviction is used in following cases:
When Native Persistence is on, a similar process — called _page replacement_ — is used to free up off-heap memory when Ignite cannot allocate a new page.
The difference is that the data is not lost (because it is stored in the persistent storage), and therefore you are less concerned about losing data than about efficiency.
-Page replacement is automatically handled by Ignite and is not user-configurable.
+Refer to the link:memory-configuration/replacement-policies[Replacement Policies] page for information about page replacement configuration.
== Off-Heap Memory Eviction
diff --git a/docs/_docs/memory-configuration/replacement-policies.adoc b/docs/_docs/memory-configuration/replacement-policies.adoc
new file mode 100644
index 0000000..a41d91f
--- /dev/null
+++ b/docs/_docs/memory-configuration/replacement-policies.adoc
@@ -0,0 +1,96 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Replacement Policies
+
+When link:persistence/native-persistence[Native Persistence] is on and the amount of data, which Ignite stores on the disk, is bigger than the off-heap memory amount allocated for the data region, another page should be evicted from the off-heap to the disk to preload a page from the disk to the completely full off-heap memory. This process is called _page rotation_ or _page replacement_.
+
+When Native Persistence is off, _eviction_ is used instead of _page replacement_. See the link:memory-configuration/eviction-policies[Eviction Policies] page for more information.
+
+Page replacement is implemented as follows:
+
+When Ignite requires a page, it tries to find this page in the off-heap memory. If the page is not currently in the off-heap memory (a page fault occurs), this page is preloaded from the disk. At the same time, when off-heap memory is already full, another page should be chosen to be replaced (to stored to the disk and evicted).
+
+Ignite supports three algorithms to find pages to replace:
+
+* Random-LRU algorithm;
+* Segmented-LRU algorithm;
+* CLOCK algorithm.
+
+Page replacement algorithm can be configured by the `PageReplacementMode` property of `DataRegionConfiguration`. By default, CLOCK algorithm is used.
+
+[tabs]
+--
+tab:XML[]
+[source,xml]
+----
+<bean class="org.apache.ignite.configuration.IgniteConfiguration">
+ <!-- Memory configuration. -->
+ <property name="dataStorageConfiguration">
+ <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
+ <property name="dataRegionConfigurations">
+ <list>
+ <!--
+ Defining a persistent data region with Segmented LRU page replacement mode.
+ -->
+ <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
+ <!-- Data region name. -->
+ <property name="name" value="persistent_data_region"/>
+
+ <!-- Enable persistence. -->
+ <property name="persistenceEnabled" value="true"/>
+
+ <!-- 20 GB maximum size (RAM). -->
+ <property name="maxSize" value="#{20L * 1024 * 1024 * 1024}"/>
+
+ <!-- Enabling SEGMENTED_LRU page replacement for this region. -->
+ <property name="pageReplacementMode" value="SEGMENTED_LRU"/>
+ </bean>
+ </list>
+ </property>
+ </bean>
+ </property>
+
+ <!-- The rest of the configuration. -->
+</bean>
+----
+tab:Java[]
+[source,java]
+----
+include::{javaCodeDir}/ReplacementPolicies.java[tag=segmentedLRU,indent=0]
+----
+tab:C#/.NET[unsupported]
+----
+tab:C++[unsupported]
+--
+
+The choice of the algorithm depends on your workload. For most cases, CLOCK (default) is a good candidate, but on some workloads other algorithms can perform better.
+
+== Random-LRU Algorithm
+
+Every time a page is accessed, its timestamp is updated. When a page fault occurs and it's required to replace some pages, the algorithm randomly chooses 5 pages from the page memory and evicts a page with the latest timestamp.
+
+This algorithm has zero maintenance cost, but it is not very effective in terms of finding the next page to replace. We recommend that you use it in environments, where page replacement is not needed (when working with large enough data region to store all the amount of data) or happens very seldom.
+
+== Segmented-LRU Algorithm
+
+Segmented-LRU algorithm is a scan-resistant variation of the Least Recently Used (LRU) algorithm. Segmented-LRU pages list is divided into two segments: A probationary segment and a protected segment. Pages in each segment are ordered from the least to the most recently accessed. New pages are added to the most recently accessed end (tail) of the probationary segment. Existing pages are removed from wherever they currently reside and added to the most recently accessed end of the protect [...]
+
+This algorithm requires additional memory to store pages list that also needs to be updated on each page access. At the same time, the algorithm has a near-optimal page to replace selection policy. So, there can be a little performance drop for environments without page replacement (compared to random-LRU and CLOCK), but for environments with a high rate of page replacement and a large amount of one-time scans segmented-LRU can outperform random-LRU and CLOCK.
+
+== CLOCK Algorithm
+
+The CLOCK algorithm keeps a circular list of pages in memory, with the "hand" pointing to the last examined page frame in the list. When a page fault occurs and no empty frames exist, the hit flag of the page is inspected at the hand's location. If the hit flag is 0, the new page is put in the place of the page that the "hand" points to, and the hand is advanced one position further. Otherwise, the hit flag is cleared, then the clock hand is incremented and the process is repeated until [...]
+
+This algorithm has near to zero maintenance cost and replacement policy efficiency between random-LRU and segmented-LRU.