You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@ignite.apache.org by sk...@apache.org on 2022/10/19 16:48:11 UTC
[ignite-3] branch ignite-3.0.0-beta1 updated: IGNITE-17829 Document new memory storage structure. Fixes #1185
This is an automated email from the ASF dual-hosted git repository.
sk0x50 pushed a commit to branch ignite-3.0.0-beta1
in repository https://gitbox.apache.org/repos/asf/ignite-3.git
The following commit(s) were added to refs/heads/ignite-3.0.0-beta1 by this push:
new f906f21f68 IGNITE-17829 Document new memory storage structure. Fixes #1185
f906f21f68 is described below
commit f906f21f68cc8d3acc3face563cf33f5578e7661
Author: IgGusev <de...@mail.ru>
AuthorDate: Wed Oct 19 19:47:13 2022 +0300
IGNITE-17829 Document new memory storage structure. Fixes #1185
Signed-off-by: Slava Koptilin <sl...@gmail.com>
---
docs/_data/toc.yaml | 12 +++++--
docs/_docs/persistence.adoc | 25 --------------
docs/_docs/storage/persistent.adoc | 71 ++++++++++++++++++++++++++++++++++++++
docs/_docs/storage/rocksdb.adoc | 52 ++++++++++++++++++++++++++++
docs/_docs/storage/volatile.adoc | 50 +++++++++++++++++++++++++++
5 files changed, 183 insertions(+), 27 deletions(-)
diff --git a/docs/_data/toc.yaml b/docs/_data/toc.yaml
index c59ec368c4..55899eab11 100644
--- a/docs/_data/toc.yaml
+++ b/docs/_data/toc.yaml
@@ -40,10 +40,18 @@
url: compute/compute
- title: Data Region Configuration
url: config/data-region
+- title: Storage Configuration
+ url: storage/persistent
+ items:
+ - title: Native Persistent Storage
+ url: storage/persistent
+ - title: RocksDB Persistent Storage
+ url: storage/rocksdb
+ - title: Volatile Storage
+ url: storage/volatile
- title: Binary Client Protocol
url: binary-protocol
-- title: Native Persistence
- url: persistence
+
- title: Data Rebalancing
url: rebalance
- title: Performing Transactions
diff --git a/docs/_docs/persistence.adoc b/docs/_docs/persistence.adoc
deleted file mode 100644
index 471a073905..0000000000
--- a/docs/_docs/persistence.adoc
+++ /dev/null
@@ -1,25 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one or more
-// contributor license agreements. See the NOTICE file distributed with
-// this work for additional information regarding copyright ownership.
-// The ASF licenses this file to You under the Apache License, Version 2.0
-// (the "License"); you may not use this file except in compliance with
-// the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-= Ignite Persistence
-
-Ignite Persistence, or Native Persistence, is a set of features designed to provide persistent storage.
-When it is enabled, Ignite always stores all the data on disk, and loads as much data as it can into RAM for processing.
-For example, if there are 100 entries and RAM has the capacity to store only 20, then all 100 are stored on disk and only 20 are cached in RAM for better performance.
-
-When Native persistence is turned off and no external storage is used, Ignite behaves as a pure in-memory store.
-
-NOTE: In this Apache Ignite Alpha release, persistence is enabled by default.
-
-The current Alpha release persistence functionality is based on link:https://github.com/facebook/rocksdb/wiki/RocksDB-Overview[RocksDB,window=_blank].
diff --git a/docs/_docs/storage/persistent.adoc b/docs/_docs/storage/persistent.adoc
new file mode 100644
index 0000000000..92fa2c30ef
--- /dev/null
+++ b/docs/_docs/storage/persistent.adoc
@@ -0,0 +1,71 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+= Persistent Storage
+
+
+== Overview
+
+Ignite Persistence is designed to provide a quick and responsive persistent storage.
+When the link:config/data-region[data region] is configured to use Ignite persistence, Ignite stores all the data on disk, and loads as much data as it can into RAM for processing.
+
+The Native Persistence functionality is based on the following features:
+
+* Storing data partitions on disk
+* Checkpointing
+
+When persistence is enabled, Apache Ignite stores each partition in a separate file on disk. In addition to data partitions, Apache Ignite stores indexes and metadata.
+
+//image::images/persistent_store_structure.png[]
+
+== Checkpointing
+
+_Checkpointing_ is the process of copying dirty pages from RAM to partition files on disk. A dirty page is a page that was updated in RAM but was not written to the respective partition file.
+
+After a checkpoint is created, all changes are persisted to disk and will be available if the node crashes and is restarted.
+
+Checkpointing is designed to ensure durability of data and recovery in case of a node failure.
+
+// image:images/checkpointing-persistence.png[]
+
+This process helps to utilize disk space frugally by keeping pages in the most up-to-date state on disk.
+
+
+== Configuration Properties
+
+The following table describes some properties of javadoc:org.apache.ignite.configuration.DataStorageConfiguration[].
+
+[cols="1,1,3",opts="header", stripes=none]
+|===
+|name|| The name of the data region.
+|initSize|`256 * 1024 * 1024`| Sets the initial space allocated to the data region.
+|maxSize|`256 * 1024 * 1024`| Sets the maximum space that can be allocated to the data region.
+|replacementMode|`CLOCK`| Sets the page replacement algorithm.
+|===
+
+
+== Configuration Example
+
+The example below shows how to configure one data region that uses Ignite persistence:
+
+----
+ignite config set --type cluster \
+"{
+ aipersist.regions: [{
+ name: btree_persistent_region,
+ maxSize: 256000000
+ }]
+}"
+----
\ No newline at end of file
diff --git a/docs/_docs/storage/rocksdb.adoc b/docs/_docs/storage/rocksdb.adoc
new file mode 100644
index 0000000000..a410180816
--- /dev/null
+++ b/docs/_docs/storage/rocksdb.adoc
@@ -0,0 +1,52 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+= RocksDB Storage
+
+RocksDB is a persistent storage engine based on LSM tree. It is best used in environments with a large number of write requests.
+
+When the link:config/data-region[data region] is configured to use it, Ignite stores all the data on disk, and loads as much data as it can into RAM for processing.
+
+
+== Configuration Properties
+
+
+Below is the list of properties for link:https://rocksdb.org/[RocksDB,window=_blank] data regions:
+
+[cols="1,1,3",opts="header", stripes=none]
+|===
+|Property|Default|Description
+|name|| The name of the data region.
+|size| `256 * 1024 * 1024` | Size of the offheap cache.
+|writeBufferSize | `64 * 1024 * 1024` | Size of the write buffer.
+|cache| `lru` | The type of the cache to use. Currently only lru is supported. Using `clock` cache is not recommended.
+|numShardBits| `-1` | The number of parts the cache is sharded to.
+|===
+
+
+== Configuration Example
+
+The example below shows how to configure a data region with RocksDB storage:
+
+----
+ignite config set --type cluster \
+"{
+ rocksDb.regions: [{
+ name: lsm_region,
+ size: 256000000,
+ cache: lru
+ }]
+}"
+----
\ No newline at end of file
diff --git a/docs/_docs/storage/volatile.adoc b/docs/_docs/storage/volatile.adoc
new file mode 100644
index 0000000000..c13d7c1ac0
--- /dev/null
+++ b/docs/_docs/storage/volatile.adoc
@@ -0,0 +1,50 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+= Volatile Storage
+
+== Overview
+
+Apache Ignite Volatile storage is designed to provide a quick and responsive storage without guarantees of data persistence.
+
+
+When it is enabled for the link:config/data-region[data region], Ignite stores all data in the data region in RAM. Data will be lost on cluster shutdown, so make sure to have a separate data region for persistent storage.
+
+== Configuration Parameters
+
+[cols="1,1,3",opts="header", stripes=none]
+|===
+|name|| The name of the data region.
+|initSize|`256 * 1024 * 1024`| Sets the initial space allocated to the data region.
+|maxSize|`256 * 1024 * 1024`| Sets the maximum space that can be allocated to the data region.
+|evictionMode|`DISABLED`| Sets the eviction algorithm to use. Possible values: `DISABLED`, `RANDOM_LRU`, `RANDOM_2_LRU`.
+|evictionThreshold|`0.9`| Configures when the eviction process starts.
+|emptyPagesPoolSize|100| The maximum number of empty pages Ignite will try to keep.
+|===
+
+
+== Configuration Example
+
+The example below shows how to configure one data region that uses volatile storage:
+
+----
+ignite config set --type cluster \
+"{
+ aimem.regions: [{
+ name: btree_volatile_region,
+ maxSize: 256000000
+ }]
+}"
+----
\ No newline at end of file