You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iotdb.apache.org by xu...@apache.org on 2021/12/31 08:17:13 UTC
[iotdb] branch xkf_virtual_storage_group_doc created (now 40e1b82)
This is an automated email from the ASF dual-hosted git repository.
xuekaifeng pushed a change to branch xkf_virtual_storage_group_doc
in repository https://gitbox.apache.org/repos/asf/iotdb.git.
at 40e1b82 update doc
This branch includes the following new commits:
new 40e1b82 update doc
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[iotdb] 01/01: update doc
Posted by xu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xuekaifeng pushed a commit to branch xkf_virtual_storage_group_doc
in repository https://gitbox.apache.org/repos/asf/iotdb.git
commit 40e1b8296bde08c5214eec5d30cd69dd0204a58d
Author: 151250176 <15...@smail.nju.edu.cn>
AuthorDate: Fri Dec 31 16:12:51 2021 +0800
update doc
---
.../StorageEngine/VirtualStorageGroup.md | 65 ++++++++++++++++++++++
docs/UserGuide/Appendix/Config-Manual.md | 9 +++
.../StorageEngine/VirtualStorageGroup.md | 61 ++++++++++++++++++++
docs/zh/UserGuide/Appendix/Config-Manual.md | 9 +++
4 files changed, 144 insertions(+)
diff --git a/docs/SystemDesign/StorageEngine/VirtualStorageGroup.md b/docs/SystemDesign/StorageEngine/VirtualStorageGroup.md
new file mode 100644
index 0000000..2b6dfc3
--- /dev/null
+++ b/docs/SystemDesign/StorageEngine/VirtualStorageGroup.md
@@ -0,0 +1,65 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+
+# Background
+
+The storage group is specified by the user display.
+Use the statement "SET STORAGE GROUP TO" to specify the storage group.
+Each storage group has a corresponding StorageGroupProcessor.
+
+To ensure eventually consistency, a insert lock (exclusive lock) is used to synchronize each insert request in each storage group.
+So the server side parallelism of data ingestion is equal to the number of storage group.
+
+# Problem
+
+From background, we can infer that the parallelism of data ingestion of IoTDB is max(num of client, server side parallelism), which equals to max(num of client, num of storage group)
+
+The concept of storage group usually is related to real world entity such as factory, location, country and so on.
+The number of storage groups may be small which makes the parallelism of data ingestion of IoTDB insufficient. We can't jump out of this dilemma even we start hundreds of client for ingestion.
+
+
+# Analyze
+
+In IoTDB, every kinds of ingestion interface can only insert timeseries of on device.
+One idea by intuition is changing the granularity of synchronization from storage group level to device level.
+However, one lock for one device may occupy lots of resource that beyond our expectation. One lock means about 100 bytes of memory and one kernel object.
+Sometimes the number of devices may reach one million and we can't afford such lock resource.
+
+As the analysis progresses, we think we should trade off between granularity of synchronization and resource occupation.
+
+# Solution
+
+Our idea is to group devices into buckets and change the granularity of synchronization from storage group level to device buckets level.
+
+In detail, we use hash to group different devices into buckets called virtual storage group.
+For example, one device called "root.sg.d"(assume it's storage group is "root.sg") is belonged to virtual storage group "root.sg.[hash("root.sg.d") mod num_of_virtual_storage_group]"
+
+# Usage
+
+To use virtual storage group, you can set this config below:
+
+```
+virtual_storage_group_num
+```
+
+Recommended value is [virtual storage group number] = [CPU core number] / [user-defined storage group number]
+
+For more information, you can refer to [this page](../../UserGuide/Appendix/Config-Manual.md).
\ No newline at end of file
diff --git a/docs/UserGuide/Appendix/Config-Manual.md b/docs/UserGuide/Appendix/Config-Manual.md
index e0f119d..b3aa37f 100644
--- a/docs/UserGuide/Appendix/Config-Manual.md
+++ b/docs/UserGuide/Appendix/Config-Manual.md
@@ -739,6 +739,15 @@ The permission definitions are in ${IOTDB\_CONF}/conf/jmx.access.
|Default| 604800 |
|Effective|Only allowed to be modified in first start up|
+* virtual\_storage\_group\_num
+
+|Name| virtual\_storage\_group\_num |
+|:---:|:---|
+|Description| number of virtual storage groups per user-defined storage group, a virtual storage group is the unit of parallelism in memory as all ingestions in one virtual storage group are serialized, recommended value is [virtual storage group number] = [CPU core number] / [user-defined storage group number]|
+|Type| LONG |
+|Default| 1 |
+|Effective|Only allowed to be modified in first start up|
+
* enable\_id\_table
|Name| enable\_id\_table |
diff --git a/docs/zh/SystemDesign/StorageEngine/VirtualStorageGroup.md b/docs/zh/SystemDesign/StorageEngine/VirtualStorageGroup.md
new file mode 100644
index 0000000..c7435f0
--- /dev/null
+++ b/docs/zh/SystemDesign/StorageEngine/VirtualStorageGroup.md
@@ -0,0 +1,61 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+
+# 背景
+
+存储组由用户显示指定，使用语句"SET STORAGE GROUP TO"来指定存储组，每一个存储组有一个对应的 StorageGroupProcessor
+
+为了确保最终一致性，每一个存储组有一个数据插入锁（排它锁）来同步每一次插入操作。
+所以服务端数据写入的并行度为存储组的数量。
+
+# 问题
+
+从背景中可知，IoTDB数据写入的并行度为 max(客户端数量，服务端数据写入的并行度)，也就是max(客户端数量，存储组数量)
+
+在生产实践中，存储组的概念往往与特定真实世界实体相关（例如工厂，地点，国家等）。
+因此存储组的数量可能会比较小，这会导致IoTDB写入并行度不足。即使我们开再多的客户端写入线程，也无法走出这种困境。
+
+# 分析
+
+在IoTDB中，每一种写入接口每次仅写入一个设备的时间序列的数据。
+一个简单的想法是，将同步的粒度从存储组级别改为设备级别。然而每一个设备一个锁可能会占用超出我们预期的资源，每个锁占用大概100字节的内存和一个内核对象。
+在设备数量达到百万级的场景下，锁带来的资源开销是我们不能接受的。
+
+随着分析的深入，我们发现我们需要在同步粒度和资源占用之间做权衡。
+
+# 解决方案
+
+我们的方案是将一个存储组下的设备分为若干个设备组（称为虚拟存储组），将同步粒度从存储组级别改为虚拟存储组粒度。
+
+更具体的，我们使用哈希将设备分到不同的虚拟存储组下，例如：
+对于一个名为"root.sg.d"的设备（假设其存储组为"root.sg"），它属于的虚拟存储组为"root.sg.[hash("root.sg.d") mod num_of_virtual_storage_group]"
+
+# 使用方法
+
+通过改变如下配置来设置每一个存储组下虚拟存储组的数量：
+
+```
+virtual_storage_group_num
+```
+
+推荐值为[virtual storage group number] = [CPU core number] / [user-defined storage group number]
+
+参考[配置手册](../../UserGuide/Appendix/Config-Manual.md)以获取更多信息。
\ No newline at end of file
diff --git a/docs/zh/UserGuide/Appendix/Config-Manual.md b/docs/zh/UserGuide/Appendix/Config-Manual.md
index 78786f9..d0a0273 100644
--- a/docs/zh/UserGuide/Appendix/Config-Manual.md
+++ b/docs/zh/UserGuide/Appendix/Config-Manual.md
@@ -1485,6 +1485,15 @@
|默认值| 604800 |
|改后生效方式|仅允许在第一次启动服务前修改|
+* virtual\_storage\_group\_num
+
+|名字| virtual\_storage\_group\_num |
+|:---:|:---|
+|描述| 每一个用户定义存储组下虚拟存储组的数量, 虚拟存储组是内存中写入的并行单位，每一个虚拟存储组内的写入请求是串行的，推荐值为： [virtual storage group number] = [CPU core number] / [user-defined storage group number]|
+|类型| INT32 |
+|默认值| 1 |
+|改后生效方式|仅允许在第一次启动服务前修改|
+
* enable\_id\_table
|名字| enable\_id\_table |