You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iotdb.apache.org by lt...@apache.org on 2019/09/03 12:44:20 UTC
[incubator-iotdb] 02/02: fix merge unit test
This is an automated email from the ASF dual-hosted git repository.
lta pushed a commit to branch reimpl_sync
in repository https://gitbox.apache.org/repos/asf/incubator-iotdb.git
commit b3615329fe5b827ddefd289ccb9d2b2f2ced7c0c
Author: lta <li...@163.com>
AuthorDate: Tue Sep 3 20:44:02 2019 +0800
fix merge unit test
---
.../UserGuide/9-System Tools/1-Sync.md | 55 ++++------------------
.../UserGuide/9-System Tools/1-Sync.md | 42 +++++------------
.../resources/conf/iotdb-engine.properties | 8 ----
.../java/org/apache/iotdb/db/conf/IoTDBConfig.java | 14 ------
.../org/apache/iotdb/db/conf/IoTDBDescriptor.java | 3 --
.../apache/iotdb/db/engine/merge/MergeTest.java | 16 +++++--
6 files changed, 32 insertions(+), 106 deletions(-)
diff --git a/docs/Documentation-CHN/UserGuide/9-System Tools/1-Sync.md b/docs/Documentation-CHN/UserGuide/9-System Tools/1-Sync.md
index cddf0c1..9c6e699 100644
--- a/docs/Documentation-CHN/UserGuide/9-System Tools/1-Sync.md
+++ b/docs/Documentation-CHN/UserGuide/9-System Tools/1-Sync.md
@@ -22,7 +22,7 @@
# 第9章: 同步工具
<!-- TOC -->
-- [同步工具](#同步工具)
+- [第9章: 同步工具](#第9章-同步工具)
- [介绍](#介绍)
- [配置参数](#配置参数)
- [同步工具接收端](#同步工具接收端)
@@ -35,7 +35,7 @@
<!-- /TOC -->
# 介绍
-同步工具是定期将本地磁盘中和新增的已持久化的tsfile文件上传至云端并加载到IoTDB套件工具。
+同步工具是定期将本地磁盘中和新增的已持久化的tsfile文件上传至云端并加载到IoTDB的套件工具。
在同步工具的发送端，同步模块是一个独立的进程，独立于本地的IoTDB。通过独立的脚本进行启动和关闭(详见章节`使用方式`)，同步的频率周期可由用户设置。
@@ -62,7 +62,7 @@
</tr>
<tr>
<td>默认值</td>
- <td>false</td>
+ <td>true</td>
</tr>
<tr>
<td>改后生效方式</td>
@@ -94,29 +94,6 @@
<table>
<tr>
- <td colspan="2">参数名: update_historical_data_possibility</td>
- </tr>
- <tr>
- <td width="20%">描述</td>
- <td>同步服务端在合并同步的数据时选择的处理策略。如果同步的数据对历史数据(相比本地该存储组数据的最新时间戳)更新占比超过50%，则建议选择策略1，将参数设置为true，使用该策略对IoTDB系统的写入性能产生较大影响，对机器的CPU占用较小；如果同步的数据对历史数据更新占比少于50%，则建议选择策略2，将参数设置为false，使用该策略对IoTDB系统的写入性能产生较小影响，对机器CPU的占用较大。<br/>
-</td>
- </tr>
- <tr>
- <td>类型</td>
- <td>Boolean</td>
- </tr>
- <tr>
- <td>默认值</td>
- <td>false</td>
- </tr>
- <tr>
- <td>改后生效方式</td>
- <td>重启服务器生效</td>
- </tr>
-</table>
-
-<table>
- <tr>
<td colspan="2">参数名: sync_server_port</td>
</tr>
<tr>
@@ -207,33 +184,19 @@
<table>
<tr>
- <td colspan="2">参数名: iotdb_schema_directory</td>
+ <td colspan="2">参数名: sync_storage_groups</td>
</tr>
<tr>
<td width="20%">描述</td>
- <td>同步发送端的IoTDB schema文件的绝对路径，例如$IOTDB_HOME /data/system/schema/mlog.txt(若用户未手动设置schema元数据的路径，则该路径为默认路径)，该参数默认不生效，用户有需求时进行手动设置</td>
+ <td>进行同步的存储组列表，存储组间用逗号分隔；若列表设置为空表示同步所有存储组，默认为空</td>
</tr>
<tr>
<td>类型</td>
<td>String</td>
</tr>
<tr>
- <td>改后生效方式</td>
- <td>重启同步功能发送端生效</td>
- </tr>
-</table>
-
-<table>
- <tr>
- <td colspan="2">参数名: iotdb_bufferWrite_directory</td>
- </tr>
- <tr>
- <td width="20%">描述</td>
- <td>同步发送端的IoTDB 的bufferWrite数据(tsfile文件)目录的绝对路径，定位至bufferWrite目录下，例如： $IOTDB_HOME /data/data/settled(若用户未手动设置数据路径，则该路径为默认路径)，该参数默认不生效，用户有需求时进行手动设置。该参数需要保证和参数iotdb_schema_directory属于同一个IoTDB</td>
- </tr>
- <tr>
- <td>类型</td>
- <td>String</td>
+ <td>示例</td>
+ <td>root.sg1, root.sg2</td>
</tr>
<tr>
<td>改后生效方式</td>
@@ -244,7 +207,7 @@
# 使用方式
## 启动同步功能接收端
1. 配置接收端的参数，例如：
-<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/26211279/59494502-daaa4380-8ebf-11e9-8bce-363e2433005a.png">
+<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/26211279/64172919-a32cb100-ce88-11e9-821c-33369bff6d34.png">
2. 启动IoTDB引擎，同步功能接收端会同时启动，启动时LOG日志会出现`IoTDB: start SYNC ServerService successfully`字样，表示同步接收端启动成功，如图所示：
<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/26211279/59494513-df6ef780-8ebf-11e9-83e1-ee8ae64b76d0.png">
@@ -253,7 +216,7 @@
## 启动同步功能发送端
1. 配置发送端的参数， 如图所示:
-<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/26211279/59494559-f9a8d580-8ebf-11e9-875e-355199c1a1e9.png">
+<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/26211279/64172668-15e95c80-ce88-11e9-9700-dff7daf06bb7.png">
2. 启动同步功能发送端
用户可以使用```$IOTDB_HOME/bin```文件夹下的脚本启动同步功能的发送端
diff --git a/docs/Documentation/UserGuide/9-System Tools/1-Sync.md b/docs/Documentation/UserGuide/9-System Tools/1-Sync.md
index c41c34d..bc7ea88 100644
--- a/docs/Documentation/UserGuide/9-System Tools/1-Sync.md
+++ b/docs/Documentation/UserGuide/9-System Tools/1-Sync.md
@@ -25,7 +25,7 @@
<!-- TOC -->
-- [Chapter 7: System Tools](#chapter-7-system-tools)
+- [Chapter 9: System Tools](#chapter-9-system-tools)
- [Data Import](#data-import)
- [Introduction](#introduction)
- [Configuration](#configuration)
@@ -98,31 +98,6 @@ The parameter configuration of the sync receiver is located in the configuration
<table>
<tr>
- <td colspan="2">parameter: update_historical_data_possibility</td>
- </tr>
- <tr>
- <td width="30%">Description</td>
- <td>The processing strategy chosen by the sync receiver when merging the sync data.<br/>
- 1. If the sync data accounts for more than 50% of the update of the historical data (compared with the latest timestamp of the local storage group data),then it is recommended this parameter be set to TRUE, which has a greater impact on the write performance and reduce CPU usage.<br/>
- 2. If the sync data accounts for less than 50% of the update of the historical data (compared with the latest timestamp of the local storage group data),then it is recommended this parameter be set to FALSE,which has little impact on the write performance and takes up a large amount of CPU power.<br/>
-</td>
- </tr>
- <tr>
- <td>Type</td>
- <td>Boolean</td>
- </tr>
- <tr>
- <td>Default</td>
- <td>false</td>
- </tr>
- <tr>
- <td>Modalities for Entry into Force after Modification</td>
- <td>Restart receiver</td>
- </tr>
-</table>
-
-<table>
- <tr>
<td colspan="2">parameter: sync_server_port</td>
</tr>
<tr>
@@ -231,26 +206,31 @@ The parameters of the sync sender are configured in a separate configuration fil
<table>
<tr>
- <td colspan="2">parameter: iotdb_bufferWrite_directory</td>
+ <td colspan="2">parameter: sync_storage_groups</td>
</tr>
<tr>
- <td width="30%">Description</td>
- <td>The absolute path of the buffer write data (tsfile file) directory of the IoTDB at the sender, such as: $IOTDB_HOME/data/data/settled (if the user does not set the data path manually, the path is the default path of IoTDB engine). This parameter is not valid by default, and is set manually when the user needs it. This parameter needs to be guaranteed to belong to the same IoTDB as the parameter iotdb_schema_directory.</td>
+ <td width="20%">Description</td>
+ <td>This parameter represents storage groups that participate in the synchronization task, which distinguishes each storage group by comma.If the list is empty, it means that all storage groups participate in synchronization.By default, it is empty list.</td>
</tr>
<tr>
<td>Type</td>
<td>String</td>
</tr>
<tr>
+ <td>Example</td>
+ <td>root.sg1, root.sg2</td>
+ </tr>
+ <tr>
<td>Modalities for Entry into Force after Modification</td>
<td>Restart client</td>
</tr>
</table>
+
# Usage
## Start Sync Receiver
1. Set up parameters of sync receiver. For example:
-<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/26211279/59494502-daaa4380-8ebf-11e9-8bce-363e2433005a.png">
+<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/26211279/64172919-a32cb100-ce88-11e9-821c-33369bff6d34.png">
2. Start IoTDB engine, and the sync receiver will start at the same time, and the LOG log will start with the sentence `IoTDB: start SYNC ServerService successfully` indicating the successful start of the return receiver.
<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/26211279/59494513-df6ef780-8ebf-11e9-83e1-ee8ae64b76d0.png">
@@ -259,7 +239,7 @@ Stop IoTDB and the sync receiver will be closed at the same time.
## Start Sync Sender
1. Set up parameters of sync sender. For example:
-<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/26211279/59494559-f9a8d580-8ebf-11e9-875e-355199c1a1e9.png">
+<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/26211279/64172668-15e95c80-ce88-11e9-9700-dff7daf06bb7.png">
2. Start sync sender
Users can use the scripts under the ```$IOTDB_HOME/bin``` folder to start the sync sender.
For Linux and Mac OS X users:
diff --git a/server/src/assembly/resources/conf/iotdb-engine.properties b/server/src/assembly/resources/conf/iotdb-engine.properties
index 3d9bfa6..bed1aa0 100644
--- a/server/src/assembly/resources/conf/iotdb-engine.properties
+++ b/server/src/assembly/resources/conf/iotdb-engine.properties
@@ -254,14 +254,6 @@ sync_server_port=5555
# The default is to allow all IP to sync
ip_white_list=0.0.0.0/0
-# The processing strategy chosen by the sync server when merging the sync data.
-# 1. If the sync data accounts for more than 50% of the update of the historical data (compared with the latest timestamp of the local storage group data),then it is recommended to select strategy 1.
-# Setting the parameter to true, which has a greater impact on the insert performance of the IoTDB system and occupies less CPU of the machine.
-# 2. If the sync data accounts for less than 50% of the update of the historical data (compared with the latest timestamp of the local storage group data),then it is recommended to select strategy 2.
-# Setting the parameter to false, which has little impact on the insert performance of IoTDB system and takes up a large amount of CPU power.
-update_historical_data_possibility=false
-
-
####################
### performance statistic configuration
####################
diff --git a/server/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java b/server/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java
index adf3f12..1658cee 100644
--- a/server/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java
+++ b/server/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java
@@ -211,12 +211,6 @@ public class IoTDBConfig {
* Set the language version when loading file including error information, default value is "EN"
*/
private String languageVersion = "EN";
- /**
- * Choose a postBack strategy of merging historical data: 1. It's more likely to update historical
- * data, choose "true". 2. It's more likely not to update historical data or you don't know
- * exactly, choose "false".
- */
- private boolean updateHistoricalDataPossibility = false;
private String ipWhiteList = "0.0.0.0/0";
/**
@@ -569,14 +563,6 @@ public class IoTDBConfig {
this.languageVersion = languageVersion;
}
- public boolean isUpdateHistoricalDataPossibility() {
- return updateHistoricalDataPossibility;
- }
-
- void setUpdateHistoricalDataPossibility(boolean updateHistoricalDataPossibility) {
- this.updateHistoricalDataPossibility = updateHistoricalDataPossibility;
- }
-
public String getBaseDir() {
return baseDir;
}
diff --git a/server/src/main/java/org/apache/iotdb/db/conf/IoTDBDescriptor.java b/server/src/main/java/org/apache/iotdb/db/conf/IoTDBDescriptor.java
index c232aea..efb2d03 100644
--- a/server/src/main/java/org/apache/iotdb/db/conf/IoTDBDescriptor.java
+++ b/server/src/main/java/org/apache/iotdb/db/conf/IoTDBDescriptor.java
@@ -195,9 +195,6 @@ public class IoTDBDescriptor {
.parseInt(properties.getProperty("sync_server_port",
Integer.toString(conf.getSyncServerPort())).trim()));
- conf.setUpdateHistoricalDataPossibility(Boolean.parseBoolean(
- properties.getProperty("update_historical_data_possibility",
- Boolean.toString(conf.isSyncEnable()))));
conf.setIpWhiteList(properties.getProperty("ip_white_list", conf.getIpWhiteList()));
conf.setConcurrentFlushThread(Integer
diff --git a/server/src/test/java/org/apache/iotdb/db/engine/merge/MergeTest.java b/server/src/test/java/org/apache/iotdb/db/engine/merge/MergeTest.java
index cab3a59..25892ab 100644
--- a/server/src/test/java/org/apache/iotdb/db/engine/merge/MergeTest.java
+++ b/server/src/test/java/org/apache/iotdb/db/engine/merge/MergeTest.java
@@ -26,6 +26,7 @@ import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
+import org.apache.iotdb.db.conf.IoTDBConstant;
import org.apache.iotdb.db.conf.IoTDBDescriptor;
import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
import org.apache.iotdb.db.engine.cache.TsFileMetaDataCache;
@@ -112,20 +113,27 @@ abstract class MergeTest {
}
}
- private void prepareFiles(int seqFileNum, int unseqFileNum) throws IOException, WriteProcessException {
+ private void prepareFiles(int seqFileNum, int unseqFileNum)
+ throws IOException, WriteProcessException {
for (int i = 0; i < seqFileNum; i++) {
- File file = new File(i + "seq.tsfile");
+ File file = new File(
+ i + "seq" + IoTDBConstant.FILE_NAME_SEPARATOR + i + IoTDBConstant.FILE_NAME_SEPARATOR + i
+ + ".tsfile");
TsFileResource tsFileResource = new TsFileResource(file);
seqResources.add(tsFileResource);
prepareFile(tsFileResource, i * ptNum, ptNum, 0);
}
for (int i = 0; i < unseqFileNum; i++) {
- File file = new File(i + "unseq.tsfile");
+ File file = new File(
+ i + "unseq" + IoTDBConstant.FILE_NAME_SEPARATOR + i + IoTDBConstant.FILE_NAME_SEPARATOR
+ + i
+ + ".tsfile");
TsFileResource tsFileResource = new TsFileResource(file);
unseqResources.add(tsFileResource);
prepareFile(tsFileResource, i * ptNum, ptNum * (i + 1) / unseqFileNum, 10000);
}
- File file = new File(unseqFileNum + "unseq.tsfile");
+ File file = new File(unseqFileNum + "unseq" + IoTDBConstant.FILE_NAME_SEPARATOR + unseqFileNum
+ + IoTDBConstant.FILE_NAME_SEPARATOR + unseqFileNum + ".tsfile");
TsFileResource tsFileResource = new TsFileResource(file);
unseqResources.add(tsFileResource);
prepareFile(tsFileResource, 0, ptNum * unseqFileNum, 20000);