You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iotdb.apache.org by lt...@apache.org on 2019/11/14 11:48:07 UTC
[incubator-iotdb] branch master updated: [IOTDB-220]Add hot-load
configuration function (#537)
This is an automated email from the ASF dual-hosted git repository.
lta pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-iotdb.git
The following commit(s) were added to refs/heads/master by this push:
new 3e18e11 [IOTDB-220]Add hot-load configuration function (#537)
3e18e11 is described below
commit 3e18e112f97156ff918fa33f9308d07ab531eb58
Author: Tianan Li <li...@163.com>
AuthorDate: Thu Nov 14 19:47:57 2019 +0800
[IOTDB-220]Add hot-load configuration function (#537)
* add hot load config
* add load configuration in anltr
* update iotdb-env.bat
---
.../UserGuide/3-Server/4-Config Manual.md | 53 +++--
.../UserGuide/3-Server/4-Config Manual.md | 43 ++--
server/src/assembly/resources/conf/iotdb-env.bat | 19 +-
.../org/apache/iotdb/db/sql/parse/TqlLexer.g | 4 +
.../org/apache/iotdb/db/sql/parse/TqlParser.g | 34 +--
.../java/org/apache/iotdb/db/conf/IoTDBConfig.java | 79 +++----
.../org/apache/iotdb/db/conf/IoTDBConfigCheck.java | 4 +-
.../org/apache/iotdb/db/conf/IoTDBDescriptor.java | 247 +++++++++++++--------
.../db/conf/directories/DirectoryManager.java | 57 ++++-
.../directories/strategy/DirectoryStrategy.java | 11 +-
.../directories/strategy/SequenceStrategy.java | 6 +-
.../db/exception/LoadConfigurationException.java | 37 +++
.../org/apache/iotdb/db/qp/QueryProcessor.java | 1 +
.../apache/iotdb/db/qp/constant/SQLConstant.java | 3 +
.../iotdb/db/qp/executor/QueryProcessExecutor.java | 3 +
.../org/apache/iotdb/db/qp/logical/Operator.java | 2 +-
.../qp/logical/sys/LoadConfigurationOperator.java | 31 +++
.../db/qp/physical/sys/LoadConfigurationPlan.java | 43 ++++
.../iotdb/db/qp/strategy/LogicalGenerator.java | 8 +-
.../iotdb/db/qp/strategy/PhysicalGenerator.java | 8 +-
.../strategy/DirectoryStrategyTest.java | 12 +-
.../apache/iotdb/db/qp/plan/PhysicalPlanTest.java | 9 +
22 files changed, 490 insertions(+), 224 deletions(-)
diff --git a/docs/Documentation-CHN/UserGuide/3-Server/4-Config Manual.md b/docs/Documentation-CHN/UserGuide/3-Server/4-Config Manual.md
index 6914218..c68534f 100644
--- a/docs/Documentation-CHN/UserGuide/3-Server/4-Config Manual.md
+++ b/docs/Documentation-CHN/UserGuide/3-Server/4-Config Manual.md
@@ -21,19 +21,24 @@
# 第3章 服务器端
-## 系统配置
+## 配置手册
为方便IoTDB Server的配置与管理，IoTDB Server为用户提供三种配置项，使得用户可以在启动服务器或服务器运行时对其进行配置。
-三种配置项的配置文件均位于IoTDB安装目录：`$IOTDB_HOME/conf`文件夹下,其中涉及server配置的共有3个文件，分别为：`iotdb-env.sh`, `tsfile-format.properties`, `iotdb-engine.properties`。用户可以通过更改其中的配置项对系统运行的相关配置项进行配置。
+三种配置项的配置文件均位于IoTDB安装目录：`$IOTDB_HOME/conf`文件夹下,其中涉及server配置的共有2个文件，分别为：`iotdb-env.sh`, `iotdb-engine.properties`。用户可以通过更改其中的配置项对系统运行的相关配置项进行配置。
配置文件的说明如下：
* `iotdb-env.sh`：环境配置项的默认配置文件。用户可以在文件中配置JAVA-JVM的相关系统配置项。
-* `tsfile-format.properties`：IoTDB文件层系统配置项的默认配置文件。用户可以在文件中配置IoTDB存储时TsFile文件的相关信息，如每次将内存中的数据写入到磁盘时的数据大小(`group_size_in_byte`)，内存中每个列打一次包的大小(`page_size_in_byte`)等。
+* `iotdb-engine.properties`：IoTDB引擎层系统配置项的默认配置文件。用户可以在文件中配置IoTDB引擎运行时的相关参数，如JDBC服务监听端口(`rpc_port`)、overflow数据文件存储目录(`overflow_data_dir`)等。此外，用户可以在文件中配置IoTDB存储时TsFile文件的相关信息，如每次将内存中的数据写入到磁盘时的数据大小(`group_size_in_byte`)，内存中每个列打一次包的大小(`page_size_in_byte`)等。
-* `iotdb-engine.properties`：IoTDB引擎层系统配置项的默认配置文件。用户可以在文件中配置IoTDB引擎运行时的相关参数，如JDBC服务监听端口(`rpc_port`)、overflow数据文件存储目录(`overflow_data_dir`)等。
+### 热修改配置项
+
+为方便用户使用，IoTDB Server为用户提供了热修改功能，即在系统运行过程中修改`iotdb-engine.properties`中部分配置参数并即时应用到系统中。下面介绍的参数中，改后
+生效方式为`触发生效`的均为支持热修改的配置参数。
+
+触发方式：客户端发送```load configuration```命令至IoTDB Server，客户端的使用方式详见第4章
### 环境配置项
@@ -78,7 +83,7 @@
### 系统配置项
-系统配置项是IoTDB Server运行的核心配置，它主要用于设置IoTDB Server文件层和引擎层的参数，便于用户根据自身需求调整Server的相关配置，以达到较好的性能表现。系统配置项可分为两大模块：文件层配置项和引擎层配置项。用户可以通过查看`tsfile-format.properties`, `iotdb-engine.properties`,文件查看和修改两种配置项的内容。在0.7.0版本中字符串类型的配置项大小写敏感。
+系统配置项是IoTDB Server运行的核心配置，它主要用于设置IoTDB Server文件层和引擎层的参数，便于用户根据自身需求调整Server的相关配置，以达到较好的性能表现。系统配置项可分为两大模块：文件层配置项和引擎层配置项。用户可以通过查看`iotdb-engine.properties`,文件查看和修改两种配置项的内容。在0.7.0版本中字符串类型的配置项大小写敏感。
#### 文件层配置
@@ -89,7 +94,7 @@
|描述|数据压缩方法|
|类型|枚举String : “UNCOMPRESSED”, “SNAPPY”|
|默认值| UNCOMPRESSED |
-|改后生效方式|即时生效|
+|改后生效方式|触发生效|
* group\_size\_in\_byte
@@ -98,7 +103,7 @@
|描述|每次将内存中的数据写入到磁盘时的最大写入字节数|
|类型|Int32|
|默认值| 134217728 |
-|改后生效方式|即时生效|
+|改后生效方式|触发生效|
* max\_number\_of\_points\_in\_page
@@ -107,7 +112,7 @@
|描述|一个页中最多包含的数据点（时间戳-值的二元组）数量|
|类型|Int32|
|默认值| 1048576 |
-|改后生效方式|即时生效|
+|改后生效方式|触发生效|
* max\_string\_length
@@ -116,7 +121,7 @@
|描述|针对字符串类型的数据，单个字符串最大长度，单位为字符|
|类型|Int32|
|默认值| 128 |
-|改后生效方式|即时生效|
+|改后生效方式|触发生效|
* page\_size\_in\_byte
@@ -125,7 +130,7 @@
|描述|内存中每个列写出时，写成的单页最大的大小，单位为字节|
|类型|Int32|
|默认值| 65536 |
-|改后生效方式|即时生效|
+|改后生效方式|触发生效|
* time\_series\_data\_type
@@ -134,7 +139,7 @@
|描述|时间戳数据类型|
|类型|枚举String: "INT32", "INT64"|
|默认值| Int64 |
-|改后生效方式|即时生效|
+|改后生效方式|触发生效|
* time\_encoder
@@ -143,7 +148,7 @@
|描述| 时间列编码方式|
|类型|枚举String: “TS_2DIFF”,“PLAIN”,“RLE”|
|默认值| TS_2DIFF |
-|改后生效方式|即时生效|
+|改后生效方式|触发生效|
* float_precision
@@ -152,27 +157,27 @@
|描述| 浮点数精度，为小数点后数字的位数 |
|类型|Int32|
|默认值| 默认为2位。注意：32位浮点数的十进制精度为7位，64位浮点数的十进制精度为15位。如果设置超过机器精度将没有实际意义。|
-|改后生效方式|即时生效|
+|改后生效方式|触发生效|
#### 引擎层配置
-* back\_loop\_period
+* back\_loop\_period\_in\_second
-|名字| back\_loop\_period |
+|名字| back\_loop\_period\_in\_second |
|:---:|:---|
|描述| 系统统计量触发统计的频率，单位为秒。|
|类型|Int32|
|默认值| 5 |
|改后生效方式|重启服务器生效|
-* data\_dir
+* data\_dirs
-|名字| data\_dir |
+|名字| data\_dirs |
|:---:|:---|
|描述| IoTDB数据存储路径，默认存放在和bin目录同级的data目录下。相对路径的起始目录与操作系统相关，建议使用绝对路径。|
|类型|String|
|默认值| data |
-|改后生效方式|重启服务器生效|
+|改后生效方式|触发生效|
* enable_wal
@@ -181,7 +186,7 @@
|描述| 是否开启写前日志，默认值为true表示开启，配置成false表示关闭 |
|类型|Bool|
|默认值| true |
-|改后生效方式|重启服务器生效|
+|改后生效方式|触发生效|
* fetch_size
@@ -199,7 +204,7 @@
|描述| 写前日志定期刷新到磁盘的周期，单位毫秒，有可能丢失至多flush\_wal\_period\_in\_ms毫秒的操作。 |
|类型|Int32|
|默认值| 10 |
-|改后生效方式|重启服务器生效|
+|改后生效方式|触发生效|
* flush\_wal\_threshold
@@ -208,7 +213,7 @@
|描述| 写前日志的条数达到该值之后，刷新到磁盘，有可能丢失至多flush\_wal\_threshold个操作 |
|类型|Int32|
|默认值| 10000 |
-|改后生效方式|重启服务器生效|
+|改后生效方式|触发生效|
* merge\_concurrent\_threads
@@ -226,7 +231,7 @@
|描述| IoTDB在tsfile\_dir中为TsFile选择目录时采用的策略。可使用简单类名或类名全称。系统提供以下三种策略：<br>1. SequenceStrategy：IoTDB按顺序从tsfile\_dir中选择目录，依次遍历tsfile\_dir中的所有目录，并不断轮循；<br>2. MaxDiskUsableSpaceFirstStrategy：IoTDB优先选择tsfile\_dir中对应磁盘空余空间最大的目录；<br>3. MinFolderOccupiedSpaceFirstStrategy：IoTDB优先选择tsfile\_dir中已使用空间最小的目录；<br>4. <UserDfineStrategyPackage>（用户自定义策略）<br>您可以通过以下方法完成用户自定义策略：<br>1. 继承cn.edu.tsinghua.iotdb.conf.directories.strategy.DirectoryStrategy类并实现自身的Strategy方法；<br>2. 将实现的类的完整类名（包名加类名，UserDfineStrategyPa [...]
|类型|String|
|默认值| MaxDiskUsableSpaceFirstStrategy |
-|改后生效方式|重启服务器生效|
+|改后生效方式|触发生效|
* rpc_address
@@ -246,14 +251,14 @@
|默认值| 6667 |
|改后生效方式|重启服务器生效||
-* time_zone
+* time\_zone
|名字| time_zone |
|:---:|:---|
|描述| 服务器所处的时区，默认为北京时间（东八区） |
|类型|Time Zone String|
|默认值| +08:00 |
-|改后生效方式|重启服务器生效|
+|改后生效方式|触发生效|
* enable\_stat\_monitor
diff --git a/docs/Documentation/UserGuide/3-Server/4-Config Manual.md b/docs/Documentation/UserGuide/3-Server/4-Config Manual.md
index 450d645..cd9a07d 100644
--- a/docs/Documentation/UserGuide/3-Server/4-Config Manual.md
+++ b/docs/Documentation/UserGuide/3-Server/4-Config Manual.md
@@ -29,13 +29,20 @@ Before starting to use IoTDB, you need to config the configuration files first.
In total, we provide users three kinds of configurations module:
* environment configuration file (`iotdb-env.bat`, `iotdb-env.sh`). The default configuration file for the environment configuration item. Users can configure the relevant system configuration items of JAVA-JVM in the file.
-* system configuration file (`tsfile-format.properties`, `iotdb-engine.properties`).
- * `tsfile-format.properties`: The default configuration file for the IoTDB file layer configuration item. Users can configure the information about the TsFile, such as the data size written to the disk per time(`group_size_in_byte`).
- * `iotdb-engine.properties`: The default configuration file for the IoTDB engine layer configuration item. Users can configure the IoTDB engine related parameters in the file, such as JDBC service listening port (`rpc_port`), unsequence data storage directory (`unsequence_data_dir`), etc.
+* system configuration file (`iotdb-engine.properties`).
+ * `iotdb-engine.properties`: The default configuration file for the IoTDB engine layer configuration item. Users can configure the IoTDB engine related parameters in the file, such as JDBC service listening port (`rpc_port`), unsequence data storage directory (`unsequence_data_dir`), etc. What's more, Users can configure the information about the TsFile, such as the data size written to the disk per time(`group_size_in_byte`).
+
* log configuration file (`logback.xml`)
The configuration files of the three configuration items are located in the IoTDB installation directory: `$IOTDB_HOME/conf` folder.
+### Hot Modification Configuration
+
+For the convenience of users, IoTDB server provides users with hot modification function, that is, modifying some configuration parameters in `iotdb engine. Properties` during the system operation and applying them to the system immediately.
+In the parameters described below, these parameters whose way of `Effective` is `trigger` support hot modification.
+
+Trigger way: The client sends the command `load configuration` to the IoTDB server. See Chapter 4 for the usage of the client.
+
### IoTDB Environment Configuration File
The environment configuration file is mainly used to configure the Java environment related parameters when IoTDB Server is running, such as JVM related configuration. This part of the configuration is passed to the JVM when the IoTDB Server starts. Users can view the contents of the environment configuration file by viewing the `iotdb-env.sh` (or `iotdb-env.bat`) file.
@@ -90,7 +97,7 @@ The detail of each variables are as follows:
|Description|Data compression method|
|Type|Enum String : “UNCOMPRESSED”, “SNAPPY”|
|Default| UNCOMPRESSED |
-|Effective|Immediately|
+|Effective|Trigger|
* group\_size\_in\_byte
@@ -99,7 +106,7 @@ The detail of each variables are as follows:
|Description|The data size written to the disk per time|
|Type|Int32|
|Default| 134217728 |
-|Effective|Immediately|
+|Effective|Trigger|
* page\_size\_in\_byte
@@ -108,7 +115,7 @@ The detail of each variables are as follows:
|Description|The maximum size of a single page written in memory when each column in memory is written (in bytes)|
|Type|Int32|
|Default| 65536 |
-|Effective|Immediately|
+|Effective|Trigger|
* max\_number\_of\_points\_in\_page
@@ -117,7 +124,7 @@ The detail of each variables are as follows:
|Description|The maximum number of data points (timestamps - valued groups) contained in a page|
|Type|Int32|
|Default| 1048576 |
-|Effective|Immediately|
+|Effective|Trigger|
* max\_string\_length
@@ -126,7 +133,7 @@ The detail of each variables are as follows:
|Description|The maximum length of a single string (number of character)|
|Type|Int32|
|Default| 128 |
-|Effective|Immediately|
+|Effective|Trigger|
* time\_series\_data\_type
@@ -135,7 +142,7 @@ The detail of each variables are as follows:
|Description|Timestamp data type|
|Type|Enum String: "INT32", "INT64"|
|Default| Int64 |
-|Effective|Immediately|
+|Effective|Trigger|
* time\_encoder
@@ -144,7 +151,7 @@ The detail of each variables are as follows:
|Description| Encoding type of time column|
|Type|Enum String: “TS_2DIFF”,“PLAIN”,“RLE”|
|Default| TS_2DIFF |
-|Effective|Immediately|
+|Effective|Trigger|
* value\_encoder
@@ -153,7 +160,7 @@ The detail of each variables are as follows:
|Description| Encoding type of value column|
|Type|Enum String: “TS_2DIFF”,“PLAIN”,“RLE”|
|Default| PLAIN |
-|Effective|Immediately|
+|Effective|Trigger|
* float_precision
@@ -162,7 +169,7 @@ The detail of each variables are as follows:
|Description| The precision of the floating point number.(The number of digits after the decimal point) |
|Type|Int32|
|Default| The default is 2 digits. Note: The 32-bit floating point number has a decimal precision of 7 bits, and the 64-bit floating point number has a decimal precision of 15 bits. If the setting is out of the range, it will have no practical significance. |
-|Effective|Immediately|
+|Effective|Trigger|
#### Engine Layer
@@ -191,7 +198,7 @@ The detail of each variables are as follows:
|Description| The time zone in which the server is located, the default is Beijing time (+8) |
|Type|Time Zone String|
|Default| +08:00 |
-|Effective|After restart system|
+|Effective|Trigger|
* base\_dir
@@ -209,7 +216,7 @@ The detail of each variables are as follows:
|Description| The directories of data files. Multiple directories are separated by comma. The starting directory of the relative path is related to the operating system. It is recommended to use an absolute path. If the path does not exist, the system will automatically create it.|
|Type|String[]|
|Default| data/data |
-|Effective|After restart system|
+|Effective|Trigger|
* wal\_dir
@@ -227,7 +234,7 @@ The detail of each variables are as follows:
|Description| Whether to enable the pre-write log. The default value is true(enabled), and false means closed. |
|Type|Bool|
|Default| true |
-|Effective|After restart system|
+|Effective|Trigger|
* multi\_dir\_strategy
@@ -236,7 +243,7 @@ The detail of each variables are as follows:
|Description| IoTDB's strategy for selecting directories for TsFile in tsfile_dir. You can use a simple class name or a full name of the class. The system provides the following three strategies: <br>1. SequenceStrategy: IoTDB selects the directory from tsfile\_dir in order, traverses all the directories in tsfile\_dir in turn, and keeps counting;<br>2. MaxDiskUsableSpaceFirstStrategy: IoTDB first selects the directory with the largest free disk space in tsfile\_dir;<br>3. MinFolderOccup [...]
|Type|String|
|Default| MaxDiskUsableSpaceFirstStrategy |
-|Effective|After restart system|
+|Effective|Trigger|
* tsfile\_size\_threshold
@@ -254,7 +261,7 @@ The detail of each variables are as follows:
|Description| After the WAL reaches this value, it is flushed to disk, and it is possible to lose at most flush_wal_threshold operations. |
|Type|Int32|
|Default| 10000 |
-|Effective|After restart system|
+|Effective|Trigger|
* force\_wal\_period\_in\_ms
@@ -263,7 +270,7 @@ The detail of each variables are as follows:
|Description| The period during which the log is periodically forced to flush to disk(in milliseconds) |
|Type|Int32|
|Default| 10 |
-|Effective|After restart system|
+|Effective|Trigger|
* fetch\_size
diff --git a/server/src/assembly/resources/conf/iotdb-env.bat b/server/src/assembly/resources/conf/iotdb-env.bat
index 5484cd9..e04a409 100644
--- a/server/src/assembly/resources/conf/iotdb-env.bat
+++ b/server/src/assembly/resources/conf/iotdb-env.bat
@@ -54,15 +54,22 @@ for /f "tokens=1-3" %%j in ('java -version 2^>^&1') do (
set BIT_VERSION=%%l
)
IF "%BIT_VERSION%" == "64-Bit" (
- rem 64-bit Java
- echo Detect 64-bit Java, maximum memory allocation pool = 2GB, initial memory allocation pool = 2GB
- set IOTDB_HEAP_OPTS=-Xmx2G -Xms2G
+ rem 64bit, Maximum heap size
+ set MAX_HEAP_SIZE="2G"
+ rem 64bit, Minimum heap size
+ set HEAP_NEWSIZE="2G"
) ELSE (
- rem 32-bit Java
- echo Detect 32-bit Java, maximum memory allocation pool = 512MB, initial memory allocation pool = 512MB
- set IOTDB_HEAP_OPTS=-Xmx512M -Xms512M
+ rem 32bit, Maximum heap size
+ set MAX_HEAP_SIZE="512M"
+ rem 32bit, Minimum heap size
+ set HEAP_NEWSIZE="512M"
)
+@REM MAX_HEAP_SIZE="2G"
+@REM HEAP_NEWSIZE="2G"
+
+set IOTDB_HEAP_OPTS=-Xmx%MAX_HEAP_SIZE% -Xms%HEAP_NEWSIZE%
+
@REM You can put your env variable here
@REM set JAVA_HOME=%JAVA_HOME%
diff --git a/server/src/main/antlr3/org/apache/iotdb/db/sql/parse/TqlLexer.g b/server/src/main/antlr3/org/apache/iotdb/db/sql/parse/TqlLexer.g
index c49fb6f..6a00dde 100644
--- a/server/src/main/antlr3/org/apache/iotdb/db/sql/parse/TqlLexer.g
+++ b/server/src/main/antlr3/org/apache/iotdb/db/sql/parse/TqlLexer.g
@@ -273,6 +273,10 @@ K_UNSET
: U N S E T
;
+K_CONFIGURATION
+ : C O N F I G U R A T I O N
+ ;
+
//************** logical operator***********
OPERATOR_AND
: A N D
diff --git a/server/src/main/antlr3/org/apache/iotdb/db/sql/parse/TqlParser.g b/server/src/main/antlr3/org/apache/iotdb/db/sql/parse/TqlParser.g
index 734adc2..9f420c4 100644
--- a/server/src/main/antlr3/org/apache/iotdb/db/sql/parse/TqlParser.g
+++ b/server/src/main/antlr3/org/apache/iotdb/db/sql/parse/TqlParser.g
@@ -107,6 +107,7 @@ tokens{
TOK_SHOW;
TOK_DATE_EXPR;
TOK_DURATION;
+ TOK_LOAD_CONFIGURATION;
}
@header{
@@ -121,13 +122,13 @@ ArrayList<ParseError> errors = new ArrayList<ParseError>();
Stack messages = new Stack<String>();
private static HashMap<String, String> tokenNameMap;
static {
- tokenNameMap = new HashMap<String, String>();
- tokenNameMap.put("K_AND", "AND");
- tokenNameMap.put("K_OR", "OR");
- tokenNameMap.put("K_NOT", "NOT");
- tokenNameMap.put("K_LIKE", "LIKE");
- tokenNameMap.put("K_BY", "BY");
- tokenNameMap.put("K_GROUP", "GROUP");
+ tokenNameMap = new HashMap<String, String>();
+ tokenNameMap.put("K_AND", "AND");
+ tokenNameMap.put("K_OR", "OR");
+ tokenNameMap.put("K_NOT", "NOT");
+ tokenNameMap.put("K_LIKE", "LIKE");
+ tokenNameMap.put("K_BY", "BY");
+ tokenNameMap.put("K_GROUP", "GROUP");
tokenNameMap.put("K_FILL", "FILL");
tokenNameMap.put("K_LINEAR", "LINEAR");
tokenNameMap.put("K_PREVIOUS", "PREVIOUS");
@@ -274,6 +275,7 @@ sqlStatement
: ddlStatement
| dmlStatement
| administrationStatement
+ | configurationStatement
;
dmlStatement
@@ -754,6 +756,15 @@ rootOrId
| ID
;
+configurationStatement
+ : loadConfigurationStatement
+ ;
+
+loadConfigurationStatement
+ : K_LOAD K_CONFIGURATION
+ -> ^(TOK_LOAD_CONFIGURATION)
+ ;
+
/*
****
*************
@@ -763,21 +774,18 @@ TTL
*/
ttlStatement
- :
- setTTLStatement
+ : setTTLStatement
| unsetTTLStatement
| showTTLStatement
;
setTTLStatement
- :
- K_SET K_TTL K_TO path=prefixPath time=INT
+ : K_SET K_TTL K_TO path=prefixPath time=INT
-> ^(TOK_TTL TOK_SET $path $time)
;
unsetTTLStatement
- :
- K_UNSET K_TTL K_TO path=prefixPath
+ : K_UNSET K_TTL K_TO path=prefixPath
-> ^(TOK_TTL TOK_UNSET $path)
;
diff --git a/server/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java b/server/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java
index 7f6ea30..1124058 100644
--- a/server/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java
+++ b/server/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java
@@ -20,12 +20,11 @@ package org.apache.iotdb.db.conf;
import java.io.File;
import java.time.ZoneId;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
+import org.apache.iotdb.db.conf.directories.DirectoryManager;
import org.apache.iotdb.db.engine.merge.selector.MergeFileStrategy;
+import org.apache.iotdb.db.exception.LoadConfigurationException;
import org.apache.iotdb.db.metadata.MManager;
import org.apache.iotdb.db.service.TSServiceImpl;
import org.apache.iotdb.tsfile.common.conf.TSFileDescriptor;
@@ -467,50 +466,41 @@ public class IoTDBConfig {
* if the folders are relative paths, add IOTDB_HOME as the path prefix
*/
private void formulateFolders() {
- List<String> dirs = new ArrayList<>();
- dirs.add(baseDir);
- dirs.add(systemDir);
- dirs.add(schemaDir);
- dirs.add(walFolder);
- dirs.add(indexFileDir);
- dirs.add(queryDir);
- dirs.addAll(Arrays.asList(dataDirs));
-
- for (int i = 0; i < 4; i++) {
- addHomeDir(dirs, i);
- }
+ baseDir = addHomeDir(baseDir);
+ systemDir = addHomeDir(systemDir);
+ schemaDir = addHomeDir(schemaDir);
+ walFolder = addHomeDir(walFolder);
if (TSFileDescriptor.getInstance().getConfig().getTSFileStorageFs().equals(FSType.HDFS)) {
- String[] hdfsIps = TSFileDescriptor.getInstance().getConfig().getHdfsIp();
- String hdfsDir = "hdfs://";
- if (hdfsIps.length > 1) {
- hdfsDir += TSFileDescriptor.getInstance().getConfig().getDfsNameServices();
- } else {
- hdfsDir += hdfsIps[0] + ":" + TSFileDescriptor.getInstance().getConfig().getHdfsPort();
- }
- for (int i = 5; i < dirs.size(); i++) {
- String dir = dirs.get(i);
- dir = hdfsDir + File.separatorChar + dir;
- dirs.set(i, dir);
+ String hdfsDir = getHdfsDir();
+ queryDir = hdfsDir + File.separatorChar + queryDir;
+ for (int i = 0; i < dataDirs.length; i++) {
+ dataDirs[i] = hdfsDir + File.separatorChar + dataDirs[i];
}
} else {
- for (int i = 5; i < dirs.size(); i++) {
- addHomeDir(dirs, i);
+ queryDir = addHomeDir(queryDir);
+ for (int i = 0; i < dataDirs.length; i++) {
+ dataDirs[i] = addHomeDir(dataDirs[i]);
}
}
- baseDir = dirs.get(0);
- systemDir = dirs.get(1);
- schemaDir = dirs.get(2);
- walFolder = dirs.get(3);
- indexFileDir = dirs.get(4);
- queryDir = dirs.get(5);
- for (int i = 0; i < dataDirs.length; i++) {
- dataDirs[i] = dirs.get(i + 6);
+ }
+
+ void reloadDataDirs(String[] dataDirs) throws LoadConfigurationException {
+ if (TSFileDescriptor.getInstance().getConfig().getTSFileStorageFs().equals(FSType.HDFS)) {
+ String hdfsDir = getHdfsDir();
+ for (int i = 0; i < dataDirs.length; i++) {
+ dataDirs[i] = hdfsDir + File.separatorChar + dataDirs[i];
+ }
+ } else {
+ for (int i = 0; i < dataDirs.length; i++) {
+ dataDirs[i] = addHomeDir(dataDirs[i]);
+ }
}
+ this.dataDirs = dataDirs;
+ DirectoryManager.getInstance().updateFileFolders();
}
- private void addHomeDir(List<String> dirs, int i) {
- String dir = dirs.get(i);
+ private String addHomeDir(String dir) {
String homeDir = System.getProperty(IoTDBConstant.IOTDB_HOME, null);
if (!new File(dir).isAbsolute() && homeDir != null && homeDir.length() > 0) {
if (!homeDir.endsWith(File.separator)) {
@@ -518,8 +508,8 @@ public class IoTDBConfig {
} else {
dir = homeDir + dir;
}
- dirs.set(i, dir);
}
+ return dir;
}
private void confirmMultiDirStrategy() {
@@ -539,6 +529,17 @@ public class IoTDBConfig {
}
}
+ private String getHdfsDir(){
+ String[] hdfsIps = TSFileDescriptor.getInstance().getConfig().getHdfsIp();
+ String hdfsDir = "hdfs://";
+ if (hdfsIps.length > 1) {
+ hdfsDir += TSFileDescriptor.getInstance().getConfig().getDfsNameServices();
+ } else {
+ hdfsDir += hdfsIps[0] + ":" + TSFileDescriptor.getInstance().getConfig().getHdfsPort();
+ }
+ return hdfsDir;
+ }
+
public String[] getDataDirs() {
return dataDirs;
}
diff --git a/server/src/main/java/org/apache/iotdb/db/conf/IoTDBConfigCheck.java b/server/src/main/java/org/apache/iotdb/db/conf/IoTDBConfigCheck.java
index 7b6bdef..09c648f 100644
--- a/server/src/main/java/org/apache/iotdb/db/conf/IoTDBConfigCheck.java
+++ b/server/src/main/java/org/apache/iotdb/db/conf/IoTDBConfigCheck.java
@@ -54,7 +54,7 @@ public class IoTDBConfigCheck {
logger.info("System configuration is ok.");
}
- public void createDir(String filepath) {
+ private void createDir(String filepath) {
File dir = SystemFileFactory.INSTANCE.getFile(filepath);
if (!dir.exists()) {
dir.mkdirs();
@@ -62,7 +62,7 @@ public class IoTDBConfigCheck {
}
}
- public void checkFile(String filepath) {
+ private void checkFile(String filepath) {
// create file : read timestamp precision from engine.properties, create system_properties.txt
// use output stream to write timestamp precision to file.
File file = SystemFileFactory.INSTANCE.getFile(filepath + File.separator + PROPERTIES_FILE_NAME);
diff --git a/server/src/main/java/org/apache/iotdb/db/conf/IoTDBDescriptor.java b/server/src/main/java/org/apache/iotdb/db/conf/IoTDBDescriptor.java
index 8c4661e..97bfa37 100644
--- a/server/src/main/java/org/apache/iotdb/db/conf/IoTDBDescriptor.java
+++ b/server/src/main/java/org/apache/iotdb/db/conf/IoTDBDescriptor.java
@@ -25,6 +25,8 @@ import java.io.IOException;
import java.io.InputStream;
import java.time.ZoneId;
import java.util.Properties;
+import org.apache.iotdb.db.conf.directories.DirectoryManager;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
import org.apache.iotdb.db.utils.FilePathUtils;
import org.apache.iotdb.tsfile.common.conf.TSFileDescriptor;
import org.slf4j.Logger;
@@ -72,25 +74,15 @@ public class IoTDBDescriptor {
* load an property file and set TsfileDBConfig variables.
*/
private void loadProps() {
- InputStream inputStream;
-
String url = getPropsUrl();
if (url == null) {
return;
}
- try {
- inputStream = new FileInputStream(new File(url));
- } catch (FileNotFoundException e) {
- logger.warn("Fail to find config file {}", url, e);
- // update all data seriesPath
- conf.updatePath();
- return;
- }
+ try (InputStream inputStream = new FileInputStream(new File(url))) {
- logger.info("Start to read config file {}", url);
- Properties properties = new Properties();
- try {
+ logger.info("Start to read config file {}", url);
+ Properties properties = new Properties();
properties.load(inputStream);
conf.setEnableStatMonitor(Boolean
.parseBoolean(properties.getProperty("enable_stat_monitor",
@@ -144,8 +136,7 @@ public class IoTDBDescriptor {
initMemoryAllocate(properties);
- conf.setEnableWal(Boolean.parseBoolean(properties.getProperty("enable_wal",
- Boolean.toString(conf.isEnableWal()))));
+ loadWALProps(properties);
conf.setBaseDir(properties.getProperty("base_dir", conf.getBaseDir()));
@@ -160,14 +151,6 @@ public class IoTDBDescriptor {
conf.setWalFolder(properties.getProperty("wal_dir", conf.getWalFolder()));
- conf.setFlushWalThreshold(Integer
- .parseInt(properties.getProperty("flush_wal_threshold",
- Integer.toString(conf.getFlushWalThreshold()))));
-
- conf.setForceWalPeriodInMs(Long
- .parseLong(properties.getProperty("force_wal_period_in_ms",
- Long.toString(conf.getForceWalPeriodInMs()))));
-
int walBufferSize = Integer.parseInt(properties.getProperty("wal_buffer_size",
Integer.toString(conf.getWalBufferSize())));
if (walBufferSize > 0) {
@@ -223,8 +206,8 @@ public class IoTDBDescriptor {
conf.setChunkBufferPoolEnable(Boolean
.parseBoolean(properties.getProperty("chunk_buffer_pool_enable")));
}
- String tmpTimeZone = properties.getProperty("time_zone", conf.getZoneID().toString());
- conf.setZoneID(ZoneId.of(tmpTimeZone.trim()));
+ conf.setZoneID(
+ ZoneId.of(properties.getProperty("time_zone", conf.getZoneID().toString().trim())));
logger.info("Time zone has been set to {}", conf.getZoneID());
conf.setEnableExternalSort(Boolean.parseBoolean(properties
@@ -280,30 +263,7 @@ public class IoTDBDescriptor {
conf.setWatermarkMethod(
properties.getProperty("watermark_method", conf.getWatermarkMethod()));
- conf.setAutoCreateSchemaEnabled(
- Boolean.parseBoolean(properties.getProperty("enable_auto_create_schema",
- Boolean.toString(conf.isAutoCreateSchemaEnabled()).trim())));
- conf.setDefaultStorageGroupLevel(
- Integer.parseInt(properties.getProperty("default_storage_group_level",
- Integer.toString(conf.getDefaultStorageGroupLevel()))));
- conf.setDefaultBooleanEncoding(
- properties.getProperty("default_boolean_encoding",
- conf.getDefaultBooleanEncoding().toString()));
- conf.setDefaultInt32Encoding(
- properties.getProperty("default_int32_encoding",
- conf.getDefaultInt32Encoding().toString()));
- conf.setDefaultInt64Encoding(
- properties.getProperty("default_int64_encoding",
- conf.getDefaultInt64Encoding().toString()));
- conf.setDefaultFloatEncoding(
- properties.getProperty("default_float_encoding",
- conf.getDefaultFloatEncoding().toString()));
- conf.setDefaultDoubleEncoding(
- properties.getProperty("default_double_encoding",
- conf.getDefaultDoubleEncoding().toString()));
- conf.setDefaultTextEncoding(
- properties.getProperty("default_text_encoding",
- conf.getDefaultTextEncoding().toString()));
+ loadAutoCreateSchemaProps(properties);
conf.setRpcMaxConcurrentClientNum(maxConcurrentClientNum);
@@ -353,58 +313,159 @@ public class IoTDBDescriptor {
// set tsfile-format config
- TSFileDescriptor.getInstance().getConfig().setGroupSizeInByte(Integer
- .parseInt(properties.getProperty("group_size_in_byte",
- Integer.toString(TSFileDescriptor.getInstance().getConfig().getGroupSizeInByte()))));
- TSFileDescriptor.getInstance().getConfig().setPageSizeInByte(Integer
- .parseInt(properties.getProperty("page_size_in_byte",
- Integer.toString(TSFileDescriptor.getInstance().getConfig().getPageSizeInByte()))));
- if (TSFileDescriptor.getInstance().getConfig().getPageSizeInByte() > TSFileDescriptor
- .getInstance().getConfig().getGroupSizeInByte()) {
- logger
- .warn("page_size is greater than group size, will set it as the same with group size");
- TSFileDescriptor.getInstance().getConfig()
- .setPageSizeInByte(TSFileDescriptor.getInstance().getConfig().getGroupSizeInByte());
- }
- TSFileDescriptor.getInstance().getConfig().setMaxNumberOfPointsInPage(Integer
- .parseInt(properties.getProperty("max_number_of_points_in_page",
- Integer.toString(
- TSFileDescriptor.getInstance().getConfig().getMaxNumberOfPointsInPage()))));
- TSFileDescriptor.getInstance().getConfig().setTimeSeriesDataType(properties
- .getProperty("time_series_data_type",
- TSFileDescriptor.getInstance().getConfig().getTimeSeriesDataType()));
- TSFileDescriptor.getInstance().getConfig().setMaxStringLength(Integer
- .parseInt(properties.getProperty("max_string_length",
- Integer.toString(TSFileDescriptor.getInstance().getConfig().getMaxStringLength()))));
- TSFileDescriptor.getInstance().getConfig().setBloomFilterErrorRate(Double
- .parseDouble(properties.getProperty("bloom_filter_error_rate",
- Double.toString(
- TSFileDescriptor.getInstance().getConfig().getBloomFilterErrorRate()))));
- TSFileDescriptor.getInstance().getConfig().setFloatPrecision(Integer
- .parseInt(properties
- .getProperty("float_precision", Integer
- .toString(TSFileDescriptor.getInstance().getConfig().getFloatPrecision()))));
- TSFileDescriptor.getInstance().getConfig().setTimeEncoder(properties
- .getProperty("time_encoder",
- TSFileDescriptor.getInstance().getConfig().getTimeEncoder()));
- TSFileDescriptor.getInstance().getConfig().setValueEncoder(properties
- .getProperty("value_encoder",
- TSFileDescriptor.getInstance().getConfig().getValueEncoder()));
- TSFileDescriptor.getInstance().getConfig().setCompressor(properties
- .getProperty("compressor", TSFileDescriptor.getInstance().getConfig().getCompressor()));
+ loadTsFileProps(properties);
+ } catch (FileNotFoundException e) {
+ logger.warn("Fail to find config file {}", url, e);
} catch (IOException e) {
- logger.warn("Cannot load config file because, use default configuration", e);
+ logger.warn("Cannot load config file, use default configuration", e);
} catch (Exception e) {
logger.warn("Incorrect format in config file, use default configuration", e);
} finally {
// update all data seriesPath
conf.updatePath();
- try {
- inputStream.close();
- } catch (IOException e) {
- logger.error("Fail to close config file input stream because ", e);
+ }
+ }
+
+ private void loadWALProps(Properties properties){
+ conf.setEnableWal(Boolean.parseBoolean(properties.getProperty("enable_wal",
+ Boolean.toString(conf.isEnableWal()))));
+
+ conf.setFlushWalThreshold(Integer
+ .parseInt(properties.getProperty("flush_wal_threshold",
+ Integer.toString(conf.getFlushWalThreshold()))));
+
+ conf.setForceWalPeriodInMs(Long
+ .parseLong(properties.getProperty("force_wal_period_in_ms",
+ Long.toString(conf.getForceWalPeriodInMs()))));
+
+ }
+
+ private void loadAutoCreateSchemaProps(Properties properties){
+ conf.setAutoCreateSchemaEnabled(
+ Boolean.parseBoolean(properties.getProperty("enable_auto_create_schema",
+ Boolean.toString(conf.isAutoCreateSchemaEnabled()).trim())));
+ conf.setDefaultStorageGroupLevel(
+ Integer.parseInt(properties.getProperty("default_storage_group_level",
+ Integer.toString(conf.getDefaultStorageGroupLevel()))));
+ conf.setDefaultBooleanEncoding(
+ properties.getProperty("default_boolean_encoding",
+ conf.getDefaultBooleanEncoding().toString()));
+ conf.setDefaultInt32Encoding(
+ properties.getProperty("default_int32_encoding",
+ conf.getDefaultInt32Encoding().toString()));
+ conf.setDefaultInt64Encoding(
+ properties.getProperty("default_int64_encoding",
+ conf.getDefaultInt64Encoding().toString()));
+ conf.setDefaultFloatEncoding(
+ properties.getProperty("default_float_encoding",
+ conf.getDefaultFloatEncoding().toString()));
+ conf.setDefaultDoubleEncoding(
+ properties.getProperty("default_double_encoding",
+ conf.getDefaultDoubleEncoding().toString()));
+ conf.setDefaultTextEncoding(
+ properties.getProperty("default_text_encoding",
+ conf.getDefaultTextEncoding().toString()));
+ }
+
+ private void loadTsFileProps(Properties properties){
+ TSFileDescriptor.getInstance().getConfig().setGroupSizeInByte(Integer
+ .parseInt(properties.getProperty("group_size_in_byte",
+ Integer.toString(TSFileDescriptor.getInstance().getConfig().getGroupSizeInByte()))));
+ TSFileDescriptor.getInstance().getConfig().setPageSizeInByte(Integer
+ .parseInt(properties.getProperty("page_size_in_byte",
+ Integer.toString(TSFileDescriptor.getInstance().getConfig().getPageSizeInByte()))));
+ if (TSFileDescriptor.getInstance().getConfig().getPageSizeInByte() > TSFileDescriptor
+ .getInstance().getConfig().getGroupSizeInByte()) {
+ logger
+ .warn("page_size is greater than group size, will set it as the same with group size");
+ TSFileDescriptor.getInstance().getConfig()
+ .setPageSizeInByte(TSFileDescriptor.getInstance().getConfig().getGroupSizeInByte());
+ }
+ TSFileDescriptor.getInstance().getConfig().setMaxNumberOfPointsInPage(Integer
+ .parseInt(properties.getProperty("max_number_of_points_in_page",
+ Integer.toString(
+ TSFileDescriptor.getInstance().getConfig().getMaxNumberOfPointsInPage()))));
+ TSFileDescriptor.getInstance().getConfig().setTimeSeriesDataType(properties
+ .getProperty("time_series_data_type",
+ TSFileDescriptor.getInstance().getConfig().getTimeSeriesDataType()));
+ TSFileDescriptor.getInstance().getConfig().setMaxStringLength(Integer
+ .parseInt(properties.getProperty("max_string_length",
+ Integer.toString(TSFileDescriptor.getInstance().getConfig().getMaxStringLength()))));
+ TSFileDescriptor.getInstance().getConfig().setBloomFilterErrorRate(Double
+ .parseDouble(properties.getProperty("bloom_filter_error_rate",
+ Double.toString(
+ TSFileDescriptor.getInstance().getConfig().getBloomFilterErrorRate()))));
+ TSFileDescriptor.getInstance().getConfig().setFloatPrecision(Integer
+ .parseInt(properties
+ .getProperty("float_precision", Integer
+ .toString(TSFileDescriptor.getInstance().getConfig().getFloatPrecision()))));
+ TSFileDescriptor.getInstance().getConfig().setTimeEncoder(properties
+ .getProperty("time_encoder",
+ TSFileDescriptor.getInstance().getConfig().getTimeEncoder()));
+ TSFileDescriptor.getInstance().getConfig().setValueEncoder(properties
+ .getProperty("value_encoder",
+ TSFileDescriptor.getInstance().getConfig().getValueEncoder()));
+ TSFileDescriptor.getInstance().getConfig().setCompressor(properties
+ .getProperty("compressor", TSFileDescriptor.getInstance().getConfig().getCompressor()));
+ }
+
+ public void loadHotModifiedProps() throws QueryProcessException {
+ String url = getPropsUrl();
+ if (url == null) {
+ return;
+ }
+
+ try (InputStream inputStream = new FileInputStream(new File(url))) {
+ logger.info("Start to reload config file {}", url);
+ Properties properties = new Properties();
+ properties.load(inputStream);
+
+ // update data dirs
+ String dataDirs = properties.getProperty("data_dirs", null);
+ if(dataDirs != null){
+ conf.reloadDataDirs(dataDirs.split(","));
}
+
+ // update dir strategy
+ String multiDirStrategyClassName = properties.getProperty("multi_dir_strategy", null);
+ if (multiDirStrategyClassName != null && !multiDirStrategyClassName
+ .equals(conf.getMultiDirStrategyClassName())) {
+ conf.setMultiDirStrategyClassName(multiDirStrategyClassName);
+ DirectoryManager.getInstance().updateDirectoryStrategy();
+ }
+
+ // update WAL conf
+ loadWALProps(properties);
+
+ // time zone
+ conf.setZoneID(
+ ZoneId.of(properties.getProperty("time_zone", conf.getZoneID().toString().trim())));
+
+ // dynamic parameters
+ long tsfileSizeThreshold = Long.parseLong(properties
+ .getProperty("tsfile_size_threshold",
+ Long.toString(conf.getTsFileSizeThreshold())).trim());
+ if (tsfileSizeThreshold > 0 && !conf.isEnableParameterAdapter()) {
+ conf.setTsFileSizeThreshold(tsfileSizeThreshold);
+ }
+
+ long memTableSizeThreshold = Long.parseLong(properties
+ .getProperty("memtable_size_threshold",
+ Long.toString(conf.getMemtableSizeThreshold())).trim());
+ if (memTableSizeThreshold > 0 && !conf.isEnableParameterAdapter()) {
+ conf.setMemtableSizeThreshold(memTableSizeThreshold);
+ }
+
+ // update params of creating schema automatically
+ loadAutoCreateSchemaProps(properties);
+
+ // update tsfile-format config
+ loadTsFileProps(properties);
+
+ } catch (Exception e) {
+ logger.warn("Fail to reload config file {}", url, e);
+ throw new QueryProcessException(String.format("Fail to reload config file %s because %s", url, e.getMessage()));
}
}
diff --git a/server/src/main/java/org/apache/iotdb/db/conf/directories/DirectoryManager.java b/server/src/main/java/org/apache/iotdb/db/conf/directories/DirectoryManager.java
index 8c14adb..5d3c37c 100644
--- a/server/src/main/java/org/apache/iotdb/db/conf/directories/DirectoryManager.java
+++ b/server/src/main/java/org/apache/iotdb/db/conf/directories/DirectoryManager.java
@@ -26,6 +26,7 @@ import org.apache.iotdb.db.conf.IoTDBConstant;
import org.apache.iotdb.db.conf.IoTDBDescriptor;
import org.apache.iotdb.db.conf.directories.strategy.DirectoryStrategy;
import org.apache.iotdb.db.exception.DiskSpaceInsufficientException;
+import org.apache.iotdb.db.exception.LoadConfigurationException;
import org.apache.iotdb.tsfile.fileSystem.FSFactoryProducer;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -64,11 +65,53 @@ public class DirectoryManager {
strategyName = IoTDBDescriptor.getInstance().getConfig().getMultiDirStrategyClassName();
Class<?> clazz = Class.forName(strategyName);
sequenceStrategy = (DirectoryStrategy) clazz.newInstance();
- sequenceStrategy.init(sequenceFileFolders);
+ sequenceStrategy.setFolders(sequenceFileFolders);
unsequenceStrategy = (DirectoryStrategy) clazz.newInstance();
- unsequenceStrategy.init(unsequenceFileFolders);
+ unsequenceStrategy.setFolders(unsequenceFileFolders);
+ } catch (DiskSpaceInsufficientException e) {
+ logger.error("All disks of folders are full.", e);
} catch (Exception e) {
- logger.error("can't find sequenceStrategy {} for mult-directories.", strategyName, e);
+ logger.error("Can't find strategy {} for mult-directories.", strategyName, e);
+ }
+ }
+
+ public void updateFileFolders() throws LoadConfigurationException {
+ try {
+ List<String> sequenceFileFolders =
+ new ArrayList<>(Arrays.asList(IoTDBDescriptor.getInstance().getConfig().getDataDirs()));
+ mkDataDirs(sequenceFileFolders);
+
+ List<String> unsequenceFileFolders =
+ new ArrayList<>(Arrays.asList(IoTDBDescriptor.getInstance().getConfig().getDataDirs()));
+ mkDataDirs(unsequenceFileFolders);
+ sequenceStrategy.setFolders(sequenceFileFolders);
+ unsequenceStrategy.setFolders(unsequenceFileFolders);
+ this.sequenceFileFolders = sequenceFileFolders;
+ this.unsequenceFileFolders = unsequenceFileFolders;
+ logger.info("Success to update file folders.");
+ } catch (DiskSpaceInsufficientException e) {
+ logger.error(
+ "Fail to update file folders, use previous folders.", e);
+ throw new LoadConfigurationException(
+ "Fail to update file folders because all disks of folders are full, use previous folders.");
+ }
+ }
+
+ public void updateDirectoryStrategy() throws LoadConfigurationException {
+ String strategyName = "";
+ try {
+ strategyName = IoTDBDescriptor.getInstance().getConfig().getMultiDirStrategyClassName();
+ Class<?> clazz = Class.forName(strategyName);
+ sequenceStrategy = (DirectoryStrategy) clazz.newInstance();
+ sequenceStrategy.setFolders(sequenceFileFolders);
+ unsequenceStrategy = (DirectoryStrategy) clazz.newInstance();
+ unsequenceStrategy.setFolders(unsequenceFileFolders);
+ logger.info("Success to update directory strategy.");
+ } catch (Exception e) {
+ logger.error("Fail to update directory strategy {}, use previous strategy", strategyName, e);
+ throw new LoadConfigurationException(String.format(
+ "Fail to update directory strategy because can't find strategy %s for mult-directories, use previous strategy",
+ strategyName));
}
}
@@ -105,10 +148,6 @@ public class DirectoryManager {
return sequenceFileFolders.get(index);
}
- public int getSequenceFileFolderIndex(String folder) {
- return sequenceFileFolders.indexOf(folder);
- }
-
public List<String> getAllSequenceFileFolders() {
return new ArrayList<>(sequenceFileFolders);
}
@@ -147,8 +186,4 @@ public class DirectoryManager {
return new ArrayList<>(unsequenceFileFolders);
}
- // only used by test
- public String getUnSequenceFolderForTest() {
- return unsequenceFileFolders.get(0);
- }
}
diff --git a/server/src/main/java/org/apache/iotdb/db/conf/directories/strategy/DirectoryStrategy.java b/server/src/main/java/org/apache/iotdb/db/conf/directories/strategy/DirectoryStrategy.java
index 46fce4c..aa92590 100644
--- a/server/src/main/java/org/apache/iotdb/db/conf/directories/strategy/DirectoryStrategy.java
+++ b/server/src/main/java/org/apache/iotdb/db/conf/directories/strategy/DirectoryStrategy.java
@@ -18,7 +18,9 @@
*/
package org.apache.iotdb.db.conf.directories.strategy;
+import java.util.ArrayList;
import java.util.List;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
import org.apache.iotdb.db.exception.DiskSpaceInsufficientException;
import org.apache.iotdb.db.utils.CommonUtils;
import org.slf4j.Logger;
@@ -35,7 +37,7 @@ public abstract class DirectoryStrategy {
/**
* All the folders of data files, should be init once the subclass is created.
*/
- List<String> folders;
+ List<String> folders = new ArrayList<>();
/**
* To init folders. Do not recommend to overwrite.
@@ -43,9 +45,7 @@ public abstract class DirectoryStrategy {
*
* @param folders the folders from conf
*/
- public void init(List<String> folders) throws DiskSpaceInsufficientException {
- this.folders = folders;
-
+ public void setFolders(List<String> folders) throws DiskSpaceInsufficientException {
boolean hasSpace = false;
for (String folder : folders) {
if (CommonUtils.hasSpace(folder)) {
@@ -54,8 +54,11 @@ public abstract class DirectoryStrategy {
}
}
if (!hasSpace) {
+ IoTDBDescriptor.getInstance().getConfig().setReadOnly(true);
throw new DiskSpaceInsufficientException(folders);
}
+
+ this.folders = folders;
}
/**
diff --git a/server/src/main/java/org/apache/iotdb/db/conf/directories/strategy/SequenceStrategy.java b/server/src/main/java/org/apache/iotdb/db/conf/directories/strategy/SequenceStrategy.java
index 75a7870..6917f27 100644
--- a/server/src/main/java/org/apache/iotdb/db/conf/directories/strategy/SequenceStrategy.java
+++ b/server/src/main/java/org/apache/iotdb/db/conf/directories/strategy/SequenceStrategy.java
@@ -27,10 +27,10 @@ public class SequenceStrategy extends DirectoryStrategy {
private int currentIndex;
@Override
- public void init(List<String> folders) throws DiskSpaceInsufficientException {
- super.init(folders);
+ public void setFolders(List<String> folders) throws DiskSpaceInsufficientException {
+ super.setFolders(folders);
- // super.init() ensures at least one folder is not full,
+ // super.setFolders() ensures at least one folder is not full,
// so currentIndex will not be -1 after loop
currentIndex = -1;
for (int i = 0; i < folders.size(); i++) {
diff --git a/server/src/main/java/org/apache/iotdb/db/exception/LoadConfigurationException.java b/server/src/main/java/org/apache/iotdb/db/exception/LoadConfigurationException.java
new file mode 100644
index 0000000..da52227
--- /dev/null
+++ b/server/src/main/java/org/apache/iotdb/db/exception/LoadConfigurationException.java
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.exception;
+
+public class LoadConfigurationException extends Exception {
+
+ private static final long serialVersionUID = -1950532739374479184L;
+
+ public LoadConfigurationException(String message) {
+ super(message);
+ }
+
+ public LoadConfigurationException(String message, Throwable cause) {
+ super(message, cause);
+ }
+
+ public LoadConfigurationException(Throwable cause) {
+ super(cause);
+ }
+
+}
diff --git a/server/src/main/java/org/apache/iotdb/db/qp/QueryProcessor.java b/server/src/main/java/org/apache/iotdb/db/qp/QueryProcessor.java
index ed33976..2e60a29 100644
--- a/server/src/main/java/org/apache/iotdb/db/qp/QueryProcessor.java
+++ b/server/src/main/java/org/apache/iotdb/db/qp/QueryProcessor.java
@@ -128,6 +128,7 @@ public class QueryProcessor {
case GRANT_WATERMARK_EMBEDDING:
case REVOKE_WATERMARK_EMBEDDING:
case TTL:
+ case LOAD_CONFIGURATION:
return operator;
case QUERY:
case UPDATE:
diff --git a/server/src/main/java/org/apache/iotdb/db/qp/constant/SQLConstant.java b/server/src/main/java/org/apache/iotdb/db/qp/constant/SQLConstant.java
index 8479e7a..adc63f6 100644
--- a/server/src/main/java/org/apache/iotdb/db/qp/constant/SQLConstant.java
+++ b/server/src/main/java/org/apache/iotdb/db/qp/constant/SQLConstant.java
@@ -102,6 +102,7 @@ public class SQLConstant {
public static final int TOK_SET = 63;
public static final int TOK_UNSET = 64;
public static final int TOK_SHOW = 65;
+ public static final int TOK_LOAD_CONFIGURATION = 66;
public static final Map<Integer, String> tokenSymbol = new HashMap<>();
public static final Map<Integer, String> tokenNames = new HashMap<>();
@@ -161,6 +162,8 @@ public class SQLConstant {
tokenNames.put(TOK_SET, "TOK_SET");
tokenNames.put(TOK_UNSET, "TOK_UNSET");
tokenNames.put(TOK_SHOW, "TOK_SHOW");
+
+ tokenNames.put(TOK_LOAD_CONFIGURATION, "TOK_LOAD_CONFIGURATION");
}
static {
diff --git a/server/src/main/java/org/apache/iotdb/db/qp/executor/QueryProcessExecutor.java b/server/src/main/java/org/apache/iotdb/db/qp/executor/QueryProcessExecutor.java
index 94669c1..d3418bc 100644
--- a/server/src/main/java/org/apache/iotdb/db/qp/executor/QueryProcessExecutor.java
+++ b/server/src/main/java/org/apache/iotdb/db/qp/executor/QueryProcessExecutor.java
@@ -137,6 +137,9 @@ public class QueryProcessExecutor extends AbstractQueryProcessExecutor {
case TTL:
operateTTL((SetTTLPlan) plan);
return true;
+ case LOAD_CONFIGURATION:
+ IoTDBDescriptor.getInstance().loadHotModifiedProps();
+ return true;
default:
throw new UnsupportedOperationException(
String.format("operation %s is not supported", plan.getOperatorType()));
diff --git a/server/src/main/java/org/apache/iotdb/db/qp/logical/Operator.java b/server/src/main/java/org/apache/iotdb/db/qp/logical/Operator.java
index 8b889a8..06fc4be 100644
--- a/server/src/main/java/org/apache/iotdb/db/qp/logical/Operator.java
+++ b/server/src/main/java/org/apache/iotdb/db/qp/logical/Operator.java
@@ -74,6 +74,6 @@ public abstract class Operator {
DELETE_ROLE, GRANT_ROLE_PRIVILEGE, REVOKE_ROLE_PRIVILEGE, LIST_USER, LIST_ROLE,
LIST_USER_PRIVILEGE, LIST_ROLE_PRIVILEGE, LIST_USER_ROLES, LIST_ROLE_USERS,
GRANT_WATERMARK_EMBEDDING, REVOKE_WATERMARK_EMBEDDING,
- TTL, DELETE_STORAGE_GROUP
+ TTL, DELETE_STORAGE_GROUP, LOAD_CONFIGURATION
}
}
diff --git a/server/src/main/java/org/apache/iotdb/db/qp/logical/sys/LoadConfigurationOperator.java b/server/src/main/java/org/apache/iotdb/db/qp/logical/sys/LoadConfigurationOperator.java
new file mode 100644
index 0000000..18ad4ae
--- /dev/null
+++ b/server/src/main/java/org/apache/iotdb/db/qp/logical/sys/LoadConfigurationOperator.java
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+package org.apache.iotdb.db.qp.logical.sys;
+
+import org.apache.iotdb.db.qp.constant.SQLConstant;
+import org.apache.iotdb.db.qp.logical.RootOperator;
+
+public class LoadConfigurationOperator extends RootOperator {
+
+ public LoadConfigurationOperator() {
+ super(SQLConstant.TOK_LOAD_CONFIGURATION);
+ this.operatorType = OperatorType.LOAD_CONFIGURATION;
+ }
+}
diff --git a/server/src/main/java/org/apache/iotdb/db/qp/physical/sys/LoadConfigurationPlan.java b/server/src/main/java/org/apache/iotdb/db/qp/physical/sys/LoadConfigurationPlan.java
new file mode 100644
index 0000000..ae0f77e
--- /dev/null
+++ b/server/src/main/java/org/apache/iotdb/db/qp/physical/sys/LoadConfigurationPlan.java
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+package org.apache.iotdb.db.qp.physical.sys;
+
+import java.util.List;
+import org.apache.iotdb.db.qp.logical.Operator.OperatorType;
+import org.apache.iotdb.db.qp.physical.PhysicalPlan;
+import org.apache.iotdb.tsfile.read.common.Path;
+
+public class LoadConfigurationPlan extends PhysicalPlan {
+
+ public LoadConfigurationPlan() {
+ super(false);
+ this.setOperatorType(OperatorType.LOAD_CONFIGURATION);
+ }
+
+ @Override
+ public List<Path> getPaths() {
+ return null;
+ }
+
+ @Override
+ public String toString() {
+ return getOperatorType().toString();
+ }
+}
diff --git a/server/src/main/java/org/apache/iotdb/db/qp/strategy/LogicalGenerator.java b/server/src/main/java/org/apache/iotdb/db/qp/strategy/LogicalGenerator.java
index 928b21b..2e7c962 100644
--- a/server/src/main/java/org/apache/iotdb/db/qp/strategy/LogicalGenerator.java
+++ b/server/src/main/java/org/apache/iotdb/db/qp/strategy/LogicalGenerator.java
@@ -50,6 +50,7 @@ import static org.apache.iotdb.db.sql.parse.TqlParser.TOK_LINEAR;
import static org.apache.iotdb.db.sql.parse.TqlParser.TOK_LINK;
import static org.apache.iotdb.db.sql.parse.TqlParser.TOK_LIST;
import static org.apache.iotdb.db.sql.parse.TqlParser.TOK_LOAD;
+import static org.apache.iotdb.db.sql.parse.TqlParser.TOK_LOAD_CONFIGURATION;
import static org.apache.iotdb.db.sql.parse.TqlParser.TOK_PATH;
import static org.apache.iotdb.db.sql.parse.TqlParser.TOK_PREVIOUS;
import static org.apache.iotdb.db.sql.parse.TqlParser.TOK_PRIVILEGES;
@@ -103,6 +104,7 @@ import org.apache.iotdb.db.qp.logical.sys.CreateTimeSeriesOperator;
import org.apache.iotdb.db.qp.logical.sys.DataAuthOperator;
import org.apache.iotdb.db.qp.logical.sys.DeleteStorageGroupOperator;
import org.apache.iotdb.db.qp.logical.sys.DeleteTimeSeriesOperator;
+import org.apache.iotdb.db.qp.logical.sys.LoadConfigurationOperator;
import org.apache.iotdb.db.qp.logical.sys.LoadDataOperator;
import org.apache.iotdb.db.qp.logical.sys.PropertyOperator;
import org.apache.iotdb.db.qp.logical.sys.SetStorageGroupOperator;
@@ -275,6 +277,9 @@ public class LogicalGenerator {
case TOK_GROUPBY_DEVICE:
((QueryOperator) initializedOperator).setGroupByDevice(true);
return;
+ case TOK_LOAD_CONFIGURATION:
+ initializedOperator = new LoadConfigurationOperator();
+ return;
default:
throw new QueryProcessException("Not supported TqlParser type " + token.getText());
}
@@ -302,8 +307,7 @@ public class LogicalGenerator {
private void analyzeSetTTL(AstNode astNode) {
String path = parsePath(astNode.getChild(1)).getFullPath();
- long dataTTL;
- dataTTL = Long.parseLong(astNode.getChild(2).getText());
+ long dataTTL = Long.parseLong(astNode.getChild(2).getText());
SetTTLOperator operator = new SetTTLOperator(SQLConstant.TOK_SET);
initializedOperator = operator;
operator.setStorageGroup(path);
diff --git a/server/src/main/java/org/apache/iotdb/db/qp/strategy/PhysicalGenerator.java b/server/src/main/java/org/apache/iotdb/db/qp/strategy/PhysicalGenerator.java
index e6629da..0aeef61 100644
--- a/server/src/main/java/org/apache/iotdb/db/qp/strategy/PhysicalGenerator.java
+++ b/server/src/main/java/org/apache/iotdb/db/qp/strategy/PhysicalGenerator.java
@@ -16,7 +16,6 @@
* specific language governing permissions and limitations
* under the License.
*/
-
package org.apache.iotdb.db.qp.strategy;
import java.util.ArrayList;
@@ -62,6 +61,7 @@ import org.apache.iotdb.db.qp.physical.sys.CreateTimeSeriesPlan;
import org.apache.iotdb.db.qp.physical.sys.DataAuthPlan;
import org.apache.iotdb.db.qp.physical.sys.DeleteStorageGroupPlan;
import org.apache.iotdb.db.qp.physical.sys.DeleteTimeSeriesPlan;
+import org.apache.iotdb.db.qp.physical.sys.LoadConfigurationPlan;
import org.apache.iotdb.db.qp.physical.sys.LoadDataPlan;
import org.apache.iotdb.db.qp.physical.sys.PropertyPlan;
import org.apache.iotdb.db.qp.physical.sys.SetStorageGroupPlan;
@@ -149,8 +149,12 @@ public class PhysicalGenerator {
case SQLConstant.TOK_SHOW:
ShowTTLOperator showTTLOperator = (ShowTTLOperator) operator;
return new ShowTTLPlan(showTTLOperator.getStorageGroups());
+ default:
+ throw new LogicalOperatorException(String
+ .format("not supported operator type %s in ttl operation.", operator.getType()));
}
-
+ case LOAD_CONFIGURATION:
+ return new LoadConfigurationPlan();
default:
throw new LogicalOperatorException(operator.getType().toString(), "");
}
diff --git a/server/src/test/java/org/apache/iotdb/db/conf/directories/strategy/DirectoryStrategyTest.java b/server/src/test/java/org/apache/iotdb/db/conf/directories/strategy/DirectoryStrategyTest.java
index 4158f79..7d92328 100644
--- a/server/src/test/java/org/apache/iotdb/db/conf/directories/strategy/DirectoryStrategyTest.java
+++ b/server/src/test/java/org/apache/iotdb/db/conf/directories/strategy/DirectoryStrategyTest.java
@@ -73,7 +73,7 @@ public class DirectoryStrategyTest {
@Test
public void testSequenceStrategy() throws DiskSpaceInsufficientException {
SequenceStrategy sequenceStrategy = new SequenceStrategy();
- sequenceStrategy.init(dataDirList);
+ sequenceStrategy.setFolders(dataDirList);
// loop two times of data dir size to fully loop
int index = 0;
@@ -89,7 +89,7 @@ public class DirectoryStrategyTest {
@Test
public void testMaxDiskUsableSpaceFirstStrategy() throws DiskSpaceInsufficientException {
MaxDiskUsableSpaceFirstStrategy maxDiskUsableSpaceFirstStrategy = new MaxDiskUsableSpaceFirstStrategy();
- maxDiskUsableSpaceFirstStrategy.init(dataDirList);
+ maxDiskUsableSpaceFirstStrategy.setFolders(dataDirList);
int maxIndex = getIndexOfMaxSpace();
for (int i = 0; i < dataDirList.size(); i++) {
@@ -120,7 +120,7 @@ public class DirectoryStrategyTest {
public void testMinFolderOccupiedSpaceFirstStrategy()
throws DiskSpaceInsufficientException, IOException {
MinFolderOccupiedSpaceFirstStrategy minFolderOccupiedSpaceFirstStrategy = new MinFolderOccupiedSpaceFirstStrategy();
- minFolderOccupiedSpaceFirstStrategy.init(dataDirList);
+ minFolderOccupiedSpaceFirstStrategy.setFolders(dataDirList);
int minIndex = getIndexOfMinOccupiedSpace();
for (int i = 0; i < dataDirList.size(); i++) {
@@ -155,21 +155,21 @@ public class DirectoryStrategyTest {
SequenceStrategy sequenceStrategy = new SequenceStrategy();
try {
- sequenceStrategy.init(dataDirList);
+ sequenceStrategy.setFolders(dataDirList);
fail();
} catch (DiskSpaceInsufficientException e) {
}
MaxDiskUsableSpaceFirstStrategy maxDiskUsableSpaceFirstStrategy = new MaxDiskUsableSpaceFirstStrategy();
try {
- maxDiskUsableSpaceFirstStrategy.init(dataDirList);
+ maxDiskUsableSpaceFirstStrategy.setFolders(dataDirList);
fail();
} catch (DiskSpaceInsufficientException e) {
}
MinFolderOccupiedSpaceFirstStrategy minFolderOccupiedSpaceFirstStrategy = new MinFolderOccupiedSpaceFirstStrategy();
try {
- minFolderOccupiedSpaceFirstStrategy.init(dataDirList);
+ minFolderOccupiedSpaceFirstStrategy.setFolders(dataDirList);
fail();
} catch (DiskSpaceInsufficientException e) {
}
diff --git a/server/src/test/java/org/apache/iotdb/db/qp/plan/PhysicalPlanTest.java b/server/src/test/java/org/apache/iotdb/db/qp/plan/PhysicalPlanTest.java
index 5284622..eec5c7b 100644
--- a/server/src/test/java/org/apache/iotdb/db/qp/plan/PhysicalPlanTest.java
+++ b/server/src/test/java/org/apache/iotdb/db/qp/plan/PhysicalPlanTest.java
@@ -35,6 +35,7 @@ import org.apache.iotdb.db.qp.physical.crud.QueryPlan;
import org.apache.iotdb.db.qp.physical.sys.AuthorPlan;
import org.apache.iotdb.db.qp.physical.sys.CreateTimeSeriesPlan;
import org.apache.iotdb.db.qp.physical.sys.DataAuthPlan;
+import org.apache.iotdb.db.qp.physical.sys.LoadConfigurationPlan;
import org.apache.iotdb.db.qp.physical.sys.PropertyPlan;
import org.apache.iotdb.db.qp.utils.MemIntQpExecutor;
import org.apache.iotdb.db.query.fill.LinearFill;
@@ -485,4 +486,12 @@ public class PhysicalPlanTest {
Assert.assertEquals(2, dataAuthPlan.getUsers().size());
Assert.assertEquals(OperatorType.REVOKE_WATERMARK_EMBEDDING, dataAuthPlan.getOperatorType());
}
+
+ @Test
+ public void testConfiguration() throws QueryProcessException, MetadataException {
+ String metadata = "load configuration";
+ QueryProcessor processor = new QueryProcessor(new MemIntQpExecutor());
+ LoadConfigurationPlan plan = (LoadConfigurationPlan) processor.parseSQLToPhysicalPlan(metadata);
+ assertEquals("LOAD_CONFIGURATION", plan.toString());
+ }
}