You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iotdb.apache.org by qi...@apache.org on 2020/04/07 04:01:10 UTC
[incubator-iotdb] branch master updated (7e8299d -> 3565f13)
This is an automated email from the ASF dual-hosted git repository.
qiaojialin pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-iotdb.git.
from 7e8299d modify website (#988)
new 41c85a4 remove tsfile-format.properties
new 6d80fc9 Merge remote-tracking branch 'origin/master'
new 3565f13 update delete doc
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
README.md | 2 +-
docs/SystemDesign/5-DataQuery/8-ModificationHandle.md | 18 ++++++++++++++----
docs/UserGuide/0-Get Started/1-QuickStart.md | 2 +-
docs/UserGuide/2-Concept/.~2-Data Type.md | 2 +-
docs/UserGuide/2-Concept/2-Data Type.md | 2 +-
docs/UserGuide/4-Client/5-Programming - TsFile API.md | 14 ++++++--------
.../SystemDesign/5-DataQuery/8-ModificationHandle.md | 18 ++++++++++++++----
docs/zh/UserGuide/0-Get Started/1-QuickStart.md | 2 +-
docs/zh/UserGuide/2-Concept/2-Data Type.md | 2 +-
.../UserGuide/4-Client/5-Programming - TsFile API.md | 14 ++++++--------
10 files changed, 46 insertions(+), 30 deletions(-)
[incubator-iotdb] 03/03: update delete doc
Posted by qi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
qiaojialin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-iotdb.git
commit 3565f133a4322e55911a964d16128959a6b69308
Author: qiaojialin <64...@qq.com>
AuthorDate: Tue Apr 7 12:00:59 2020 +0800
update delete doc
---
docs/SystemDesign/5-DataQuery/8-ModificationHandle.md | 18 ++++++++++++++----
.../SystemDesign/5-DataQuery/8-ModificationHandle.md | 18 ++++++++++++++----
2 files changed, 28 insertions(+), 8 deletions(-)
diff --git a/docs/SystemDesign/5-DataQuery/8-ModificationHandle.md b/docs/SystemDesign/5-DataQuery/8-ModificationHandle.md
index 1166250..2871332 100644
--- a/docs/SystemDesign/5-DataQuery/8-ModificationHandle.md
+++ b/docs/SystemDesign/5-DataQuery/8-ModificationHandle.md
@@ -21,25 +21,35 @@
# Modification handling in query
+Background:
+
Data deletion only record a mods file for disk data, the data is not really deleted. Therefore, we need to consider the modifications in query.
-Each timeseries is treated independently in query process. For each timeseries, there are 5 levels: TsFileResource -> TimeseriesMetadata -> ChunkMetadata -> IPageReader -> BatchData
+If a TsFile is influenced by deletion, a deletion operation will be recorded in its mods file. The log contains 3 parts: path, deleted time, version
+
+## Related class
+
+mods file: org.apache.iotdb.db.engine.modification.ModificationFile
-Query resource: TsFileResource and possibly exist mods file. If a TsFile is influenced by deletion, a modification log will be recorded in its mods file. The log contains 3 parts: path, deleted time, version
+deletion operation: org.apache.iotdb.db.engine.modification.Modification
+
+## Query process
+Each timeseries is treated independently in query process. For each timeseries, there are 5 levels: TsFileResource -> TimeseriesMetadata -> ChunkMetadata -> IPageReader -> BatchData
+
* TsFileResource -> TimeseriesMetadata
```
-// Set the statistics in TimeseriesMetadata unusable if the timeseries contains modifications
+// Set the statistics in TimeseriesMetadata unusable if the timeseries contains deletion operations
FileLoaderUtils.loadTimeseriesMetadata()
```
* TimeseriesMetadata -> List\<ChunkMetadata\>
```
-// For each ChunkMetadata, find the largest timestamp in all modifications whose version is larger than it. Set deleted time to ChunkMetadata.
+// For each ChunkMetadata, find the largest timestamp in all deletion operations whose version is larger than it. Set deleted time to ChunkMetadata.
// set the statistics in ChunkMetadata is unusable if it is affected by deletion
FileLoaderUtils.loadChunkMetadataList()
```
diff --git a/docs/zh/SystemDesign/5-DataQuery/8-ModificationHandle.md b/docs/zh/SystemDesign/5-DataQuery/8-ModificationHandle.md
index ce4ff7b..d3357ed 100644
--- a/docs/zh/SystemDesign/5-DataQuery/8-ModificationHandle.md
+++ b/docs/zh/SystemDesign/5-DataQuery/8-ModificationHandle.md
@@ -21,25 +21,35 @@
# 查询中的数据修改处理
+背景介绍：
+
数据删除操作对磁盘数据只记录了 mods 文件，并未真正执行删除逻辑，因此查询时需要考虑数据删除的逻辑。
-查询时每个时间序列会单独处理。针对一个时间序列，由大到小有 5 个层次：TsFileResource -> TimeseriesMetadata -> ChunkMetadata -> IPageReader -> BatchData
+如果一个文件中有数据被删除了，将删除操作记录到 mods 文件中。记录三列：删除的时间序列，删除范围的最大时间点，删除操作对应的版本。
+
+## 相关类
+
+mods 文件: org.apache.iotdb.db.engine.modification.ModificationFile
-查询资源：TsFileResource 以及可能存在的 mods 文件，如果一个文件中有数据被删除了，将删除操作记录到 mods 文件中。记录三列：删除的时间序列，删除范围的最大时间点，删除操作对应的版本。
+删除操作: org.apache.iotdb.db.engine.modification.Modification
+
+## 查询流程
+查询时每个时间序列会单独处理。针对一个时间序列，由大到小有 5 个层次：TsFileResource -> TimeseriesMetadata -> ChunkMetadata -> IPageReader -> BatchData
+
* TsFileResource -> TimeseriesMetadata
```
-// 只要这个时间序列有对应的 modification，就标记 TimeseriesMetadata 中的统计信息不可用
+// 只要这个时间序列有对应的删除操作，就标记 TimeseriesMetadata 中的统计信息不可用
FileLoaderUtils.loadTimeseriesMetadata()
```
* TimeseriesMetadata -> List\<ChunkMetadata\>
```
-// 对于每个 ChunkMetadata，找到比其 version 大的所有 modification 中最大时间戳, 设置到 ChunkMetadata 的 deleteAt 中，并标记 统计信息不可用
+// 对于每个 ChunkMetadata，找到比其 version 大的所有删除操作中最大时间戳, 设置到 ChunkMetadata 的 deleteAt 中，并标记 统计信息不可用
FileLoaderUtils.loadChunkMetadataList()
```
[incubator-iotdb] 02/03: Merge remote-tracking branch
'origin/master'
Posted by qi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
qiaojialin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-iotdb.git
commit 6d80fc9c1e898b40c7b3e554c91f93a5c7675f20
Merge: 41c85a4 7e8299d
Author: qiaojialin <64...@qq.com>
AuthorDate: Tue Apr 7 11:52:33 2020 +0800
Merge remote-tracking branch 'origin/master'
site/README-zh.md | 206 ++++++++++++++++++++++++++++++++++++++
site/README.md | 173 +++++++++++++++++++++++++++-----
site/src/main/.vuepress/config.js | 32 +++---
3 files changed, 368 insertions(+), 43 deletions(-)
[incubator-iotdb] 01/03: remove tsfile-format.properties
Posted by qi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
qiaojialin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-iotdb.git
commit 41c85a4db82d5b3ee7840da3bb2dcfca16d156e1
Author: qiaojialin <64...@qq.com>
AuthorDate: Mon Apr 6 17:31:45 2020 +0800
remove tsfile-format.properties
---
README.md | 2 +-
docs/UserGuide/0-Get Started/1-QuickStart.md | 2 +-
docs/UserGuide/2-Concept/.~2-Data Type.md | 2 +-
docs/UserGuide/2-Concept/2-Data Type.md | 2 +-
docs/UserGuide/4-Client/5-Programming - TsFile API.md | 14 ++++++--------
docs/zh/UserGuide/0-Get Started/1-QuickStart.md | 2 +-
docs/zh/UserGuide/2-Concept/2-Data Type.md | 2 +-
docs/zh/UserGuide/4-Client/5-Programming - TsFile API.md | 14 ++++++--------
8 files changed, 18 insertions(+), 22 deletions(-)
diff --git a/README.md b/README.md
index 9525144..e0eb609 100644
--- a/README.md
+++ b/README.md
@@ -130,7 +130,7 @@ Then the binary version (including both server and client) can be found at **dis
configuration files are under "conf" folder
* environment config module (`iotdb-env.bat`, `iotdb-env.sh`),
- * system config module (`tsfile-format.properties`, `iotdb-engine.properties`)
+ * system config module (`iotdb-engine.properties`)
* log config module (`logback.xml`).
For more information, please see [Chapter3: Server](https://iotdb.apache.org/#/Documents/progress/chap3/sec1) and [Chapter4: Client](https://iotdb.apache.org/#/Documents/progress/chap4/sec1) in detail.
diff --git a/docs/UserGuide/0-Get Started/1-QuickStart.md b/docs/UserGuide/0-Get Started/1-QuickStart.md
index 4786bde..af53929 100755
--- a/docs/UserGuide/0-Get Started/1-QuickStart.md
+++ b/docs/UserGuide/0-Get Started/1-QuickStart.md
@@ -69,7 +69,7 @@ You can download the binary file from:
configuration files are under "conf" folder
* environment config module (`iotdb-env.bat`, `iotdb-env.sh`),
- * system config module (`tsfile-format.properties`, `iotdb-engine.properties`)
+ * system config module (`iotdb-engine.properties`)
* log config module (`logback.xml`).
For more, see [Chapter3: Server](../3-Server/1-Download.html) in detail.
diff --git a/docs/UserGuide/2-Concept/.~2-Data Type.md b/docs/UserGuide/2-Concept/.~2-Data Type.md
index ebfece9..e7c0dbc 100644
--- a/docs/UserGuide/2-Concept/.~2-Data Type.md
+++ b/docs/UserGuide/2-Concept/.~2-Data Type.md
@@ -30,7 +30,7 @@ IoTDB supports six data types in total:
* TEXT (String).
-The time series of **FLOAT** and **DOUBLE** type can specify (MAX\_POINT\_NUMBER, see [this page](../5-Operation%20Manual/4-SQL%20Reference.html) for more information on how to specify), which is the number of digits after the decimal point of the floating point number, if the encoding method is [RLE](../2-Concept/3-Encoding.html) or [TS\_2DIFF](../2-Concept/3-Encoding.html) (Refer to [Create Timeseries Statement](../5-Operation%20Manual/4-SQL%20Reference.html) for more information on ho [...]
+The time series of **FLOAT** and **DOUBLE** type can specify (MAX\_POINT\_NUMBER, see [this page](../5-Operation%20Manual/4-SQL%20Reference.html) for more information on how to specify), which is the number of digits after the decimal point of the floating point number, if the encoding method is [RLE](../2-Concept/3-Encoding.html) or [TS\_2DIFF](../2-Concept/3-Encoding.html) (Refer to [Create Timeseries Statement](../5-Operation%20Manual/4-SQL%20Reference.html) for more information on ho [...]
* For Float data value, The data range is (-Integer.MAX_VALUE, Integer.MAX_VALUE), rather than Float.MAX_VALUE, and the max_point_number is 19, it is because of the limition of function Math.round(float) in Java.
* For Double data value, The data range is (-Long.MAX_VALUE, Long.MAX_VALUE), rather than Double.MAX_VALUE, and the max_point_number is 19, it is because of the limition of function Math.round(double) in Java (Long.MAX_VALUE=9.22E18).
diff --git a/docs/UserGuide/2-Concept/2-Data Type.md b/docs/UserGuide/2-Concept/2-Data Type.md
index ebfece9..e7c0dbc 100644
--- a/docs/UserGuide/2-Concept/2-Data Type.md
+++ b/docs/UserGuide/2-Concept/2-Data Type.md
@@ -30,7 +30,7 @@ IoTDB supports six data types in total:
* TEXT (String).
-The time series of **FLOAT** and **DOUBLE** type can specify (MAX\_POINT\_NUMBER, see [this page](../5-Operation%20Manual/4-SQL%20Reference.html) for more information on how to specify), which is the number of digits after the decimal point of the floating point number, if the encoding method is [RLE](../2-Concept/3-Encoding.html) or [TS\_2DIFF](../2-Concept/3-Encoding.html) (Refer to [Create Timeseries Statement](../5-Operation%20Manual/4-SQL%20Reference.html) for more information on ho [...]
+The time series of **FLOAT** and **DOUBLE** type can specify (MAX\_POINT\_NUMBER, see [this page](../5-Operation%20Manual/4-SQL%20Reference.html) for more information on how to specify), which is the number of digits after the decimal point of the floating point number, if the encoding method is [RLE](../2-Concept/3-Encoding.html) or [TS\_2DIFF](../2-Concept/3-Encoding.html) (Refer to [Create Timeseries Statement](../5-Operation%20Manual/4-SQL%20Reference.html) for more information on ho [...]
* For Float data value, The data range is (-Integer.MAX_VALUE, Integer.MAX_VALUE), rather than Float.MAX_VALUE, and the max_point_number is 19, it is because of the limition of function Math.round(float) in Java.
* For Double data value, The data range is (-Long.MAX_VALUE, Long.MAX_VALUE), rather than Double.MAX_VALUE, and the max_point_number is 19, it is because of the limition of function Math.round(double) in Java (Long.MAX_VALUE=9.22E18).
diff --git a/docs/UserGuide/4-Client/5-Programming - TsFile API.md b/docs/UserGuide/4-Client/5-Programming - TsFile API.md
index 27cb396..9aaad0f 100644
--- a/docs/UserGuide/4-Client/5-Programming - TsFile API.md
+++ b/docs/UserGuide/4-Client/5-Programming - TsFile API.md
@@ -687,15 +687,11 @@ public class TsFileRead {
```
-## User-specified config file path
+## Change TsFile Configuration
-Default config file `tsfile-format.properties.template` is located at `/tsfile/src/main/resources` directory. If you want to use your own path, you can:
-```
-System.setProperty(TsFileConstant.TSFILE_CONF, "your config file path");
-```
-and then call:
```
TSFileConfig config = TSFileDescriptor.getInstance().getConfig();
+config.setXXX();
```
## Bloom filter
@@ -704,9 +700,11 @@ Bloom filter checks whether a given time series is in the tsfile before loading
If you want to learn more about its mechanism, you can refer to: [wiki page of bloom filter](https://en.wikipedia.org/wiki/Bloom_filter).
#### configuration
-you can control the false positive rate of bloom filter by the following parameter in the config file `tsfile-format.properties` which located at `/server/src/assembly/resources/conf` directory
+
+you can control the false positive rate of bloom filter by the following parameter in the config
+
```
# The acceptable error rate of bloom filter, should be in [0.01, 0.1], default is 0.05
-bloom_filter_error_rate=0.05
+bloomFilterErrorRate=0.05
```
diff --git a/docs/zh/UserGuide/0-Get Started/1-QuickStart.md b/docs/zh/UserGuide/0-Get Started/1-QuickStart.md
index c779980..134c21a 100755
--- a/docs/zh/UserGuide/0-Get Started/1-QuickStart.md
+++ b/docs/zh/UserGuide/0-Get Started/1-QuickStart.md
@@ -68,7 +68,7 @@ IoTDB支持多种安装途径。用户可以使用三种方式对IoTDB进行安
配置文件在"conf"文件夹下，包括：
* 环境配置模块 (`iotdb-env.bat`, `iotdb-env.sh`),
- * 系统配置模块 (`tsfile-format.properties`, `iotdb-engine.properties`)
+ * 系统配置模块 (`iotdb-engine.properties`)
* 日志配置模块 (`logback.xml`).
想要了解更多，请浏览[Chapter3: Server](../3-Server/1-Download.html)
diff --git a/docs/zh/UserGuide/2-Concept/2-Data Type.md b/docs/zh/UserGuide/2-Concept/2-Data Type.md
index b343dfc..928b00c 100644
--- a/docs/zh/UserGuide/2-Concept/2-Data Type.md
+++ b/docs/zh/UserGuide/2-Concept/2-Data Type.md
@@ -31,7 +31,7 @@ IoTDB支持:
一共六种数据类型。
-其中**FLOAT**与**DOUBLE**类型的序列，如果编码方式采用[RLE](../2-Concept/3-Encoding.html)或[TS_2DIFF](../2-Concept/3-Encoding.html)可以指定MAX_POINT_NUMBER，该项为浮点数的小数点后位数，具体指定方式请参见本文[第5.4节](../5-Operation%20Manual/4-SQL%20Reference.html)，若不指定则系统会根据配置文件`tsfile-format.properties`文件中的[float_precision项](../3-Server/4-Config%20Manual.html)配置。
+其中**FLOAT**与**DOUBLE**类型的序列，如果编码方式采用[RLE](../2-Concept/3-Encoding.html)或[TS_2DIFF](../2-Concept/3-Encoding.html)可以指定MAX_POINT_NUMBER，该项为浮点数的小数点后位数，具体指定方式请参见本文[第5.4节](../5-Operation%20Manual/4-SQL%20Reference.html)，若不指定则系统会根据配置文件`iotdb-engine.properties`文件中的[float_precision项](../3-Server/4-Config%20Manual.html)配置。
当系统中用户输入的数据类型与该时间序列的数据类型不对应时，系统会提醒类型错误，如下所示，二阶差分不支持布尔类型的编码：
diff --git a/docs/zh/UserGuide/4-Client/5-Programming - TsFile API.md b/docs/zh/UserGuide/4-Client/5-Programming - TsFile API.md
index e5be619..b2fb0c3 100644
--- a/docs/zh/UserGuide/4-Client/5-Programming - TsFile API.md
+++ b/docs/zh/UserGuide/4-Client/5-Programming - TsFile API.md
@@ -681,15 +681,11 @@ public class TsFileRead {
```
-## 指定配置文件路径
+## 修改 TsFile 配置项
-默认的配置文件`tsfile-format.properties.template`存放在`/tsfile/src/main/resources`目录下。如果您想使用自定义的路径:
-```
-System.setProperty(TsFileConstant.TSFILE_CONF, "your config file path");
-```
-然后调用:
```
TSFileConfig config = TSFileDescriptor.getInstance().getConfig();
+config.setXXX();
```
## Bloom filter
@@ -698,8 +694,10 @@ TSFileConfig config = TSFileDescriptor.getInstance().getConfig();
如果你想了解更多关于它的细节，你可以参考: [wiki page of bloom filter](https://en.wikipedia.org/wiki/Bloom_filter).
#### 配置
-您可以通过`/server/src/assembly/resources/conf`目录下的`tsfile-format.properties`配置文件中的以下参数来控制bloom过滤器的误报率：
+
+您可以通过修改 TsFileConfig 中的配置项来控制bloom过滤器的误报率：
+
```
# The acceptable error rate of bloom filter, should be in [0.01, 0.1], default is 0.05
-bloom_filter_error_rate=0.05
+bloomFilterErrorRate=0.05
```