You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iotdb.apache.org by ha...@apache.org on 2021/06/11 09:47:57 UTC
[iotdb] branch rel/0.11 updated: [To rel/0.12] add TsFile Vis Tool
(#3330)
This is an automated email from the ASF dual-hosted git repository.
haonan pushed a commit to branch rel/0.11
in repository https://gitbox.apache.org/repos/asf/iotdb.git
The following commit(s) were added to refs/heads/rel/0.11 by this push:
new 86acb8a [To rel/0.12] add TsFile Vis Tool (#3330)
86acb8a is described below
commit 86acb8a6c1406b51d477babaa067678f3b72b771
Author: Rui,Lei <33...@users.noreply.github.com>
AuthorDate: Fri Jun 11 17:47:40 2021 +0800
[To rel/0.12] add TsFile Vis Tool (#3330)
---
docs/SystemDesign/TsFile/Format.md | 346 ++++++++++++++++-----
docs/zh/SystemDesign/TsFile/Format.md | 246 +++++++++++++--
.../tools/tsfileToolSet/print-tsfile-visdata.bat | 62 ++++
.../tools/tsfileToolSet/print-tsfile-visdata.sh | 48 +++
.../iotdb/db/tools/vis/TsFileExtractVisdata.java | 182 +++++++++++
.../main/java/org/apache/iotdb/db/tools/vis/vis.m | 288 +++++++++++++++++
6 files changed, 1080 insertions(+), 92 deletions(-)
diff --git a/docs/SystemDesign/TsFile/Format.md b/docs/SystemDesign/TsFile/Format.md
index c08b917..1f4e4f8 100644
--- a/docs/SystemDesign/TsFile/Format.md
+++ b/docs/SystemDesign/TsFile/Format.md
@@ -21,48 +21,58 @@
# TsFile Format
+Overview:
+1. TsFile Design
+2. TsFile Visualization Examples
+3. TsFile Tool Set
+ - IoTDB Data Directory Overview Tool
+ - TsFileResource Print Tool
+ - TsFile Sketch Tool
+ - TsFileSequenceRead
+ - Vis Tool
+
## 1. TsFile Design
- This is an introduction to the design details of TsFile.
+This is an introduction to the design details of TsFile.
### 1.1 Variable Storage
- **Big Endian**
-
- - For Example, the `int` `0x8` will be stored as `00 00 00 08`, not `08 00 00 00`
+
+ - For Example, the `int` `0x8` will be stored as `00 00 00 08`, not `08 00 00 00`
- **String with Variable Length**
- - The format is `int size` plus `String literal`. Size can be zero.
- - Size equals the number of bytes this string will take, and it may not equal to the length of the string.
- - For example "sensor_1" will be stored as `00 00 00 08` plus the encoding(ASCII) of "sensor_1".
- - Note that for the file signature "TsFile000001" (`MAGIC STRING` + `Version Number`), the size(12) and encoding(ASCII)
- is fixed so there is no need to put the size before this string literal.
+ - The format is `int size` plus `String literal`. Size can be zero.
+ - Size equals the number of bytes this string will take, and it may not equal to the length of the string.
+ - For example "sensor_1" will be stored as `00 00 00 08` plus the encoding(ASCII) of "sensor_1".
+ - Note that for the file signature "TsFile000001" (`MAGIC STRING` + `Version Number`), the size(12) and encoding(ASCII)
+ is fixed so there is no need to put the size before this string literal.
- **Data Type Hardcode**
- - 0: BOOLEAN
- - 1: INT32 (`int`)
- - 2: INT64 (`long`)
- - 3: FLOAT
- - 4: DOUBLE
- - 5: TEXT (`String`)
+ - 0: BOOLEAN
+ - 1: INT32 (`int`)
+ - 2: INT64 (`long`)
+ - 3: FLOAT
+ - 4: DOUBLE
+ - 5: TEXT (`String`)
- **Encoding Type Hardcode**
- - 0: PLAIN
- - 1: PLAIN_DICTIONARY
- - 2: RLE
- - 3: DIFF
- - 4: TS_2DIFF
- - 5: BITMAP
- - 6: GORILLA_V1
- - 7: REGULAR
- - 8: GORILLA
+ - 0: PLAIN
+ - 1: PLAIN_DICTIONARY
+ - 2: RLE
+ - 3: DIFF
+ - 4: TS_2DIFF
+ - 5: BITMAP
+ - 6: GORILLA_V1
+ - 7: REGULAR
+ - 8: GORILLA
- **Compressing Type Hardcode**
- - 0: UNCOMPRESSED
- - 1: SNAPPY
- - 7: LZ4
+ - 0: UNCOMPRESSED
+ - 1: SNAPPY
+ - 7: LZ4
- **TsDigest Statistics Type Hardcode**
- - 0: min_value
- - 1: max_value
- - 2: first_value
- - 3: last_value
- - 4: sum_value
+ - 0: min_value
+ - 1: max_value
+ - 2: first_value
+ - 3: last_value
+ - 4: sum_value
### 1.2 TsFile Overview
@@ -82,7 +92,7 @@ Query Process：e.g., read d1.s1
* deserialize TsFileMetadata，get the position of TimeseriesMetadata of d1.s1
* deserialize and get the TimeseriesMetadata of d1.s1
-* according to TimeseriesMetadata of d1.s1，deserialize all ChunkMetadata of d1.s1
+* according to TimeseriesMetadata of d1.s1，deserialize all ChunkMetadata of d1.s1
* according to each ChunkMetadata of d1.s1，read its Chunk
#### 1.2.1 Magic String and Version Number
@@ -128,17 +138,17 @@ PageHeader Structure
Here is the detailed information for `statistics`:
- | Member | Description | DoubleStatistics | FloatStatistics | IntegerStatistics | LongStatistics | BinaryStatistics | BooleanStatistics |
+| Member | Description | DoubleStatistics | FloatStatistics | IntegerStatistics | LongStatistics | BinaryStatistics | BooleanStatistics |
| :----------------------------------: | :--------------: | :----: | :----: | :----: | :----: | :----: | :----: |
- | count | number of time-value points | long | long | long | long | long | long |
- | startTime | start time | long | long | long | long | long | long |
- | endTime | end time | long | long | long | long | long | long |
- | minValue | min value | double | float | int | long | - | - |
- | maxValue | max value | double | float | int | long | - | - |
- | firstValue | first value | double | float | int | long | Binary | boolean|
- | lastValue | last value | double | float | int | long | Binary | boolean|
- | sumValue | sum value | double | double | double | double | - | - |
-
+| count | number of time-value points | long | long | long | long | long | long |
+| startTime | start time | long | long | long | long | long | long |
+| endTime | end time | long | long | long | long | long | long |
+| minValue | min value | double | float | int | long | - | - |
+| maxValue | max value | double | float | int | long | - | - |
+| firstValue | first value | double | float | int | long | Binary | boolean|
+| lastValue | last value | double | float | int | long | Binary | boolean|
+| sumValue | sum value | double | double | double | double | - | - |
+
##### ChunkGroupFooter
| Member | Type | Description |
@@ -151,7 +161,7 @@ Here is the detailed information for `statistics`:
##### 1.2.3.1 ChunkMetadata
-The first part of metadata is `ChunkMetadata`
+The first part of metadata is `ChunkMetadata`
| Member | Type | Description |
| :------------------------------------------------: | :------: | :----: |
@@ -236,9 +246,26 @@ A TsFile ends with a 6-byte magic string (`TsFile`).
Congratulations! You have finished the journey of discovering TsFile.
-### 1.3 TsFile Tool Set
-#### 1.3.1 IoTDB Data Directory Overview Tool
+## 2. TsFile Visualization Examples
+
+### v0.8
+
+<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/33376433/65209576-2bd36000-dacb-11e9-9e43-49e0dd01274e.png">
+
+### v0.9 / 000001
+
+<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/33376433/69341240-26012300-0ca4-11ea-91a1-d516810cad44.png">
+
+### v0.10 / 000002
+
+<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/19167280/95296983-492cc500-08ac-11eb-9f66-c9c78401c61d.png">
+
+
+
+## 3. TsFile Tool Set
+
+### 3.1 IoTDB Data Directory Overview Tool
After building the server, the startup script of this tool will appear under the `server\target\iotdb-server-0.11.1\tools\tsfileToolSet` directory.
@@ -260,9 +287,10 @@ An example on Windows:
```
D:\iotdb\server\target\iotdb-server-0.11.1-SNAPSHOT\tools\tsfileToolSet>.\print-iotdb-data-dir.bat D:\\data\data
-````````````````````````
+```
+
Starting Printing the IoTDB Data Directory Overview
-````````````````````````
+```
output save path:IoTDB_data_dir_overview.txt
TsFile data dir num:1
21:17:38.841 [main] WARN org.apache.iotdb.tsfile.common.conf.TSFileDescriptor - Failed to find config file iotdb-engine.properties at classpath, use default configuration
@@ -287,9 +315,7 @@ TsFile data dir num:1
|==============================================================
```
-
-
-#### 1.3.2 TsFileResource Print Tool
+### 3.2 TsFileResource Print Tool
After building the server, the startup script of this tool will appear under the `server\target\iotdb-server-0.11.1\tools\tsfileToolSet` directory.
@@ -298,29 +324,30 @@ Command:
For Windows:
```
-.\print-tsfile-sketch.bat <path of your TsFileResource directory>
+.\print-tsfile-resource-files.bat <path of your TsFileResource directory>
```
For Linux or MacOs:
```
-./print-tsfile-sketch.sh <path of your TsFileResource directory>
+./print-tsfile-resource-files.sh <path of your TsFileResource directory>
```
An example on Windows:
```
D:\iotdb\server\target\iotdb-server-0.11.1\tools\tsfileToolSet>.\print-tsfile-resource-files.bat D:\data\data\sequence\root.vehicle
-````````````````````````
+```
+
Starting Printing the TsFileResources
-````````````````````````
+```
12:31:59.861 [main] WARN org.apache.iotdb.db.conf.IoTDBDescriptor - Cannot find IOTDB_HOME or IOTDB_CONF environment variable when loading config file iotdb-engine.properties, use default configuration
analyzing D:\data\data\sequence\root.vehicle\1572496142067-101-0.tsfile ...
device root.vehicle.d0, start time 3000 (1970-01-01T08:00:03+08:00[GMT+08:00]), end time 100999 (1970-01-01T08:01:40.999+08:00[GMT+08:00])
analyzing the resource file finished.
-````````````````````````
+```
-#### 1.3.3 TsFile Sketch Tool
+### 3.3 TsFile Sketch Tool
After building the server, the startup script of this tool will appear under the `server\target\iotdb-server-0.11.1\tools\tsfileToolSet` directory.
@@ -332,7 +359,7 @@ For Windows:
.\print-tsfile-sketch.bat <path of your TsFile> (<path of the file for saving the output result>)
```
-- Note that if `<path of the file for saving the output result>` is not set, the default path "TsFile_sketch_view.txt" will be used.
+- Note that if `<path of the file for saving the output result>` is not set, the default path "TsFile_sketch_view.txt" will be used.
For Linux or MacOs:
@@ -340,15 +367,16 @@ For Linux or MacOs:
./print-tsfile-sketch.sh <path of your TsFile> (<path of the file for saving the output result>)
```
-- Note that if `<path of the file for saving the output result>` is not set, the default path "TsFile_sketch_view.txt" will be used.
+- Note that if `<path of the file for saving the output result>` is not set, the default path "TsFile_sketch_view.txt" will be used.
An example on macOS:
-```shell
+```
/iotdb/server/target/iotdb-server-0.11.1/tools/tsfileToolSet$ ./print-tsfile-sketch.sh test.tsfile
-````````````````````````
+```
+
Starting Printing the TsFile Sketch
-````````````````````````
+```
TsFile path:test.tsfile
Sketch save path:TsFile_sketch_view.txt
-------------------------------- TsFile Sketch --------------------------------
@@ -560,22 +588,200 @@ file length: 33436
---------------------------------- TsFile Sketch End ----------------------------------
-````````````````````````
+```
-#### 1.3.4 TsFileSequenceRead
+### 3.4 TsFileSequenceRead
You can also use `example/tsfile/org/apache/iotdb/tsfile/TsFileSequenceRead` to sequentially print a TsFile's content.
-### 1.4 A TsFile Visualization Example
+### 3.5 Vis Tool
-#### v0.8
+Vis is a tool that visualizes the time layouts and cout aggregation of chunk data in TsFiles. You can use this tool to facilitate debugging, check the distribution of data, etc. Please feel free to play around with it, and let us know your thoughts.
-<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/33376433/65209576-2bd36000-dacb-11e9-9e43-49e0dd01274e.png">
+
-#### v0.9 / 000001
+- A single long narrow rectangle in the figure shows the visdata of a single chunk in a TsFile.
+Visdata contains \[tsName, fileName, versionNum, startTime, endTime, countNum\].
+- The position of a rectangle on the x-axis is defined by the startTime and endTime of the chunk data.
+- The position of a rectangle on the y-axis is defined simultaneously by
+ - (a)`showSpecific`: the specific set of time series to be plotted;
+ - (b) seqKey/unseqKey display policies: extract seqKey or unseqKey from statisfied keys under
+ different display policies:
+ - b-1) unseqKey identifies tsName and fileName, so chunk data with the same fileName and
+ tsName but different versionNums are plotted in the same line.
+ - b-2) seqKey identifies tsName, so chunk data with the same tsName but different fileNames
+ and versionNums are ploteed in the same line;
+ - (c)`isFileOrder`: sort seqKey&unseqKey according to `isFileOrder`, true to sort
+ seqKeys&unseqKeys by fileName priority, false to sort seqKeys&unseqKeys by tsName priority.
+ When multiple time series are displayed on a graph at the same time, this parameter can provide
+ users with these two observation perspectives.
-<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/33376433/69341240-26012300-0ca4-11ea-91a1-d516810cad44.png">
+#### 3.5.1 How to run Vis
+
+The source code contains two files: `TsFileExtractVisdata.java` and `vis.m`. `TsFileExtractVisdata.java` extracts, from input tsfiles, necessary visualization information, which is what `vis.m` needs to plot figures.
+
+Simply put, you first run `TsFileExtractVisdata.java` and then run `vis.m`.
+
+##### Step 1: run TsFileExtractVisdata.java
+
+`TsFileExtractVisdata.java` extracts visdata [tsName, fileName, versionNum, startTime, endTime, countNum] from every chunk of the input TsFiles and write them to the specified output path.
+
+After building the server, the startup script of this tool will appear under the `server\target\iotdb-server-0.11.1\tools\tsfileToolSet` directory.
+
+Command:
+
+For Windows:
+
+```
+.\print-tsfile-visdata.bat path1 seqIndicator1 path2 seqIndicator2 ... pathN seqIndicatorN outputPath
+```
+
+For Linux or MacOs:
+
+```
+./print-tsfile-visdata.sh path1 seqIndicator1 path2 seqIndicator2 ... pathN seqIndicatorN outputPath
+```
+
+Args: [`path1` `seqIndicator1` `path2` `seqIndicator2` ... `pathN` `seqIndicatorN` `outputPath`]
+
+Details:
+
+- 2N+1 args in total.
+- `seqIndicator` should be 'true' or 'false' (not case sensitive). 'true' means is the file is sequence, 'false' means the file is unsequence.
+- `Path` can be the full path of a tsfile or a directory path. If it is a directory path, make sure that all tsfiles in this directory have the same `seqIndicator`.
+
+##### Step 2: run vis.m
+
+`vis.m` load visdata generated by `TsFileExtractVisdata`, and then plot figures given the loaded visdata and two plot parameters: `showSpecific` and `isFileOrder`.
+
+```matlab
+function [timeMap,countMap] = loadVisData(filePath,timestampUnit)
+% Load visdata generated by TsFileExtractVisdata.
+%
+% filePath: the path of visdata.
+% The format is [tsName,fileName,versionNum,startTime,endTime,countNum].
+% `tsName` and `fileName` are string, the others are long value.
+% If the tsfile is unsequence file, `fileName` will contain "unseq" as an
+% indicator, which is guaranteed by TsFileExtractVisdata.
+%
+% timestampUnit(not case sensitive):
+% 'us' if the timestamp is microsecond, e.g., 1621993620816000
+% 'ms' if it is millisecond, e.g., 1621993620816
+% 's' if it is second, e.g., 1621993620
+%
+% timeMap: record the time range of every chunk.
+% Key [tsName][fileName][version] identifies the only chunk. Value is
+% [startTime,endTime] of the chunk.
+%
+% countMap: record the point count number of every chunk. Key is the same
+% as that of timeMap. Value is countNum.
+```
+
+```matlab
+function draw(timeMap,countMap,showSpecific,isFileOrder)
+% Plot figures given the loaded data and two plot parameters:
+% `showSpecific` and `isFileOrder`.
+%
+% process: 1) traverse `keys(timeMap)` to get the position arrangements on
+% the y axis dynamically, which is defined simultaneously by
+% (a)`showSpecific`: traverse `keys(timeMap)`, filter out keys
+% that don't statisfy `showSpecific`.
+% (b) seqKey/unseqKey display policies: extract seqKey or unseqKey
+% from statisfied keys under different display policies:
+% b-1) unseqKey identifies tsName and fileName, so data with the
+% same fileName and tsName but different versionNums are
+% plotted in the same line.
+% b-2) seqKey identifies tsName, so data with the same tsName but
+% different fileNames and versionNums are ploteed in the same
+% line.
+% (c)`isFileOrder`: sort seqKey&unseqKey according to `isFileOrder`,
+% finally get the position arrangements on the y axis.
+% 2) traverse `keys(timeMap)` again, get startTime&endTime from
+% `treeMap` as positions on the x axis, combined with the
+% positions on the y axis from the last step, finish plot.
+%
+% timeMap,countMap: generated by loadVisData function.
+%
+% showSpecific: the specific set of time series to be plotted.
+% If showSpecific is empty{}, then all loaded time series
+% will be plotted.
+% Note: Wildcard matching is not supported now. In other
+% words, showSpecific only support full time series path
+% names.
+%
+% isFileOrder: true to sort seqKeys&unseqKeys by fileName priority, false
+% to sort seqKeys&unseqKeys by tsName priority.
+```
+
+
+
+#### 3.5.2 Examples
+
+##### Example 1
+
+Use the tsfiles written by `IoTDBLargeDataIT.insertData` with a little modification: add `statement.execute("flush");` at the end of `IoTDBLargeDataIT.insertData`.
+
+Step 1: run `TsFileExtractVisdata.java`
+
+```
+.\print-tsfile-visdata.bat D:\rel0.11\iotdb\server\target\data\sequence true D:\rel0.11\iotdb\server\target\data\unsequence false D:\visdata1.csv
+```
+or equivalently:
+```
+.\print-tsfile-visdata.bat D:\rel0.11\iotdb\server\target\data\sequence\root.vehicle\0\1622743492580-1-0.tsfile true D:\rel0.11\iotdb\server\target\data\sequence\root.vehicle\0\1622743505092-2-0.tsfile true D:\rel0.11\iotdb\server\target\data\sequence\root.vehicle\0\1622743505573-3-0.tsfile true D:\rel0.11\iotdb\server\target\data\unsequence\root.vehicle\0\1622743505901-4-0.tsfile false D:\visdata1.csv
+```
+
+Step 2: run `vis.m`
+
+```matlab
+clear all;close all;
+
+% 1. load visdata generated by TsFileExtractDataToVisTool
+filePath = 'D:\visdata1.csv';
+[timeMap,countMap] = loadVisData(filePath,'ms'); % mind the timestamp unit
+
+% 2. plot figures given the loaded data and two plot parameters:
+% `showSpecific` and `isFileOrder`
+draw(timeMap,countMap,{},false)
+title("draw(timeMap,countMap,\{\},false)")
+
+draw(timeMap,countMap,{},true)
+title("draw(timeMap,countMap,\{\},true)")
+
+draw(timeMap,countMap,{'root.vehicle.d0.s0'},false)
+title("draw(timeMap,countMap,{'root.vehicle.d0.s0'},false)")
+
+draw(timeMap,countMap,{'root.vehicle.d0.s0','root.vehicle.d0.s1'},false)
+title("draw(timeMap,countMap,{'root.vehicle.d0.s0','root.vehicle.d0.s1'},false)")
+
+draw(timeMap,countMap,{'root.vehicle.d0.s0','root.vehicle.d0.s1'},true)
+title("draw(timeMap,countMap,{'root.vehicle.d0.s0','root.vehicle.d0.s1'},true)")
+```
+
+Plot results:
+
+
+
+
+
+
+
+
+
+
+
+
+
+##### Example 2
+
+As another example, this is from real-world.
+
+
+
+
+
+
+
+
-#### v0.10 / 000002
-<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/19167280/95296983-492cc500-08ac-11eb-9f66-c9c78401c61d.png">
diff --git a/docs/zh/SystemDesign/TsFile/Format.md b/docs/zh/SystemDesign/TsFile/Format.md
index 300d302..97f9b57 100644
--- a/docs/zh/SystemDesign/TsFile/Format.md
+++ b/docs/zh/SystemDesign/TsFile/Format.md
@@ -21,6 +21,15 @@
# TsFile 文件格式
+概览：
+1. TsFile Design
+2. TsFile Visualization Examples
+3. TsFile Tool Set
+ - IoTDB Data Directory Overview Tool
+ - TsFileResource Print Tool
+ - TsFile Sketch Tool
+ - TsFileSequenceRead
+ - Vis Tool
## 1. TsFile 设计
@@ -236,9 +245,24 @@ TsFile 是以6个字节的magic string (`TsFile`) 作为结束.
恭喜您, 至此您已经完成了 TsFile 的探秘之旅，祝您玩儿的开心!
-### 1.3 TsFile工具集
+## 2. TsFile 的总览图
-#### 1.3.1 IoTDB Data Directory 快速概览工具
+### v0.8
+
+<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/33376433/65209576-2bd36000-dacb-11e9-9e43-49e0dd01274e.png">
+
+### v0.9 / 000001
+
+<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/33376433/69341240-26012300-0ca4-11ea-91a1-d516810cad44.png">
+
+### v0.10 / 000002
+
+<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/19167280/95296983-492cc500-08ac-11eb-9f66-c9c78401c61d.png">
+
+
+## 3. TsFile工具集
+
+### 3.1 IoTDB Data Directory 快速概览工具
该工具的启动脚本会在编译 server 之后生成至 `server\target\iotdb-server-0.11.1\tools\tsfileToolSet` 目录中。
@@ -260,9 +284,10 @@ For Linux or MacOs:
```
D:\iotdb\server\target\iotdb-server-0.11.1-SNAPSHOT\tools\tsfileToolSet>.\print-iotdb-data-dir.bat D:\\data\data
-````````````````````````
+```
+
Starting Printing the IoTDB Data Directory Overview
-````````````````````````
+```
output save path:IoTDB_data_dir_overview.txt
TsFile data dir num:1
21:17:38.841 [main] WARN org.apache.iotdb.tsfile.common.conf.TSFileDescriptor - Failed to find config file iotdb-engine.properties at classpath, use default configuration
@@ -285,9 +310,9 @@ TsFile data dir num:1
| | |--1575813521063-105-0.tsfile.resource
| | | |--device root.ln.wf01.wt01, start time 10 (1970-01-01T08:00:00.010+08:00[GMT+08:00]), end time 50 (1970-01-01T08:00:00.050+08:00[GMT+08:00])
|==============================================================
-````````````````````````
+```
-#### 1.3.2 TsFileResource 打印工具
+### 3.2 TsFileResource 打印工具
该工具的启动脚本会在编译 server 之后生成至 `server\target\iotdb-server-0.11.1\tools\tsfileToolSet` 目录中。
@@ -309,16 +334,17 @@ Linux or MacOs:
```
D:\iotdb\server\target\iotdb-server-0.11.1\tools\tsfileToolSet>.\print-tsfile-resource-files.bat D:\data\data\sequence\root.vehicle
-````````````````````````
+```
+
Starting Printing the TsFileResources
-````````````````````````
+```
12:31:59.861 [main] WARN org.apache.iotdb.db.conf.IoTDBDescriptor - Cannot find IOTDB_HOME or IOTDB_CONF environment variable when loading config file iotdb-engine.properties, use default configuration
analyzing D:\data\data\sequence\root.vehicle\1572496142067-101-0.tsfile ...
device root.vehicle.d0, start time 3000 (1970-01-01T08:00:03+08:00[GMT+08:00]), end time 100999 (1970-01-01T08:01:40.999+08:00[GMT+08:00])
analyzing the resource file finished.
-````````````````````````
+```
-#### 1.3.3 TsFile 描述工具
+### 3.3 TsFile 描述工具
该工具的启动脚本会在编译 server 之后生成至 `server\target\iotdb-server-0.11.1\tools\tsfileToolSet` 目录中。
@@ -342,11 +368,12 @@ Linux or MacOs:
在mac系统中的示例:
-```shell
+```
/iotdb/server/target/iotdb-server-0.11.1/tools/tsfileToolSet$ ./print-tsfile-sketch.sh test.tsfile
-````````````````````````
+```
+
Starting Printing the TsFile Sketch
-````````````````````````
+```
TsFile path:test.tsfile
Sketch save path:TsFile_sketch_view.txt
-------------------------------- TsFile Sketch --------------------------------
@@ -557,22 +584,197 @@ file length: 33436
33436| END of TsFile
---------------------------------- TsFile Sketch End ----------------------------------
-````````````````````````
+```
-#### 1.3.4 TsFileSequenceRead
+### 3.4 TsFileSequenceRead
您可以使用示例中的类 `example/tsfile/org/apache/iotdb/tsfile/TsFileSequenceRead` 顺序打印 TsFile 中的内容.
-### 1.4 TsFile 的总览图
+### 3.5 Vis Tool
-#### v0.8
+Vis是一个把TsFiles中的chunk数据的时间分布以及点数可视化的工具。你可以使用这个工具来帮助你debug，还可以用来观察数据分布等等。
+欢迎使用这个工具，在社区里交流你的想法。
-<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/33376433/65209576-2bd36000-dacb-11e9-9e43-49e0dd01274e.png">
+
-#### v0.9 / 000001
+- 图中的一个窄长矩形代表的是一个TsFile里的一个chunk，其可视化信息为\[tsName, fileName, versionNum, startTime, endTime, countNum\]。
+- 矩形在x轴上的位置是由该chunk的startTime和endTime决定的。
+- 矩形在y轴上的位置是由以下三项共同决定的：
+ - (a)`showSpecific`: 用户指定要显示的特定时间序列集合。
+ - (b) seqKey/unseqKey显示规则: 从满足特定时间序列集合的keys提取seqKey或unseqKey时采取不同的显示规则：
+ - b-1) unseqKey识别tsName和fileName，因此有相同tsName和fileName但是不同versionNums的chunk数据将绘制在同一行；
+ - b-2) seqKey识别tsName，因此有相同tsName但是不同fileNames和versionNums的chunk数据将绘制在同一行，
+ - (c)`isFileOrder`：根据`isFileOrder`对seqKey&unseqKey进行排序，true则以fileName优先的顺序排序，
+ false则以tsName优先的顺序排序。当在一张图上同时显示多条时间序列时，该参数将给用户提供这两种可选的观察视角。
-<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/33376433/69341240-26012300-0ca4-11ea-91a1-d516810cad44.png">
+#### 3.5.1 如何运行Vis
-#### v0.10 / 000002
+源数据包含两个文件：`TsFileExtractVisdata.java`和`vis.m`。
+`TsFileExtractVisdata.java`从输入的tsfile文件中提取必要的可视化信息，`vis.m`用这些信息来完成作图。
-<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/19167280/95296983-492cc500-08ac-11eb-9f66-c9c78401c61d.png">
+简单说就是：先运行`TsFileExtractVisdata.java`然后再运行`vis.m`。
+
+##### 第一步：运行TsFileExtractVisdata.java
+
+`TsFileExtractVisdata.java`对输入的tsfiles的每一个chunk提取可视化信息[tsName, fileName, versionNum, startTime, endTime, countNum]，
+并把这些信息保存到指定的输出文件里。
+
+该工具的启动脚本会在编译 server 之后生成至 `server\target\iotdb-server-0.11.1\tools\tsfileToolSet` 目录中。
+
+使用方式:
+
+Windows:
+
+```
+.\print-tsfile-visdata.bat path1 seqIndicator1 path2 seqIndicator2 ... pathN seqIndicatorN outputPath
+```
+
+Linux or MacOs:
+
+```
+./print-tsfile-visdata.sh path1 seqIndicator1 path2 seqIndicator2 ... pathN seqIndicatorN outputPath
+```
+
+参数: [`path1` `seqIndicator1` `path2` `seqIndicator2` ... `pathN` `seqIndicatorN` `outputPath`]
+
+细节:
+
+- 一共2N+1个参数。
+- `seqIndicator`：'true'或者'false' （大小写不敏感）. 'true'表示是顺序文件, 'false'表示是乱序文件。
+- `Path`：可以是一个tsfile的全路径，也可以是一个文件夹路径。如果是文件夹路径，你需要确保这个文件夹下的所有tsfile文件的`seqIndicator`都是一样的。
+
+##### 第二步：运行vis.m
+
+`vis.m`把`TsFileExtractVisdata`生成的visdata加载进来，然后基于visdata以及两个用户绘图参数`showSpecific`和`isFileOrder`来完成作图。
+
+```matlab
+function [timeMap,countMap] = loadVisData(filePath,timestampUnit)
+% Load visdata generated by TsFileExtractVisdata.
+%
+% filePath: the path of visdata.
+% The format is [tsName,fileName,versionNum,startTime,endTime,countNum].
+% `tsName` and `fileName` are string, the others are long value.
+% If the tsfile is unsequence file, `fileName` will contain "unseq" as an
+% indicator, which is guaranteed by TsFileExtractVisdata.
+%
+% timestampUnit(not case sensitive):
+% 'us' if the timestamp is microsecond, e.g., 1621993620816000
+% 'ms' if it is millisecond, e.g., 1621993620816
+% 's' if it is second, e.g., 1621993620
+%
+% timeMap: record the time range of every chunk.
+% Key [tsName][fileName][version] identifies the only chunk. Value is
+% [startTime,endTime] of the chunk.
+%
+% countMap: record the point count number of every chunk. Key is the same
+% as that of timeMap. Value is countNum.
+```
+
+```matlab
+function draw(timeMap,countMap,showSpecific,isFileOrder)
+% Plot figures given the loaded data and two plot parameters:
+% `showSpecific` and `isFileOrder`.
+%
+% process: 1) traverse `keys(timeMap)` to get the position arrangements on
+% the y axis dynamically, which is defined simultaneously by
+% (a)`showSpecific`: traverse `keys(timeMap)`, filter out keys
+% that don't statisfy `showSpecific`.
+% (b) seqKey/unseqKey display policies: extract seqKey or unseqKey
+% from statisfied keys. Sequence and unsequence data take different
+% display policies:
+% b-1) unseqKey identifies tsName and fileName, so data with the
+% same fileName and tsName but different versionNums are
+% plotted in the same line.
+% b-2) seqKey identifies tsName, so data with the same tsName but
+% different fileNames and versionNums are ploteed in the same
+% line.
+% (c)`isFileOrder`: sort seqKey&unseqKey according to `isFileOrder`,
+% finally get the position arrangements on the y axis.
+% 2) traverse `keys(timeMap)` again, get startTime&endTime from
+% `treeMap` as positions on the x axis, combined with the
+% positions on the y axis from the last step, finish plot.
+%
+% timeMap,countMap: generated by loadVisData function.
+%
+% showSpecific: the specific set of time series to be plotted.
+% If showSpecific is empty{}, then all loaded time series
+% will be plotted.
+% Note: Wildcard matching is not supported now. In other
+% words, showSpecific only support full time series path
+% names.
+%
+% isFileOrder: true to sort seqKeys&unseqKeys by fileName priority, false
+% to sort seqKeys&unseqKeys by tsName priority.
+```
+
+
+
+#### 3.5.2 例子
+
+##### 例1
+
+使用由`IoTDBLargeDataIT.insertData`写出的tsfiles。
+小修改：`IoTDBLargeDataIT.insertData`最后添加一条`statement.execute("flush");`指令。
+
+第一步：运行`TsFileExtractVisdata.java`
+
+```
+.\print-tsfile-visdata.bat D:\rel0.11\iotdb\server\target\data\sequence true D:\rel0.11\iotdb\server\target\data\unsequence false D:\visdata1.csv
+```
+或者等价地：
+```
+.\print-tsfile-visdata.bat D:\rel0.11\iotdb\server\target\data\sequence\root.vehicle\0\1622743492580-1-0.tsfile true D:\rel0.11\iotdb\server\target\data\sequence\root.vehicle\0\1622743505092-2-0.tsfile true D:\rel0.11\iotdb\server\target\data\sequence\root.vehicle\0\1622743505573-3-0.tsfile true D:\rel0.11\iotdb\server\target\data\unsequence\root.vehicle\0\1622743505901-4-0.tsfile false D:\visdata1.csv
+```
+
+第二步：运行`vis.m`
+
+```matlab
+clear all;close all;
+
+% 1. load visdata generated by TsFileExtractDataToVisTool
+filePath = 'D:\visdata1.csv';
+[timeMap,countMap] = loadVisData(filePath,'ms'); % mind the timestamp unit
+
+% 2. plot figures given the loaded data and two plot parameters:
+% `showSpecific` and `isFileOrder`
+draw(timeMap,countMap,{},false)
+title("draw(timeMap,countMap,\{\},false)")
+
+draw(timeMap,countMap,{},true)
+title("draw(timeMap,countMap,\{\},true)")
+
+draw(timeMap,countMap,{'root.vehicle.d0.s0'},false)
+title("draw(timeMap,countMap,{'root.vehicle.d0.s0'},false)")
+
+draw(timeMap,countMap,{'root.vehicle.d0.s0','root.vehicle.d0.s1'},false)
+title("draw(timeMap,countMap,{'root.vehicle.d0.s0','root.vehicle.d0.s1'},false)")
+
+draw(timeMap,countMap,{'root.vehicle.d0.s0','root.vehicle.d0.s1'},true)
+title("draw(timeMap,countMap,{'root.vehicle.d0.s0','root.vehicle.d0.s1'},true)")
+```
+
+绘图结果：
+
+
+
+
+
+
+
+
+
+
+
+
+
+##### 例2
+
+作为另一个例子，这是来自真实世界的数据：
+
+
+
+
+
+
+
+
diff --git a/server/src/assembly/resources/tools/tsfileToolSet/print-tsfile-visdata.bat b/server/src/assembly/resources/tools/tsfileToolSet/print-tsfile-visdata.bat
new file mode 100644
index 0000000..43772b5
--- /dev/null
+++ b/server/src/assembly/resources/tools/tsfileToolSet/print-tsfile-visdata.bat
@@ -0,0 +1,62 @@
+@REM
+@REM Licensed to the Apache Software Foundation (ASF) under one
+@REM or more contributor license agreements. See the NOTICE file
+@REM distributed with this work for additional information
+@REM regarding copyright ownership. The ASF licenses this file
+@REM to you under the Apache License, Version 2.0 (the
+@REM "License"); you may not use this file except in compliance
+@REM with the License. You may obtain a copy of the License at
+@REM
+@REM http://www.apache.org/licenses/LICENSE-2.0
+@REM
+@REM Unless required by applicable law or agreed to in writing,
+@REM software distributed under the License is distributed on an
+@REM "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+@REM KIND, either express or implied. See the License for the
+@REM specific language governing permissions and limitations
+@REM under the License.
+@REM
+
+
+@echo off
+echo ````````````````````````
+echo Starting Printing the TsFile Visdata
+echo ````````````````````````
+
+if "%OS%" == "Windows_NT" setlocal
+
+pushd %~dp0..\..
+if NOT DEFINED IOTDB_HOME set IOTDB_HOME=%CD%
+popd
+
+if NOT DEFINED MAIN_CLASS set MAIN_CLASS=org.apache.iotdb.db.tools.vis.TsFileExtractVisdata
+if NOT DEFINED JAVA_HOME goto :err
+
+@REM -----------------------------------------------------------------------------
+@REM ***** CLASSPATH library setting *****
+@REM Ensure that any user defined CLASSPATH variables are not used on startup
+set CLASSPATH="%IOTDB_HOME%\lib\*"
+
+goto okClasspath
+
+:append
+set CLASSPATH=%CLASSPATH%;%1
+goto :eof
+
+@REM -----------------------------------------------------------------------------
+:okClasspath
+
+"%JAVA_HOME%\bin\java" -cp "%CLASSPATH%" %MAIN_CLASS% %*
+
+goto finally
+
+
+:err
+echo JAVA_HOME environment variable must be set!
+pause
+
+
+@REM -----------------------------------------------------------------------------
+:finally
+
+ENDLOCAL
\ No newline at end of file
diff --git a/server/src/assembly/resources/tools/tsfileToolSet/print-tsfile-visdata.sh b/server/src/assembly/resources/tools/tsfileToolSet/print-tsfile-visdata.sh
new file mode 100644
index 0000000..28b1241
--- /dev/null
+++ b/server/src/assembly/resources/tools/tsfileToolSet/print-tsfile-visdata.sh
@@ -0,0 +1,48 @@
+#!/bin/sh
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+echo ---------------------
+echo Starting Printing the TsFile Sketch
+echo ---------------------
+
+if [ -z "${IOTDB_HOME}" ]; then
+ export IOTDB_HOME="$(cd "`dirname "$0"`"/../..; pwd)"
+fi
+
+if [ -n "$JAVA_HOME" ]; then
+ for java in "$JAVA_HOME"/bin/amd64/java "$JAVA_HOME"/bin/java; do
+ if [ -x "$java" ]; then
+ JAVA="$java"
+ break
+ fi
+ done
+else
+ JAVA=java
+fi
+
+CLASSPATH=""
+for f in ${IOTDB_HOME}/lib/*.jar; do
+ CLASSPATH=${CLASSPATH}":"$f
+done
+
+MAIN_CLASS=org.apache.iotdb.db.tools.vis.TsFileExtractVisdata
+
+"$JAVA" -cp "$CLASSPATH" "$MAIN_CLASS" "$@"
+exit $?
\ No newline at end of file
diff --git a/server/src/main/java/org/apache/iotdb/db/tools/vis/TsFileExtractVisdata.java b/server/src/main/java/org/apache/iotdb/db/tools/vis/TsFileExtractVisdata.java
new file mode 100644
index 0000000..d6a143d
--- /dev/null
+++ b/server/src/main/java/org/apache/iotdb/db/tools/vis/TsFileExtractVisdata.java
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.tools.vis;
+
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import org.apache.iotdb.tsfile.common.constant.TsFileConstant;
+import org.apache.iotdb.tsfile.file.metadata.ChunkGroupMetadata;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetadata;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.utils.Pair;
+
+/**
+ * Extract, from input tsfiles, necessary visualization information, which is what "vis.m" needs to plot figures.
+ * <p>
+ * Input: [path1 seqIndicator1 path2 seqIndicator2 ... pathN seqIndicatorN outputPath]
+ *
+ * Example: G:\\debug\\data\\sequence true G:\\debug\\data\\unsequence false visdata.csv 2N+1 args in total.
+ *
+ * `seqIndicator` should be 'true' or 'false' (not case sensitive).
+ * 'true' means is the file is sequence, 'false' means the file is unsequence.
+ *
+ * `Path` can be the full path of a file or a directory path. If it is a directory path, make sure
+ * that all files in this directory have the same seqIndicator.
+ * <p>
+ * Output content format: [tsName, fileName, versionNum, startTime, endTime, countNum]
+ *
+ * `fileName`: If the tsfile is unsequence file, TsFileExtractVisdata will make sure that the
+ * fileName contains "unseq" as an indicator which will be used by "vis.m".
+ *
+ */
+public class TsFileExtractVisdata {
+
+ public static String seqFileNameSuffix = "(seque)";
+ public static String unseqFileNameSuffix = "(unseq)";
+
+ public static void main(String[] args) throws IOException {
+ int M = args.length;
+ if (M % 2 == 0) {
+ throw new IOException("2N+1 args should be:[path1 seqIndicator1 path2 seqIndicator2 "
+ + "... pathN seqIndicatorN outputPath]");
+ }
+ List<String> inputPathList = new ArrayList<>();
+ List<Boolean> seqIndicatorList = new ArrayList<>();
+ for (int i = 0; i < M - 1; i = i + 2) {
+ inputPathList.add(args[i]);
+ String indicator = args[i + 1].toLowerCase();
+ if (!indicator.equals("true") && !indicator.equals("false")) {
+ throw new IOException("seqIndicator should be 'true' or 'false' (not case sensitive).");
+ }
+ seqIndicatorList.add(Boolean.parseBoolean(args[i + 1]));
+ }
+ String outputPath = args[M - 1];
+
+ try (PrintWriter pw = new PrintWriter(new FileWriter(outputPath))) {
+ int idx = 0;
+ for (String inputPath : inputPathList) {
+ boolean isSeq = seqIndicatorList.get(idx++);
+ List<String> filelist = new ArrayList<>();
+ filelist = getFile(inputPath, filelist); // get all tsfile paths under the inputPath
+ for (String f : filelist) {
+ System.out.println(f); // note that this info need not be written to outputFile
+ String fileNameForVis;
+ // Extract the file name from f, following the rule negotiated with "vis.m".
+ // The rule is that if it is an unsequence tsfile, its extracted fileName must contain "unseq".
+ // It's not a must for the extracted fileName of a sequence tsfile to contain "seque", but good to do so.
+ File file = new File(f);
+ String fileName = file.getName();
+ if (isSeq) {
+ fileNameForVis = fileName + seqFileNameSuffix;
+ } else {
+ fileNameForVis = fileName + unseqFileNameSuffix;
+ }
+ // extract necessary vis info from the tsfile
+ try (TsFileSequenceReader reader = new TsFileSequenceReader(f)) {
+ List<ChunkGroupMetadata> allChunkGroupMetadata = new ArrayList<>();
+ List<Pair<Long, Long>> versionInfo = new ArrayList<>();
+ reader.selfCheck(null, allChunkGroupMetadata, versionInfo, false);
+ Map<Long, Long> versionMap = new TreeMap<>();
+ for (Pair<Long, Long> versionPair : versionInfo) {
+ versionMap.put(versionPair.left - Long.BYTES - 1, versionPair.right);
+ } // note that versionInfo may be empty
+ Set<Map.Entry<Long, Long>> entrySet = versionMap.entrySet();
+ Iterator<Map.Entry<Long, Long>> itr = entrySet.iterator();
+ Long currVersionPos = null;
+ Long currVersion = null;
+ for (ChunkGroupMetadata chunkGroupMetadata : allChunkGroupMetadata) {
+ List<ChunkMetadata> chunkMetadataList = chunkGroupMetadata.getChunkMetadataList();
+ for (ChunkMetadata chunkMetadata : chunkMetadataList) {
+ String tsName =
+ chunkGroupMetadata.getDevice() + TsFileConstant.PATH_SEPARATOR + chunkMetadata
+ .getMeasurementUid();
+ long startTime = chunkMetadata.getStartTime();
+ long endTime = chunkMetadata.getEndTime();
+ long countNum = chunkMetadata.getStatistics().getCount();
+ // get versionNum by comparing offsetOfChunkHeader and currVersionPos
+ long versionNum = -1; // therefore if versionMap is empty, versionNum writes -1
+ if (!versionMap.isEmpty()) {
+ long offsetOfChunkHeader = chunkMetadata.getOffsetOfChunkHeader();
+ if (currVersionPos == null || offsetOfChunkHeader > currVersionPos) {
+ if (itr.hasNext()) {
+ Map.Entry<Long, Long> v = itr.next();
+ currVersionPos = v.getKey();
+ currVersion = v.getValue();
+ } else {
+ throw new IOException(
+ String.format("Something is wrong with the tsfile %s, " +
+ "because there is a chunk with no version after it "
+ + "when there should be.", f));
+ }
+ }
+ versionNum = currVersion;
+ }
+ printlnBoth(pw, String.format("%s,%s,%d,%d,%d,%d", tsName, fileNameForVis,
+ versionNum, startTime, endTime, countNum));
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Get the list of paths of all tsfiles under the input path recursively.
+ *
+ * @param path can be the full path of a file or a directory path
+ * @param filelist
+ * @return the list of paths of all tsfiles under the input path
+ */
+ private static List<String> getFile(String path, List<String> filelist) {
+ File file = new File(path);
+ File[] array = file.listFiles();
+ if (array == null) {
+ filelist.add(path);
+ return filelist;
+ }
+ for (File value : array) {
+ if (value.isFile()) {
+ String filePath = value.getPath();
+ if (filePath.endsWith(TsFileConstant.TSFILE_SUFFIX)) {
+ // only include tsfiles
+ filelist.add(value.getPath());
+ }
+ } else if (value.isDirectory()) {
+ getFile(value.getPath(), filelist);
+ }
+ }
+ return filelist;
+ }
+
+ private static void printlnBoth(PrintWriter pw, String str) {
+ System.out.println(str);
+ pw.println(str);
+ }
+}
+
diff --git a/server/src/main/java/org/apache/iotdb/db/tools/vis/vis.m b/server/src/main/java/org/apache/iotdb/db/tools/vis/vis.m
new file mode 100644
index 0000000..bcb954e
--- /dev/null
+++ b/server/src/main/java/org/apache/iotdb/db/tools/vis/vis.m
@@ -0,0 +1,288 @@
+ %
+ % Licensed to the Apache Software Foundation (ASF) under one
+ % or more contributor license agreements. See the NOTICE file
+ % distributed with this work for additional information
+ % regarding copyright ownership. The ASF licenses this file
+ % to you under the Apache License, Version 2.0 (the
+ % "License"); you may not use this file except in compliance
+ % with the License. You may obtain a copy of the License at
+ %
+ % http://www.apache.org/licenses/LICENSE-2.0
+ %
+ % Unless required by applicable law or agreed to in writing,
+ % software distributed under the License is distributed on an
+ % "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ % KIND, either express or implied. See the License for the
+ % specific language governing permissions and limitations
+ % under the License.
+ %
+
+clear all;close all;
+
+% 1. load visdata generated by TsFileExtractDataToVisTool
+filePath = 'D:\visdata1.csv';
+[timeMap,countMap] = loadVisData(filePath,'ms'); % mind the timestamp unit
+
+% 2. plot figures given the loaded data and two plot parameters:
+% `showSpecific` and `isFileOrder`
+draw(timeMap,countMap,{},false)
+
+draw(timeMap,countMap,{},true)
+
+draw(timeMap,countMap,{'root.vehicle.d0.s0'},false)
+
+draw(timeMap,countMap,{'root.vehicle.d0.s0','root.vehicle.d0.s1'},false)
+
+draw(timeMap,countMap,{'root.vehicle.d0.s0','root.vehicle.d0.s1'},true)
+
+%% Functions
+function [timeMap,countMap] = loadVisData(filePath,timestampUnit)
+% Load visdata generated by TsFileExtractDataToVisTool.
+%
+% filePath: the path of visdata.
+% The format is [tsName,fileName,versionNum,startTime,endTime,countNum].
+% `tsName` and `fileName` are string, the others are long value.
+% If the tsfile is unsequence file, `fileName` will contain "unseq" as an
+% indicator, which is guaranteed by TsFileExtractDataToVisTool.
+%
+% timestampUnit(not case sensitive):
+% 'us' if the timestamp is microsecond, e.g., 1621993620816000
+% 'ms' if it is millisecond, e.g., 1621993620816
+% 's' if it is second, e.g., 1621993620
+%
+% timeMap: record the time range of every chunk.
+% Key [tsName][fileName][version] identifies the only chunk. Value is
+% [startTime,endTime] of the chunk.
+%
+% countMap: record the point count number of every chunk. Key is the same
+% as that of timeMap. Value is countNum.
+
+ timeMap = containers.Map();
+ countMap = containers.Map();
+ A = readtable(filePath);
+ for i=1:1:size(A,1)
+ [timeMap,countMap]=addChunkTime(A{i,1}{1},A{i,2}{1},A{i,3},...
+ A{i,4},A{i,5},A{i,6},timestampUnit,timeMap,countMap);
+ end
+ disp(['loadVisData finished. ','load ',num2str(size(timeMap,1)),' chunks in total'])
+end
+
+function [timeMap,countMap] = addChunkTime(tsName,fileName,versionNum,...
+ startTime,endTime,countNum,timestampUnit,timeMap,countMap)
+% Used by loadVisData function.
+
+ % the key [tsName][fileName][version] identifies the only chunk
+ key = ['[',tsName,'][', fileName,'][', num2str(versionNum),'V]'];
+
+ startTime = convertLongToDate(startTime,timestampUnit);
+ endTime = convertLongToDate(endTime,timestampUnit);
+ timeMap(key) = [startTime,endTime];
+
+ countMap(key) = countNum;
+end
+
+function draw(timeMap,countMap,showSpecific,isFileOrder)
+% Plot figures given the loaded data and two plot parameters:
+% `showSpecific` and `isFileOrder`.
+%
+% process: 1) traverse `keys(timeMap)` to get the position arrangements on
+% the y axis dynamically, which is defined simultaneously by
+% (a)`showSpecific`: traverse `keys(timeMap)`, filter out keys
+% that don't statisfy `showSpecific`.
+% (b) seqKey/unseqKey display policies: extract seqKey or unseqKey
+% from statisfied keys under different display policies:
+% b-1) unseqKey identifies tsName and fileName, so chunk data with the
+% same fileName and tsName but different versionNums are
+% plotted in the same line.
+% b-2) seqKey identifies tsName, so chunk data with the same tsName but
+% different fileNames and versionNums are ploteed in the same
+% line.
+% (c)`isFileOrder`: sort seqKey&unseqKey according to `isFileOrder`,
+% finally get the position arrangements on the y axis.
+% 2) traverse `keys(timeMap)` again, get startTime&endTime from
+% `treeMap` as positions on the x axis, combined with the
+% positions on the y axis from the last step, finish plot.
+%
+% timeMap,countMap: generated by loadVisData function.
+%
+% showSpecific: the specific set of time series to be plotted.
+% If showSpecific is empty{}, then all loaded time series
+% will be plotted.
+% Note: Wildcard matching is not supported now. In other
+% words, showSpecific only support full time series path
+% names.
+%
+% isFileOrder: true to sort seqKeys&unseqKeys by fileName priority, false
+% to sort seqKeys&unseqKeys by tsName priority.
+
+ % traverse `keys(timeMap)` to get the position arrangements on the y axis dynamically
+ [yticklabelMap,allkeysBoolean]=get_yticklabelMaps(keys(timeMap),showSpecific, isFileOrder);
+ disp('draw: get_yticklabelMaps finished')
+
+ % traverse `keys(timeMap)` again to plot
+ if all(allkeysBoolean(:)==false)
+ disp('no statisfied specific data')
+ return
+ end
+ figure,
+ m=1;
+ for k = keys(timeMap)
+ if ~allkeysBoolean(m)
+ m = m + 1; % don't forget this step
+ continue;
+ end
+ key = k{1};
+ % extract [tsName][fileName][versionNum] respectively
+ timeMapKeySplit=split(key,["[","]","]["]);
+ tsName = timeMapKeySplit{2};
+ fileName = timeMapKeySplit{4};
+ versionNumStr = timeMapKeySplit{6};
+ % get the positions on the x axis
+ timeRange = timeMap(key);
+ % get the positions on the y axis
+ if contains(key,'unseq') % unsequence tsfile
+ % unseqKey identifies tsName and fileName, so data with the
+ % same fileName and tsName but different versionNums are
+ % ploteed on the same line.
+ if isFileOrder
+ % unseqKey is 'fileName/tsName'
+ yPos = yticklabelMap([fileName,'/',tsName]);
+ else
+ % unseqKey is 'tsName/fileName'
+ yPos = yticklabelMap([tsName,'/',fileName]);
+ end
+ % begin plot
+ plot(timeRange,[yPos,yPos],'LineWidth',8),hold on
+ text(timeRange(1),yPos-0.3,sprintf(num2str(countMap(key))),'FontSize',7),hold on
+ text(timeRange(1),yPos+0.3,sprintf(['[',versionNumStr,']']),'FontSize',7),hold on
+ % add short vertical lines to the two ends of timeRange
+ plot([timeRange(1),timeRange(1)],[yPos,yPos+0.3]),hold on
+ plot([timeRange(2),timeRange(2)],[yPos,yPos+0.3]),hold on
+ m = m + 1;
+ else % sequence tsfile
+ % seqKey identifies tsName, so data with the same tsName but
+ % different fileNames and versionNums are ploteed on the same
+ % line.
+ yPos = yticklabelMap(tsName);
+ % begin plot
+ plot(timeRange,[yPos,yPos],'LineWidth',8),hold on
+ text(timeRange(1),yPos-0.3,sprintf(num2str(countMap(key))),'FontSize',7),hold on
+ text(timeRange(1),yPos+0.3,sprintf(['[',versionNumStr,']']),'FontSize',7),hold on
+ % add fileName text, since seqKey does not contain this information
+ text(timeRange(1),yPos+0.5,sprintf(fileName),'FontSize',7),hold on
+ % add short vertical lines to the two ends of timeRange
+ plot([timeRange(1),timeRange(1)],[yPos,yPos+0.3]),hold on
+ plot([timeRange(2),timeRange(2)],[yPos,yPos+0.3]),hold on
+ m = m + 1;
+ end
+ end
+ yticklabelRes = keys(yticklabelMap);
+ yticklabelNum = size(yticklabelRes,2);
+ yticks(1:1:yticklabelNum)
+ yticklabels(yticklabelRes)
+ xtickformat('yyyy-MM-dd HH:mm:ss.SSS')
+ ylim([0 yticklabelNum+1])
+ disp('draw finished')
+end
+
+function [yticklabelMap,allkeysBoolean]=get_yticklabelMaps(allkeys,showSpecific,isFileOrder)
+% Used by draw function. Given two plot parameters: `showSpecific` and
+% `isFileOrder`,it dynamically generate the position arrangements on
+% the y axis.
+%
+% process: traverse `keys(timeMap)`, filter out keys that don't statisfy
+% `showSpecific`, extract seqKey or unseqKey from statisfied keys,
+% sort seqKey&unseqKey according to `isFileOrder` to get the position
+% arrangements on the y axis.
+%
+% allkeys: a 1*N cell array stores the keys of all loaded data. Key
+% [tsName][fileName][version] identifies the only chunk.
+%
+% showSpecific: the specific set of time series to be plotted.
+% If showSpecific is empty{}, then all loaded time series
+% will be plotted.
+% Note: Wildcard matching is not supported now. In other
+% words, showSpecific only support full time series path
+% names.
+%
+% isFileOrder: true to sort seqKeys&unseqKeys by fileName
+% priority('fileName/tsName'), false to sort seqKeys&unseqKeys
+% by tsName priority ('tsName/fileName').
+%
+% yticklabelMap: seqKey/unseqKey->yPos, record the position arrangements on
+% the y axis.
+%
+% allkeysBoolean: save the filtered result of allkeys to avoid repeated
+% computation later in the draw function.
+
+ N=size(allkeys,2);
+ allkeysBoolean = ones(1,N);
+ m=1;
+ seqOrUnseqKeys = {};
+ for k = allkeys
+ key = k{1};
+ % extract [tsName][fileName][versionNum] respectively
+ timeMapKeySplit=split(key,["[","]","]["]);
+ tsName = timeMapKeySplit{2};
+ fileName = timeMapKeySplit{4};
+ % filter given showSpecific
+ if ~isempty(showSpecific)&&~any(strcmp(showSpecific,tsName))
+ % note that can not use "contains", because for example V1.C101
+ % containes V1.C1
+ allkeysBoolean(m)=false;
+ m = m + 1;
+ continue;
+ % Note: Wildcard matching is not supported now. In other
+ % words, showSpecific only support full time series path
+ % names.
+ end
+ allkeysBoolean(m)=true;
+ if contains(fileName,'unseq') %unsequence tsfile
+ if isFileOrder
+ unseqKey=[fileName,'/',tsName];
+ else
+ unseqKey=[tsName,'/', fileName];
+ end
+ if ~any(strcmp(seqOrUnseqKeys,unseqKey))
+ seqOrUnseqKeys{end+1}=unseqKey;
+ end
+ else %sequence tsfile
+ seqKey=tsName;
+ if ~any(strcmp(seqOrUnseqKeys,seqKey))
+ seqOrUnseqKeys{end+1}=seqKey;
+ end
+ end
+ m = m + 1;
+ end
+ seqOrUnseqKeys = sort(seqOrUnseqKeys);
+ yticklabelMap = containers.Map();
+ yposCurr = 1;
+ for yposKey = seqOrUnseqKeys
+ yticklabelMap(yposKey{1}) = yposCurr;
+ yposCurr = yposCurr + 1;
+ end
+end
+
+function t=convertLongToDate(timestamp,timestampUnit)
+% timestampUnit(not case sensitive):
+% 'us' if the timestamp is microsecond, e.g., 1621993620816000
+% 'ms' if it is millisecond, e.g., 1621993620816
+% 's' if it is second, e.g., 1621993620
+
+ timestampUnit = lower(timestampUnit);
+ if strcmp(timestampUnit,'us')
+ % microsecond, e.g., 1621993620816000
+ posix_timestamp = timestamp/1000/1000;
+ t=datetime(posix_timestamp,'convertfrom','posixtime', 'Format', 'yyyy-MM-dd HH:mm:ss.SSSSSS','TimeZone','local');
+ elseif strcmp(timestampUnit,'ms')
+ % millisecond, e.g., 1621993620816
+ posix_timestamp = timestamp/1000;
+ t=datetime(posix_timestamp,'convertfrom','posixtime', 'Format', 'yyyy-MM-dd HH:mm:ss.SSS','TimeZone','local');
+ elseif strcmp(timestampUnit,'s')
+ % second, e.g., 1621993620
+ posix_timestamp = timestamp;
+ t=datetime(posix_timestamp,'convertfrom','posixtime', 'Format', 'yyyy-MM-dd HH:mm:ss','TimeZone','local');
+ else
+ error('Wrong timestamp unit. It should be us/ms/s.')
+ end
+end