You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by wu...@apache.org on 2021/11/23 12:35:35 UTC
[skywalking-website] branch master updated: add IoTDB storage option design blog (#358)
This is an automated email from the ASF dual-hosted git repository.
wusheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/skywalking-website.git
The following commit(s) were added to refs/heads/master by this push:
new 5914e41 add IoTDB storage option design blog (#358)
5914e41 is described below
commit 5914e4146c0b7e3dbb79b11a485128bae37052c6
Author: 刘威 <51...@users.noreply.github.com>
AuthorDate: Tue Nov 23 20:35:29 2021 +0800
add IoTDB storage option design blog (#358)
---
.../index.md | 201 +++++++++++++++++++++
1 file changed, 201 insertions(+)
diff --git a/content/blog/2021-11-23-design-of-iotdb-storage-option/index.md b/content/blog/2021-11-23-design-of-iotdb-storage-option/index.md
new file mode 100644
index 0000000..fe4e597
--- /dev/null
+++ b/content/blog/2021-11-23-design-of-iotdb-storage-option/index.md
@@ -0,0 +1,201 @@
+---
+title: "The Design of Apache IoTDB Storage Option"
+date: "2021-11-23"
+author: "[Wei Liu](https://github.com/LIU-WEI-git), School of Computer Science, NPU, China"
+description: "This is an introduction to IoTDB Storage Option. It introduces some basic concept about IoTDB and the storage schema design."
+
+tags:
+- Storage
+- IoTDB
+---
+
+This plugin is the outcome of [Apache IoTDB - Apache SkyWalking Adapter](https://summer.iscas.ac.cn/#/org/prodetail/210070771) in [Summer 2021 of Open Source Promotion Plan](https://summer.iscas.ac.cn/#/homepage). The design and development work is under the guidance of [@jixuan1989](https://github.com/jixuan1989) from IoTDB and [@wu-sheng](https://github.com/wu-sheng) from SkyWalking. Thanks for their guidance and the help from community.
+
+## IoTDB Storage Plugin Setup
+IoTDB is a time-series database from Apache, which is one of the storage plugin options. If you want to use iotdb as SkyWalking backend storage, please refer to the following configuration.
+IoTDB storage plugin is still in progress. Its efficiency will improve in the future.
+
+```yaml
+storage:
+ selector: ${SW_STORAGE:iotdb}
+ iotdb:
+ host: ${SW_STORAGE_IOTDB_HOST:127.0.0.1}
+ rpcPort: ${SW_STORAGE_IOTDB_RPC_PORT:6667}
+ username: ${SW_STORAGE_IOTDB_USERNAME:root}
+ password: ${SW_STORAGE_IOTDB_PASSWORD:root}
+ storageGroup: ${SW_STORAGE_IOTDB_STORAGE_GROUP:root.skywalking}
+ sessionPoolSize: ${SW_STORAGE_IOTDB_SESSIONPOOL_SIZE:16}
+ fetchTaskLogMaxSize: ${SW_STORAGE_IOTDB_FETCH_TASK_LOG_MAX_SIZE:1000} # the max number of fetch task log in a request
+```
+All connection related settings, including host, rpcPort, username, and password are found in `application.yml`. Please ensure the IoTDB version >= 0.12.3.
+
+## IoTDB Introduction
+[Apache IoTDB](https://iotdb.apache.org/) (Database for Internet of Things) is an IoT native database with high performance for data management and analysis, deployable on the edge and the cloud. It is a time-series database donated by Tsinghua University to Apache Foundation.
+
+### The Data Model of IoTDB
+We can use the tree structure to understand the data model of iotdb. If divided according to layers, from high to low is: `Storage Group` -- (`LayerName`) -- `Device` -- `Measurement`. From the top layer to a certain layer below it is called a Path. The top layer is `Storage Group` (must start with `root.`), the penultimate layer is `Device`, and the bottom layer is `Measurement`. There can be many layers in the middle, and each layer is called a `LayerName`. For more information, please [...]
+
+## The Design of IoTDB Storage Plugin
+
+### The Data Model of SkyWalking
+Each storage model of SkyWalking can be considered as a Model, which contains multiple Columns. Each Column has ColumnName and ColumnType attributes, representing the name and type of Column respectively. Each Column named ColumnName stores multiple Value of the ColumnType. From a relational database perspective, Model is a relational table and Column is the field in a relational table.
+
+### Schema Design
+Since each `LayerName` of IoTDB is stored in memory, it can be considered as an index, and this feature can be fully utilized to improve IoTDB query performance. The default storage group is `root.skywalking`, it will occupy the first and the second layer of the path. The model name is stored at the next layer of the storage group (the third layer of the path), such as `root.skywalking.model_name`.
+
+SkyWalking has its own index requirement, but it isn't applicable to IoTDB. Considering query frequency and referring to the implementation of the other storage options, we choose `id`, `entity_id`, `node_type`, `service_id`, `service_group`, `trace_id` as indexes and fix their order in the path. If we don't fix their order, we cannot map their value to column, since we only store their value in the path but don't store their column name. The other columns are treated as Measurements.
+
+The mapping from SkyWalking data model to IoTDB data model is below.
+
+| SkyWalking | IoTDB |
+| -------------------- | -------------------------------------------- |
+| Database | Storage Group(1st and 2nd layer of the path) |
+| Model | LayerName(3rd layer of the path) |
+| Indexed Column | stored in memory through hard-code |
+| Indexed Column Value | LayerName(after 3rd layer of the path) |
+| Non-indexed Column | Measurement |
+| Non-indexed Value | the value of Measurement |
+
+#### For general example
+There are model1('column11', column12), model2('column21', 'column22', column23), model3(column31). Single quotation mark indicates that the column requires to be indexed. In this example, `modelx_name` refers to the name of modelx, `columnx_name` refers to the name of columnx and `columnx_value` refers to the value of columnx.
+
+Before these 3 model storage schema, here are some points we need to know.
+- In order to avoid the value of indexed column contains dot(`.`), all of them should be wrapped in double quotation mark since IoTDB use dot(`.`) as the separator in the path.
+- We use `align by device` in query SQL to get a more friendly result. For more information about `align by device`, please see [DML (Data Manipulation Language)](https://iotdb.apache.org/UserGuide/V0.12.x/IoTDB-SQL-Language/DML-Data-Manipulation-Language.html) and [Query by device alignment](https://iotdb.apache.org/SystemDesign/DataQuery/AlignByDeviceQuery.html).
+
+The path of them is following:
+- The Model with index:
+ - `root.skywalking.model1_name.column11_value.column12_name`
+ - `root.skywalking.model2_name.column21_value.column22_value.column23_name`
+- The Model without index:
+ - `root.skywalking.model3_name.column31_Name`
+
+Use `select * from root.skywalking.modelx_name align by device` respectively to get their schema and data. The SQL result is following:
+
+| Time | Device | column12_name |
+| ------------- | -------------------------------------------- | -------------- |
+| 1637494020000 | root.skywalking.model1_name."column11_value" | column12_value |
+
+| Time | Device | column23_name |
+| ------------- | ------------------------------------------------------------- | -------------- |
+| 1637494020000 | root.skywalking.model2_name."column21_value"."column22_value" | column23_value |
+
+| Time | Device | column31_name |
+| ------------- | --------------------------- | -------------- |
+| 1637494020000 | root.skywalking.model3_name | column31_value |
+
+#### For specific example
+Before 5 typical examples, here are some points we need to know.
+- The indexed columns and their order: `id`, `entity_id`, `node_type`, `service_id`, `service_group`, `trace_id`. Other columns are treated as non indexed and stored as Measurement.
+- The storage entity extends Metrics or Record contains a column `time_bucket`. The `time_bucket` column in SkyWalking Model can be converted to the `timestamp` of IoTDB when inserting data. We don't need to store `time_bucket` separately. In the next examples, we won't list `time_bucket` anymore.
+- The `Time` in query result corresponds to the `timestamp` in insert SQL and API.
+
+1) Metadata: service_traffic
+service_traffic entity has 4 columns: 'id', name, 'node_type', 'service_group'.
+When service_traffic entity includes a row with timestamp 1637494020000, the row should be as following:
+(**Notice**: the value of service_group is null.)
+
+| id | name | node_type | service_group |
+| ------------------------------ | -------------------- | --------- | ------------- |
+| ZTJlLXNlcnZpY2UtcHJvdmlkZXI=.1 | e2e-service-provider | 0 | |
+
+And the row stored in IoTDB should be as following:
+(Query SQL: `select from root.skywalking.service_traffic align by device`)
+
+| Time | Device | name |
+| ------------- | --------------------------------------------------------------------------- | -------------------- |
+| 1637494020000 | root.skywalking.service_traffic."ZTJlLXNlcnZpY2UtcHJvdmlkZXI=.1"."0"."null" | e2e-service-provider |
+
+The value of id, node_type and service_group are stored in the path in the specified order.
+**Notice**: If those index value is null, it will be transformed to a string "null".
+
+2) Metrics: service_cpm
+service_cpm entity has 4 columns: 'id', 'service_id', total, value.
+When service_cpm entity includes a row with timestamp 1637494020000, the row should be as following:
+
+| id | service_id | total | value |
+| ------------------------------------------- | ------------------------------ | ----- | ----- |
+| 202111211127_ZTJlLXNlcnZpY2UtY29uc3VtZXI=.1 | ZTJlLXNlcnZpY2UtY29uc3VtZXI=.1 | 4 | 4 |
+
+And the row stored in IoTDB should be as following:
+(Query SQL: `select from root.skywalking.service_cpm align by device`)
+
+| Time | Device | total | value |
+| ------------- | ---------------------------------------------------------------------------------------------------------- | ----- | ----- |
+| 1637494020000 | root.skywalking.service_cpm."202111211127_ZTJlLXNlcnZpY2UtY29uc3VtZXI=.1"."ZTJlLXNlcnZpY2UtY29uc3VtZXI=.1" | 4 | 4 |
+
+The value of id and service_id are stored in the path in the specified order.
+
+3) Trace segment: segment
+segment entity has 10 columns at least: 'id', segment_id, 'trace_id', 'service_id', service_instance_id, endpoint_id, start_time, latency, is_error, data_binary. In addition, it could have variable number of tags.
+When segment entity includes 2 rows with timestamp 1637494106000 and 1637494134000, these rows should be as following. The `db.type` and `db.instance` are two tags. The first data has two tags, and the second data doesn't have tag.
+
+| id | segment_id | trace_id | service_id | service_instance_id | endpoint_id | start_time | latency | is_error | data_binary | db.type | db.instance |
+| ---- | ------------ | ---------- | ------------ | --------------------- | ------------- | ------------- | ------- | -------- | ------------- | ------- | ----------- |
+| id_1 | segment_id_1 | trace_id_1 | service_id_1 | service_instance_id_1 | endpoint_id_1 | 1637494106515 | 1425 | 0 | data_binary_1 | sql | testdb |
+| id_2 | segment_id_2 | trace_id_2 | service_id_2 | service_instance_id_2 | endpoint_id_2 | 2637494106765 | 1254 | 0 | data_binary_2 | | |
+
+And these row stored in IoTDB should be as following:
+(Query SQL: `select from root.skywalking.segment align by device`)
+
+| Time | Device | start_time | data_binary | latency | endpoint_id | is_error | service_instance_id | segment_id | "db.type" | "db.instance" |
+| ------------- | ---------------------------------------------------------- | ------------- | ------------- | ------- | ------------- | -------- | --------------------- | ------------ | --------- | ------------- |
+| 1637494106000 | root.skywalking.segment."id_1"."service_id_1"."trace_id_1" | 1637494106515 | data_binary_1 | 1425 | endpoint_id_1 | 0 | service_instance_id_1 | segment_id_1 | sql | testdb |
+| 1637494106000 | root.skywalking.segment."id_2"."service_id_2"."trace_id_2" | 1637494106765 | data_binary_2 | 1254 | endpoint_id_2 | 0 | service_instance_id_2 | segment_id_2 | null | null |
+
+The value of id, service_id and trace_id are stored in the path in the specified order.
+**Notice**: If the measurement contains dot(`.`), it will be wrapped in double quotation mark since IoTDB doesn't allow it. In order to align, IoTDB will append null value for those data without tag in some models.
+
+4) Log
+log entity has 12 columns at least: 'id', unique_id, 'service_id', service_instance_id, endpoint_id, 'trace_id', trace_segment_id, span_id, content_type, content, tags_raw_data, timestamp. In addition, it could have variable number of tags.
+When log entity includes a row with timestamp 1637494052000, the row should be as following and the level is a tag.
+
+| id | unique_id | service_id | service_instance_id | endpoint_id | trace_id | trace_segment_id | span_id | content_type | content | tags_raw_data | timestamp | level |
+| ---- | ----------- | ------------ | --------------------- | ------------- | ---------- | ------------------ | ------- | ------------ | --------- | --------------- | ------------- | ----- |
+| id_1 | unique_id_1 | service_id_1 | service_instance_id_1 | endpoint_id_1 | trace_id_1 | trace_segment_id_1 | 0 | 1 | content_1 | tags_raw_data_1 | 1637494052118 | INFO |
+
+And the row stored in IoTDB should be as following:
+(Query SQL: `select from root.skywalking.log align by device`)
+
+| Time | Device | unique_id | content_type | span_id | tags_raw_data | "timestamp" | level | service_instance_id | content | trace_segment_id |
+| ------------- | -------------------------------------------------- | ----------- | ------------ | ------- | --------------- | ------------- | ----- | --------------------- | --------- | ------------------ |
+| 1637494052000 | root.skywalking."id_1"."service_id_1"."trace_id_1" | unique_id_1 | 1 | 0 | tags_raw_data_1 | 1637494052118 | INFO | service_instance_id_1 | content_1 | trace_segment_id_1 |
+
+The value of id, service_id and trace_id are stored in the path in the specified order.
+**Notice**: If the measurement named timestamp, it will be wrapped in double quotation mark since IoTDB doesn't allow it.
+
+5) Profiling snapshots: profile_task_segment_snapshot
+profile_task_segment_snapshot has 6 columns: 'id', task_id, segment_id, dump_time, sequence, stack_binary.
+When profile_task_segment_snapshot includes a row with timestamp 1637494131000, the row should be as following.
+
+| id | task_id | segment_id | dump_time | sequence | stack_binary |
+| ---- | --------- | ------------ | ------------- | -------- | -------------- |
+| id_1 | task_id_1 | segment_id_1 | 1637494131153 | 0 | stack_binary_1 |
+
+And the row stored in IoTDB should be as following:
+(Query SQL: `select from root.skywalking.profile_task_segment_snapshot align by device`)
+
+| Time | Device | sequence | dump_time | stack_binary | task_id | segment_id |
+| ------------- | ---------------------------------------------------- | -------- | ------------- | -------------- | --------- | ------------ |
+| 1637494131000 | root.skywalking.profile_task_segment_snapshot."id_1" | 0 | 1637494131153 | stack_binary_1 | task_id_1 | segment_id_1 |
+
+The value of id is stored in the path in the specified order.
+
+### Query
+In this design, part of the data is stored in memory through `LayerName`, so data from the same Model is spread across multiple `devices`. Queries often need to cross multiple `devices`. But in this aspect, IoTDB's support is not perfect in cross-device aggregation query, sort query and pagination query. In some cases, we have to use a **violence method** that query all data meets the condition and then aggregate, sort or paginate them. So it might not be efficient. For detailed descript [...]
+
+- Discussion:
+ - [一个有关排序查询的问题（A problem about sort query）#3888](https://github.com/apache/iotdb/discussions/3888)
+ - [一个有关聚合查询的问题（A problem about aggregation query）#3907](https://github.com/apache/iotdb/discussions/3907)
+
+Query SQL for the general example above:
+```sql
+-- query all data in model1
+select * from root.skywalking.model1_name align by device;
+-- query the data in model2 with column22_value="test"
+select * from root.skywalking.model2_name.*."test" align by device;
+-- query the sum of column23 in model2 and group by column21
+select sum(column23) from root.skywalking.model2_name.*.* group by level = 3;
+```
+
+iotdb-cli is a useful tools to connect and visit IoTDB server. More information please refer [Command Line Interface(CLI)](https://iotdb.apache.org/UserGuide/V0.12.x/CLI/Command-Line-Interface.html)