You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by GitBox <gi...@apache.org> on 2020/05/11 10:20:54 UTC
[GitHub] [incubator-iotdb] vesense commented on a change in pull request #1189: Comparison IoTDB with other TSDBs
vesense commented on a change in pull request #1189:
URL: https://github.com/apache/incubator-iotdb/pull/1189#discussion_r422938060
##########
File path: docs/UserGuide/9-Comparison/TSDB-Comparison.md
##########
@@ -0,0 +1,400 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+
+## Known Time Series Database
+
+As the time series data is more and more important,
+several open sourced time series databases are intorduced in the world.
+However, few of them are developed for IoT or IIoT (Industrial IoT) scenario in particular.
+
+
+We choose 3 kinds of TSDBs here.
+
+* InfluxDB - Native Time series database
+
+ InfluxDB is one of the most popular TSDBs.
+
+ Interface: InfluxQL and HTTP API
+
+* OpenTSDB and KairosDB - Time series database based on NoSQL
+
+ These two DBs are similar, while the first is based on HBase and the second is based on Cassandra.
+ Both of them provides RESTful style API.
+
+ Interface: Restful API
+
+* TimeSacleDB - Time series database based on Relational Database
+
+ Interface: SQL
+
+Prometheus and Druid are also famous for time series data management.
+However, Prometheus focuses on how to collect data, how to visualize data and how to alert warnings.
+Druid focuses on how to analyze data with OLAP workload. We omit them here.
+
+
+## Comparison
+We compare the above time series database from two aspects: the feature comparison and the performance
+comparison.
+
+
+### Feature Comparison
+
+I list the basic features comparison of these databases.
+
+Legend:
+- O: big support greatly
+- o: support
+- x: not support
+- :\-( : support but not very good
+- ?: unknown
+
+
+#### Basic Features
+
+| TSDB | IoTDB | InfluxDB | OpenTSDB | KairosDB | TimescaleDB |
+|-----------------------------|-----------------------------|------------|------------|------------|-------------|
+| OpenSource | **o** | o | o | **o** | o |
+| SQL\-like | o | o | x | x | **O** |
+| Schema | "Tree\-based, tag\-based\" | tag\-based | tag\-based | tag\-based | Relational |
+| Writing out\-of\-order data | o | o | o | o | o |
+| Schema\-less | o | o | o | o | o |
+| Batch insertion | o | o | o | o | o |
+| Time range filter | o | o | o | o | o |
+| Order by time | **O** | o | x | x | o |
+| Value filter | o | o | x | x | o |
+| Downsampling | **O** | o | o | o | o |
+| Fill | **O** | o | o | x | o |
+| LIMIT | o | o | o | o | o |
+| SLIMIT | o | o | x | x | ? |
+| Latest value | O | o | o | x | o |
+
+**Details**
+
+* OpenSOurce:
+
+ * IoTDB uses Apache License 2.0 and it is in Apache incubator.
+ * InfluxDB uses MIT license. However, **the cluster version is not open sourced**.
Review comment:
How about adding a separate section/item to highlight distributed functions?
This is an important advantage of iotdb over others.
##########
File path: docs/UserGuide/9-Comparison/TSDB-Comparison.md
##########
@@ -0,0 +1,400 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+
+## Known Time Series Database
+
+As the time series data is more and more important,
+several open sourced time series databases are intorduced in the world.
+However, few of them are developed for IoT or IIoT (Industrial IoT) scenario in particular.
+
+
+We choose 3 kinds of TSDBs here.
+
+* InfluxDB - Native Time series database
+
+ InfluxDB is one of the most popular TSDBs.
+
+ Interface: InfluxQL and HTTP API
+
+* OpenTSDB and KairosDB - Time series database based on NoSQL
+
+ These two DBs are similar, while the first is based on HBase and the second is based on Cassandra.
+ Both of them provides RESTful style API.
+
+ Interface: Restful API
+
+* TimeSacleDB - Time series database based on Relational Database
+
+ Interface: SQL
+
+Prometheus and Druid are also famous for time series data management.
+However, Prometheus focuses on how to collect data, how to visualize data and how to alert warnings.
+Druid focuses on how to analyze data with OLAP workload. We omit them here.
+
+
+## Comparison
+We compare the above time series database from two aspects: the feature comparison and the performance
+comparison.
+
+
+### Feature Comparison
+
+I list the basic features comparison of these databases.
+
+Legend:
+- O: big support greatly
+- o: support
+- x: not support
+- :\-( : support but not very good
+- ?: unknown
+
+
+#### Basic Features
+
+| TSDB | IoTDB | InfluxDB | OpenTSDB | KairosDB | TimescaleDB |
+|-----------------------------|-----------------------------|------------|------------|------------|-------------|
+| OpenSource | **o** | o | o | **o** | o |
+| SQL\-like | o | o | x | x | **O** |
+| Schema | "Tree\-based, tag\-based\" | tag\-based | tag\-based | tag\-based | Relational |
+| Writing out\-of\-order data | o | o | o | o | o |
+| Schema\-less | o | o | o | o | o |
+| Batch insertion | o | o | o | o | o |
+| Time range filter | o | o | o | o | o |
+| Order by time | **O** | o | x | x | o |
+| Value filter | o | o | x | x | o |
+| Downsampling | **O** | o | o | o | o |
+| Fill | **O** | o | o | x | o |
+| LIMIT | o | o | o | o | o |
+| SLIMIT | o | o | x | x | ? |
+| Latest value | O | o | o | x | o |
+
+**Details**
+
+* OpenSOurce:
+
+ * IoTDB uses Apache License 2.0 and it is in Apache incubator.
+ * InfluxDB uses MIT license. However, **the cluster version is not open sourced**.
+ * OpenTSDB uses LGPL2.1, which **is not compatible with Apache License**.
+ * KairosDB uses Apache License 2.0.
+ * TimescaleDB uses Timescale License, which is not free for enterprise.
+
+* SQL like:
+
+ * IoTDB and InfluxDB supports SQL like language. Besides, The integration of IoTDB and Calcite is alomost done (a PR has been submitted), which means IoTDB will support Standard SQL.
+ * OpenTSDB and KairosDB only support Rest API. Besides, IoTDB also supports Rest API (a PR has been submitted).
+ * TimescaleDB uses the SQL the same with PG.
+
+* Schema:
+
+ * IoTDB: IoTDB proposes a [Tree based schema](http://iotdb.apache.org/UserGuide/Master/2-Concept/1-Data%20Model%20and%20Terminology.html#data-model-and-terminology).
+ It is quite different with other TSDBs. However, the kind of schema has the following advantages:
+
+ * In many industrial scenarios, the management of devices are hierarchical, rather than flat.
+ That is why we think a tree based schema is better than tag-value based schema.
+
+ * In many real world applications, tag names are constant. For example, a wind turbine manufacturer
+ always identify their wind turbines by which country it locates, the farm name it belongs to, and it ID in the farm.
+ So, a 4-depth tree ("root.the-country.the-farm.the-id") is fine.
+ You do not need to repeat to tell IoTDB the 2nd level of the tree is for country name,
+ the 3rd level is for farm id, etc..
+
+ * A path based time series ID definition also supports flexible queries, like "root.\*.a.b.\*", wehre \* is wildcard character.
+
+ * InfluxDB, KairosDB, OpenTSDB are tag-value based, which is more popular currently.
+
+ * TimescaleDB uses relational table.
+
+* Order by time:
+
+ Order by time seems quite trivil for time series database. But... if we consider another featuer, called align by time,
+ something becomes interesting. And, that is why we mark OpenTSDB and KairosDB unsupported.
+
+ Actually, in each time series, all these TSDBs support order data by timestamps.
+
+ However, OpenTSDB and KairosDB do not support order the data from different timeseries in the time order.
+
+ Ok, considering a new case: I have two time series, one is for the wind speed in wind farm1,
+ another is for the generated energy of wind turbine1 in farm1. If we want to analyze the relation between the
+ wind speed and the generated energy, we have to know the values of both at the same time.
+ That is to say, we have to align the two time series in the time dimension.
+
+ So, the result should be:
+
+ | timestamp | wind speed | generated energy |
+ |-----------|-------------|------------------|
+ | 1 | 5.0 | 13.1 |
+ | 2 | 6.0 | 13.3 |
+ | 3 | null | 13.1 |
+
+ or,
+
+ | timestamp | series name | value |
+ |-----------|-------------------|------------|
+ | 1 | wind speed | 5.0 |
+ | 1 | generated energy | 13.1 |
+ | 2 | wind speed | 6.0 |
+ | 2 | generated energy | 13.3 |
+ | 3 | generated energy | 13.1 |
+
+ Though the second table format does not algin data by the time dimension, but it is easy to be implemented in the client-side,
+ by justing scanning data row by row.
+
+ IoTDB supports the first table format (called align by time), InfluxDB supports the second table format.
+
+* Downsampling:
+
+ Downsampling is for changing the granularity of timeseries, e.g., from 10Hz to 1Hz, or 1 point per day.
+
+ Different with other systems, IoTDB downsamples data in real time, while others serialized downsampled data on disk.
+ That is to say,
+
+ * IoTDB supports **adhoc** downsampling data in **arbitrary time**.
+ e.g., a SQL returns 1 point per 5 minutes and start with 2020-04-27 08:00:00 while another SQL returns 1 point per 5 minutes + 10 seconds and start with 2020-04-27 08:00:01.
+ （InfluxDB also supports adhoc downsampling but the performance is ..... hm)
+
+ * There is no disk loss for IoTDB.
+
+
+* Fill:
+
+ Sometimes we thought the data is collected in some fixed frequency, e.g., 1Hz (1 point per second).
+ But usually, we may lost some data points, because the network is unstalbe, the machine is busy, or the machine is down for several minutes.
+
+ In this case, filling these holes is important. Data scientists can avoid to many so called dirty work, e.g., data clean.
+
+ InfluxDB and OpenTSDB only support using fill in a group by statement, while IoTDB supports to fill data when just given a particular timestamp.
+ Besides, IoTDB supports several strategies for filling data.
+
+* Slimit:
+
+ Slimit means return limited number of measurements (or, fields in InfluxDB).
+ For example, a wind turbine may have 1000 measurements (speed, voltage, etc..), using slimit and soffset can just return a part of them.
+
+
+* Latest value:
+
+ As one of the most basic timeseries based applications is monitoring the latest data.
+ Therefore, a query to return the latest value of a time series is very important.
+ IoTDB and OpenTSDB support that with a special SQL or API,
+ while InfluxDB supports that using an aggregation function.
+ (the reason why IoTDB porvides a special SQL is IoTDB optimizes the query expressly.)
+
+
+
+**Conclusion**:
+
+Well, if we compare the basic features, we can find that OpenTSDB and KairosDB somehow lack some important query features.
+TimescaleDB can not be freely used in business.
+IoTDB and InfluxDB can meet most requirements of time series data management, while they have some difference.
+
+
+#### Advanced Features
+
+I listed some interesting features that these systems may differ.
+
+| TSDB | IoTDB | InfluxDB | OpenTSDB | KairosDB | TimescaleDB |
+|-----------------------------|---------------------------------|------------|------------|------------|-------------|
+| Align by time | **O** | o | x | x | o |
+| Compression | **O** | :\-( | :\-\( | :\-\( | :\-\( |
+| MQTT support | **O** | o | x | x | :\-\( |
+| Run on Edge-side Device | **O** | o | x | :\-\( | o |
+| Multi\-instance Sync | **O** | x | x | x | x |
+| JDBC Driver | **o** | x | x | x | x |
+| Standard SQL | o | x | x | x | **O** |
+| Spark integration | **O** | x | x | x | x |
Review comment:
add flink integration
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org