You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iotdb.apache.org by ja...@apache.org on 2023/01/13 12:26:32 UTC
[iotdb] branch master updated: [IOTDB-5372] Support data type cast in SELECT INTO
This is an automated email from the ASF dual-hosted git repository.
jackietien pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/iotdb.git
The following commit(s) were added to refs/heads/master by this push:
new a387d290f2 [IOTDB-5372] Support data type cast in SELECT INTO
a387d290f2 is described below
commit a387d290f2f031f22981a95ccad0f7357bfb8dcc
Author: liuminghui233 <36...@users.noreply.github.com>
AuthorDate: Fri Jan 13 20:26:25 2023 +0800
[IOTDB-5372] Support data type cast in SELECT INTO
---
docs/UserGuide/Data-Concept/Data-Type.md | 25 ++-
docs/UserGuide/Data-Concept/Encoding.md | 29 +--
docs/UserGuide/Query-Data/Select-Into.md | 2 +-
docs/zh/UserGuide/Data-Concept/Data-Type.md | 33 ++-
docs/zh/UserGuide/Data-Concept/Encoding.md | 33 +--
docs/zh/UserGuide/Query-Data/Select-Into.md | 2 +-
.../iotdb/db/it/selectinto/IoTDBSelectIntoIT.java | 237 +++++++++++++++++----
.../org/apache/iotdb/db/it/utils/TestUtils.java | 14 ++
.../operator/process/AbstractIntoOperator.java | 47 ++--
.../operator/process/DeviceViewIntoOperator.java | 8 +-
.../execution/operator/process/IntoOperator.java | 10 +-
.../iotdb/db/mpp/plan/analyze/AnalyzeVisitor.java | 26 +++
.../iotdb/db/mpp/plan/analyze/SelectIntoUtils.java | 63 ++++++
.../db/mpp/plan/planner/OperatorTreeGenerator.java | 30 ++-
.../parameter/DeviceViewIntoPathDescriptor.java | 45 +++-
.../planner/plan/parameter/IntoPathDescriptor.java | 27 ++-
.../apache/iotdb/db/utils/TypeInferenceUtils.java | 37 +++-
17 files changed, 539 insertions(+), 129 deletions(-)
diff --git a/docs/UserGuide/Data-Concept/Data-Type.md b/docs/UserGuide/Data-Concept/Data-Type.md
index d36a10012d..f45b601200 100644
--- a/docs/UserGuide/Data-Concept/Data-Type.md
+++ b/docs/UserGuide/Data-Concept/Data-Type.md
@@ -32,18 +32,33 @@ IoTDB supports the following data types:
* DOUBLE (Double Precision Floating Point)
* TEXT (String)
+### Float Precision
The time series of **FLOAT** and **DOUBLE** type can specify (MAX\_POINT\_NUMBER, see [this page](../Reference/SQL-Reference.md) for more information on how to specify), which is the number of digits after the decimal point of the floating point number, if the encoding method is [RLE](Encoding.md) or [TS\_2DIFF](Encoding.md). If MAX\_POINT\_NUMBER is not specified, the system will use [float\_precision](../Reference/DataNode-Config-Manual.md) in the configuration file `iotdb-common.properties`.
+```sql
+CREATE TIMESERIES root.vehicle.d0.s0 WITH DATATYPE=FLOAT, ENCODING=RLE, 'MAX_POINT_NUMBER'='2';
+```
+
* For Float data value, The data range is (-Integer.MAX_VALUE, Integer.MAX_VALUE), rather than Float.MAX_VALUE, and the max_point_number is 19, caused by the limition of function Math.round(float) in Java.
* For Double data value, The data range is (-Long.MAX_VALUE, Long.MAX_VALUE), rather than Double.MAX_VALUE, and the max_point_number is 19, caused by the limition of function Math.round(double) in Java (Long.MAX_VALUE=9.22E18).
-When the data type of data input by the user in the system does not correspond to the data type of the time series, the system will report type errors. As shown below, the second-order difference encoding does not support the Boolean type:
+### Data Type Compatibility
-```
-IoTDB> create timeseries root.ln.wf02.wt02.status WITH DATATYPE=BOOLEAN, ENCODING=TS_2DIFF
-error: encoding TS_2DIFF does not support BOOLEAN
-```
+When the written data type is inconsistent with the data type of time-series,
+- If the data type of time-series is not compatible with the written data type, the system will give an error message.
+- If the data type of time-series is compatible with the written data type, the system will automatically convert the data type.
+
+The compatibility of each data type is shown in the following table:
+
+| Series Data Type | Supported Written Data Types |
+|------------------|------------------------------|
+| BOOLEAN | BOOLEAN |
+| INT32 | INT32 |
+| INT64 | INT32 INT64 |
+| FLOAT | INT32 FLOAT |
+| DOUBLE | INT32 INT64 FLOAT DOUBLE |
+| TEXT | TEXT |
## Timestamp
diff --git a/docs/UserGuide/Data-Concept/Encoding.md b/docs/UserGuide/Data-Concept/Encoding.md
index 11f2c919c0..21306e4bab 100644
--- a/docs/UserGuide/Data-Concept/Encoding.md
+++ b/docs/UserGuide/Data-Concept/Encoding.md
@@ -66,19 +66,24 @@ ZIGZAG encoding maps signed integers to unsigned integers so that numbers with a
## Correspondence between data type and encoding
-The five encodings described in the previous sections are applicable to different data types. If the correspondence is wrong, the time series cannot be created correctly. The correspondence between the data type and its supported encodings is summarized in the Table below.
+The five encodings described in the previous sections are applicable to different data types. If the correspondence is wrong, the time series cannot be created correctly.
-<div style="text-align: center;">
+The correspondence between the data type and its supported encodings is summarized in the Table below.
-**The correspondence between the data type and its supported encodings**
+| Data Type | Supported Encoding |
+|:---------:|:-------------------------------------------:|
+| BOOLEAN | PLAIN, RLE |
+| INT32 | PLAIN, RLE, TS_2DIFF, GORILLA, FREQ, ZIGZAG |
+| INT64 | PLAIN, RLE, TS_2DIFF, GORILLA, FREQ, ZIGZAG |
+| FLOAT | PLAIN, RLE, TS_2DIFF, GORILLA, FREQ |
+| DOUBLE | PLAIN, RLE, TS_2DIFF, GORILLA, FREQ |
+| TEXT | PLAIN, DICTIONARY |
-|Data Type |Supported Encoding|
-|:---:|:---:|
-|BOOLEAN| PLAIN, RLE|
-|INT32 |PLAIN, RLE, TS_2DIFF, GORILLA, FREQ, ZIGZAG|
-|INT64 |PLAIN, RLE, TS_2DIFF, GORILLA, FREQ, ZIGZAG|
-|FLOAT |PLAIN, RLE, TS_2DIFF, GORILLA, FREQ|
-|DOUBLE |PLAIN, RLE, TS_2DIFF, GORILLA, FREQ|
-|TEXT |PLAIN, DICTIONARY|
+When the data type specified by the user does not correspond to the encoding method, the system will prompt an error.
-</div>
+As shown below, the second-order difference encoding does not support the Boolean type:
+
+```
+IoTDB> create timeseries root.ln.wf02.wt02.status WITH DATATYPE=BOOLEAN, ENCODING=TS_2DIFF
+Msg: 507: encoding TS_2DIFF does not support BOOLEAN
+```
diff --git a/docs/UserGuide/Query-Data/Select-Into.md b/docs/UserGuide/Query-Data/Select-Into.md
index 4b1a28fcc9..da9005051c 100644
--- a/docs/UserGuide/Query-Data/Select-Into.md
+++ b/docs/UserGuide/Query-Data/Select-Into.md
@@ -269,7 +269,7 @@ This statement specifies that `root.sg_copy.d1` is an unaligned device and `root
### Other points to note
- For general aggregation queries, the timestamp is meaningless, and the convention is to use 0 to store.
-- When the target time series exists, the metadata information such as the data type, compression, encoding, and whether it belongs to the aligned device of the source time series and the target time series must be consistent.
+- When the target time-series exists, the data type of the source column and the target time-series must be compatible. About data type compatibility, see the document [Data Type](../Data-Concept/Data-Type.md#Data Type Compatibility).
- When the target time series does not exist, the system automatically creates it (including the database).
- When the queried time series does not exist, or the queried sequence does not have data, the target time series will not be created automatically.
diff --git a/docs/zh/UserGuide/Data-Concept/Data-Type.md b/docs/zh/UserGuide/Data-Concept/Data-Type.md
index 0c2aae7f3f..e451bd094e 100644
--- a/docs/zh/UserGuide/Data-Concept/Data-Type.md
+++ b/docs/zh/UserGuide/Data-Concept/Data-Type.md
@@ -23,7 +23,7 @@
## 基本数据类型
-IoTDB 支持:
+IoTDB 支持以下六种数据类型:
* BOOLEAN(布尔值)
* INT32(整型)
@@ -32,17 +32,34 @@ IoTDB 支持:
* DOUBLE(双精度浮点数)
* TEXT(字符串)
-一共六种数据类型。
+### 浮点数精度配置
-其中 **FLOAT** 与 **DOUBLE** 类型的序列,如果编码方式采用 [RLE](Encoding.md) 或 [TS_2DIFF](Encoding.md) 可以指定 MAX_POINT_NUMBER,该项为浮点数的小数点后位数,若不指定则系统会根据配置文件`iotdb-common.properties`文件中的 [float_precision 项](../Reference/DataNode-Config-Manual.md) 配置。
+对于 **FLOAT** 与 **DOUBLE** 类型的序列,如果编码方式采用 [RLE](Encoding.md) 或 [TS_2DIFF](Encoding.md) ,可以在创建序列时通过 `MAX_POINT_NUMBER` 属性指定浮点数的小数点后位数。
-当系统中用户输入的数据类型与该时间序列的数据类型不对应时,系统会提醒类型错误,如下所示,二阶差分编码不支持布尔类型:
-
-```
-IoTDB> create timeseries root.ln.wf02.wt02.status WITH DATATYPE=BOOLEAN, ENCODING=TS_2DIFF
-error: encoding TS_2DIFF does not support BOOLEAN
+例如,
+```sql
+CREATE TIMESERIES root.vehicle.d0.s0 WITH DATATYPE=FLOAT, ENCODING=RLE, 'MAX_POINT_NUMBER'='2';
```
+若不指定,系统会按照配置文件 `iotdb-common.properties` 中的 [float_precision](../Reference/Common-Config-Manual.md) 项配置(默认为 2 位)。
+
+### 数据类型兼容性
+
+当写入数据的类型与序列注册的数据类型不一致时,
+- 如果序列数据类型不兼容写入数据类型,系统会给出错误提示。
+- 如果序列数据类型兼容写入数据类型,系统会进行数据类型的自动转换,将写入的数据类型更正为注册序列的类型。
+
+各数据类型的兼容情况如下表所示:
+
+| 序列数据类型 | 支持的写入数据类型 |
+|--------------|--------------------------|
+| BOOLEAN | BOOLEAN |
+| INT32 | INT32 |
+| INT64 | INT32 INT64 |
+| FLOAT | INT32 FLOAT |
+| DOUBLE | INT32 INT64 FLOAT DOUBLE |
+| TEXT | TEXT |
+
## 时间戳类型
时间戳是一个数据到来的时间点,其中包括绝对时间戳和相对时间戳。
diff --git a/docs/zh/UserGuide/Data-Concept/Encoding.md b/docs/zh/UserGuide/Data-Concept/Encoding.md
index 7671971317..475f3a13b5 100644
--- a/docs/zh/UserGuide/Data-Concept/Encoding.md
+++ b/docs/zh/UserGuide/Data-Concept/Encoding.md
@@ -65,19 +65,20 @@ ZigZag编码将有符号整型映射到无符号整型,适合比较小的整
## 数据类型与编码的对应关系
-前文介绍的五种编码适用于不同的数据类型,若对应关系错误,则无法正确创建时间序列。数据类型与支持其编码的编码方式对应关系总结如表格 2-3。
-
-<div style="text-align: center;">
-
-**表格 2-3 数据类型与支持其编码的对应关系**
-
-|数据类型 |支持的编码|
-|:---:|:---:|
-|BOOLEAN| PLAIN, RLE|
-|INT32 |PLAIN, RLE, TS_2DIFF, GORILLA, FREQ, ZIGZAG|
-|INT64 |PLAIN, RLE, TS_2DIFF, GORILLA, FREQ, ZIGZAG|
-|FLOAT |PLAIN, RLE, TS_2DIFF, GORILLA, FREQ|
-|DOUBLE |PLAIN, RLE, TS_2DIFF, GORILLA, FREQ|
-|TEXT |PLAIN, DICTIONARY|
-
-</div>
+前文介绍的五种编码适用于不同的数据类型,若对应关系错误,则无法正确创建时间序列。数据类型与支持其编码的编码方式对应关系总结如下表所示。
+
+| 数据类型 | 支持的编码 |
+|-:-:------|-:-:-----------------------------------------|
+| BOOLEAN | PLAIN, RLE |
+| INT32 | PLAIN, RLE, TS_2DIFF, GORILLA, FREQ, ZIGZAG |
+| INT64 | PLAIN, RLE, TS_2DIFF, GORILLA, FREQ, ZIGZAG |
+| FLOAT | PLAIN, RLE, TS_2DIFF, GORILLA, FREQ |
+| DOUBLE | PLAIN, RLE, TS_2DIFF, GORILLA, FREQ |
+| TEXT | PLAIN, DICTIONARY |
+
+当用户输入的数据类型与编码方式不对应时,系统会提示错误。如下所示,二阶差分编码不支持布尔类型:
+
+```
+IoTDB> create timeseries root.ln.wf02.wt02.status WITH DATATYPE=BOOLEAN, ENCODING=TS_2DIFF
+Msg: 507: encoding TS_2DIFF does not support BOOLEAN
+```
\ No newline at end of file
diff --git a/docs/zh/UserGuide/Query-Data/Select-Into.md b/docs/zh/UserGuide/Query-Data/Select-Into.md
index f39206048c..5ed15b0d50 100644
--- a/docs/zh/UserGuide/Query-Data/Select-Into.md
+++ b/docs/zh/UserGuide/Query-Data/Select-Into.md
@@ -270,7 +270,7 @@ select s1, s2 into root.sg_copy.d1(t1, t2), aligned root.sg_copy.d2(t1, t2) from
### 其他要注意的点
- 对于一般的聚合查询,时间戳是无意义的,约定使用 0 来存储。
-- 当目标序列存在时,需要保证源序列和目标时间序列的数据类型、压缩和编码方式、是否属于对齐设备等元数据信息一致。
+- 当目标序列存在时,需要保证源序列和目标时间序列的数据类型兼容。关于数据类型的兼容性,查看文档 [数据类型](../Data-Concept/Data-Type.md#数据类型兼容性)。
- 当目标序列不存在时,系统将自动创建目标序列(包括 database)。
- 当查询的序列不存在或查询的序列不存在数据,则不会自动创建目标序列。
diff --git a/integration-test/src/test/java/org/apache/iotdb/db/it/selectinto/IoTDBSelectIntoIT.java b/integration-test/src/test/java/org/apache/iotdb/db/it/selectinto/IoTDBSelectIntoIT.java
index 50ef7441e6..3712ce3b9b 100644
--- a/integration-test/src/test/java/org/apache/iotdb/db/it/selectinto/IoTDBSelectIntoIT.java
+++ b/integration-test/src/test/java/org/apache/iotdb/db/it/selectinto/IoTDBSelectIntoIT.java
@@ -23,6 +23,7 @@ import org.apache.iotdb.it.env.EnvFactory;
import org.apache.iotdb.it.framework.IoTDBTestRunner;
import org.apache.iotdb.itbase.category.ClusterIT;
import org.apache.iotdb.itbase.category.LocalStandaloneIT;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
import org.junit.AfterClass;
import org.junit.Assert;
@@ -34,6 +35,9 @@ import org.junit.runner.RunWith;
import java.sql.Connection;
import java.sql.SQLException;
import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
import static org.apache.iotdb.db.it.utils.TestUtils.assertTestFail;
import static org.apache.iotdb.db.it.utils.TestUtils.executeNonQuery;
@@ -45,41 +49,61 @@ import static org.junit.Assert.fail;
@Category({LocalStandaloneIT.class, ClusterIT.class})
public class IoTDBSelectIntoIT {
- protected static final String[] SQLs =
- new String[] {
- "CREATE DATABASE root.sg",
- "CREATE TIMESERIES root.sg.d1.s1 WITH DATATYPE=INT32, ENCODING=RLE",
- "CREATE TIMESERIES root.sg.d1.s2 WITH DATATYPE=FLOAT, ENCODING=RLE",
- "CREATE TIMESERIES root.sg.d2.s1 WITH DATATYPE=INT32, ENCODING=RLE",
- "CREATE TIMESERIES root.sg.d2.s2 WITH DATATYPE=FLOAT, ENCODING=RLE",
- "INSERT INTO root.sg.d1(time, s1) VALUES (1, 1)",
- "INSERT INTO root.sg.d1(time, s1, s2) VALUES (2, 2, 2)",
- "INSERT INTO root.sg.d1(time, s1, s2) VALUES (3, 3, 3)",
- "INSERT INTO root.sg.d1(time, s2) VALUES (4, 4)",
- "INSERT INTO root.sg.d1(time, s1) VALUES (5, 5)",
- "INSERT INTO root.sg.d1(time, s1, s2) VALUES (6, 6, 6)",
- "INSERT INTO root.sg.d1(time, s1, s2) VALUES (7, 7, 7)",
- "INSERT INTO root.sg.d1(time, s1, s2) VALUES (8, 8, 8)",
- "INSERT INTO root.sg.d1(time, s2) VALUES (9, 9)",
- "INSERT INTO root.sg.d1(time, s1) VALUES (10, 10)",
- "INSERT INTO root.sg.d1(time, s1, s2) VALUES (11, 11, 11)",
- "INSERT INTO root.sg.d1(time, s1, s2) VALUES (12, 12, 12)",
- "INSERT INTO root.sg.d2(time, s1, s2) VALUES (1, 1, 1)",
- "INSERT INTO root.sg.d2(time, s2) VALUES (2, 2)",
- "INSERT INTO root.sg.d2(time, s1) VALUES (3, 3)",
- "INSERT INTO root.sg.d2(time, s1, s2) VALUES (4, 4, 4)",
- "INSERT INTO root.sg.d2(time, s2) VALUES (5, 5)",
- "INSERT INTO root.sg.d2(time, s2) VALUES (6, 6)",
- "INSERT INTO root.sg.d2(time, s1) VALUES (7, 7)",
- "INSERT INTO root.sg.d2(time, s1, s2) VALUES (8, 8, 8)",
- "INSERT INTO root.sg.d2(time, s1, s2) VALUES (10, 10, 10)",
- "INSERT INTO root.sg.d2(time, s2) VALUES (11, 11)",
- "INSERT INTO root.sg.d2(time, s1) VALUES (12, 12)",
- "flush",
- "CREATE DATABASE root.sg1",
- "CREATE TIMESERIES root.sg1.d1.s1 WITH DATATYPE=INT32, ENCODING=RLE",
- "CREATE TIMESERIES root.sg1.d1.s2 WITH DATATYPE=FLOAT, ENCODING=RLE"
- };
+ protected static final List<String> SQLs =
+ new ArrayList<>(
+ Arrays.asList(
+ "CREATE DATABASE root.sg",
+ "CREATE TIMESERIES root.sg.d1.s1 WITH DATATYPE=INT32, ENCODING=RLE",
+ "CREATE TIMESERIES root.sg.d1.s2 WITH DATATYPE=FLOAT, ENCODING=RLE",
+ "CREATE TIMESERIES root.sg.d2.s1 WITH DATATYPE=INT32, ENCODING=RLE",
+ "CREATE TIMESERIES root.sg.d2.s2 WITH DATATYPE=FLOAT, ENCODING=RLE",
+ "INSERT INTO root.sg.d1(time, s1) VALUES (1, 1)",
+ "INSERT INTO root.sg.d1(time, s1, s2) VALUES (2, 2, 2)",
+ "INSERT INTO root.sg.d1(time, s1, s2) VALUES (3, 3, 3)",
+ "INSERT INTO root.sg.d1(time, s2) VALUES (4, 4)",
+ "INSERT INTO root.sg.d1(time, s1) VALUES (5, 5)",
+ "INSERT INTO root.sg.d1(time, s1, s2) VALUES (6, 6, 6)",
+ "INSERT INTO root.sg.d1(time, s1, s2) VALUES (7, 7, 7)",
+ "INSERT INTO root.sg.d1(time, s1, s2) VALUES (8, 8, 8)",
+ "INSERT INTO root.sg.d1(time, s2) VALUES (9, 9)",
+ "INSERT INTO root.sg.d1(time, s1) VALUES (10, 10)",
+ "INSERT INTO root.sg.d1(time, s1, s2) VALUES (11, 11, 11)",
+ "INSERT INTO root.sg.d1(time, s1, s2) VALUES (12, 12, 12)",
+ "INSERT INTO root.sg.d2(time, s1, s2) VALUES (1, 1, 1)",
+ "INSERT INTO root.sg.d2(time, s2) VALUES (2, 2)",
+ "INSERT INTO root.sg.d2(time, s1) VALUES (3, 3)",
+ "INSERT INTO root.sg.d2(time, s1, s2) VALUES (4, 4, 4)",
+ "INSERT INTO root.sg.d2(time, s2) VALUES (5, 5)",
+ "INSERT INTO root.sg.d2(time, s2) VALUES (6, 6)",
+ "INSERT INTO root.sg.d2(time, s1) VALUES (7, 7)",
+ "INSERT INTO root.sg.d2(time, s1, s2) VALUES (8, 8, 8)",
+ "INSERT INTO root.sg.d2(time, s1, s2) VALUES (10, 10, 10)",
+ "INSERT INTO root.sg.d2(time, s2) VALUES (11, 11)",
+ "INSERT INTO root.sg.d2(time, s1) VALUES (12, 12)",
+ "flush",
+ "CREATE DATABASE root.sg1",
+ "CREATE TIMESERIES root.sg1.d1.s1 WITH DATATYPE=INT32, ENCODING=RLE",
+ "CREATE TIMESERIES root.sg1.d1.s2 WITH DATATYPE=FLOAT, ENCODING=RLE"));
+
+ static {
+ SQLs.add("CREATE DATABASE root.sg_type");
+ for (int deviceId = 0; deviceId < 6; deviceId++) {
+ for (TSDataType dataType : TSDataType.values()) {
+ if (!dataType.equals(TSDataType.VECTOR)) {
+ SQLs.add(
+ String.format(
+ "CREATE TIMESERIES root.sg_type.d_%d.s_%s %s",
+ deviceId, dataType.name().toLowerCase(), dataType));
+ }
+ }
+ }
+ for (int time = 0; time < 12; time++) {
+ SQLs.add(
+ String.format(
+ "INSERT INTO root.sg_type.d_0(time, s_int32, s_int64, s_float, s_double, s_boolean, s_text) VALUES (%d, %d, %d, %f, %f, %s, 'text%d')",
+ time, time, time, (float) time, (double) time, time % 2 == 0, time));
+ }
+ }
protected static final String selectIntoHeader = "SourceColumn,TargetTimeseries,Written,";
protected static final String selectIntoAlignByDeviceHeader =
@@ -461,20 +485,12 @@ public class IoTDBSelectIntoIT {
// -------------------------------------- CHECK EXCEPTION -------------------------------------
- @Test
- public void testDataTypeInconsistent() {
- executeNonQuery("CREATE TIMESERIES root.sg_error_bk1.new_d.t1 TEXT;");
- assertTestFail(
- "select s1, s2 into root.sg_error_bk1.new_d(t1, t2, t3, t4) from root.sg.*;",
- "Fail to insert measurements [t1] caused by [data type of root.sg_error_bk1.new_d.t1 is not consistent, registered type TEXT, inserting type INT32, timestamp 1, value 1]");
- }
-
@Test
public void testAlignmentInconsistent() {
executeNonQuery("CREATE ALIGNED TIMESERIES root.sg_error_bk2.new_d(t1 INT32, t2 INT32);");
assertTestFail(
"select s1, s2 into root.sg_error_bk2.new_d(t1, t2, t3, t4) from root.sg.*;",
- "timeseries under this entity is aligned, please use createAlignedTimeseries or change entity. (Path: root.sg_error_bk2.new_d)");
+ "The specified alignment property of the target device (root.sg_error_bk2.new_d) conflicts with the actual (isAligned = true).");
}
@Test
@@ -520,4 +536,137 @@ public class IoTDBSelectIntoIT {
}
}
}
+
+ // -------------------------------------- DATATYPE CAST TEST -------------------------------------
+
+ @Test
+ public void testDataTypeIncompatible() {
+ // test INT32
+ assertTestFail(
+ "select s_int32 into root.sg_type.d_1(s_boolean) from root.sg_type.d_0;",
+ "The data type of target path (root.sg_type.d_1.s_boolean[BOOLEAN]) is not compatible with the data type of source column (root.sg_type.d_0.s_int32[INT32]).");
+ assertTestFail(
+ "select s_int32 into root.sg_type.d_1(s_text) from root.sg_type.d_0;",
+ "The data type of target path (root.sg_type.d_1.s_text[TEXT]) is not compatible with the data type of source column (root.sg_type.d_0.s_int32[INT32]).");
+
+ // test INT64
+ assertTestFail(
+ "select s_int64 into root.sg_type.d_1(s_int32) from root.sg_type.d_0;",
+ "The data type of target path (root.sg_type.d_1.s_int32[INT32]) is not compatible with the data type of source column (root.sg_type.d_0.s_int64[INT64]).");
+ assertTestFail(
+ "select s_int64 into root.sg_type.d_1(s_float) from root.sg_type.d_0;",
+ "The data type of target path (root.sg_type.d_1.s_float[FLOAT]) is not compatible with the data type of source column (root.sg_type.d_0.s_int64[INT64]).");
+ assertTestFail(
+ "select s_int64 into root.sg_type.d_1(s_boolean) from root.sg_type.d_0;",
+ "The data type of target path (root.sg_type.d_1.s_boolean[BOOLEAN]) is not compatible with the data type of source column (root.sg_type.d_0.s_int64[INT64]).");
+ assertTestFail(
+ "select s_int64 into root.sg_type.d_1(s_text) from root.sg_type.d_0;",
+ "The data type of target path (root.sg_type.d_1.s_text[TEXT]) is not compatible with the data type of source column (root.sg_type.d_0.s_int64[INT64]).");
+
+ // test FLOAT
+ assertTestFail(
+ "select s_float into root.sg_type.d_1(s_int32) from root.sg_type.d_0;",
+ "The data type of target path (root.sg_type.d_1.s_int32[INT32]) is not compatible with the data type of source column (root.sg_type.d_0.s_float[FLOAT]).");
+ assertTestFail(
+ "select s_float into root.sg_type.d_1(s_int64) from root.sg_type.d_0;",
+ "The data type of target path (root.sg_type.d_1.s_int64[INT64]) is not compatible with the data type of source column (root.sg_type.d_0.s_float[FLOAT]).");
+ assertTestFail(
+ "select s_float into root.sg_type.d_1(s_boolean) from root.sg_type.d_0;",
+ "The data type of target path (root.sg_type.d_1.s_boolean[BOOLEAN]) is not compatible with the data type of source column (root.sg_type.d_0.s_float[FLOAT]).");
+ assertTestFail(
+ "select s_float into root.sg_type.d_1(s_text) from root.sg_type.d_0;",
+ "The data type of target path (root.sg_type.d_1.s_text[TEXT]) is not compatible with the data type of source column (root.sg_type.d_0.s_float[FLOAT]).");
+
+ // test DOUBLE
+ assertTestFail(
+ "select s_double into root.sg_type.d_1(s_int32) from root.sg_type.d_0;",
+ "The data type of target path (root.sg_type.d_1.s_int32[INT32]) is not compatible with the data type of source column (root.sg_type.d_0.s_double[DOUBLE]).");
+ assertTestFail(
+ "select s_double into root.sg_type.d_1(s_int64) from root.sg_type.d_0;",
+ "The data type of target path (root.sg_type.d_1.s_int64[INT64]) is not compatible with the data type of source column (root.sg_type.d_0.s_double[DOUBLE]).");
+ assertTestFail(
+ "select s_double into root.sg_type.d_1(s_float) from root.sg_type.d_0;",
+ "The data type of target path (root.sg_type.d_1.s_float[FLOAT]) is not compatible with the data type of source column (root.sg_type.d_0.s_double[DOUBLE]).");
+ assertTestFail(
+ "select s_double into root.sg_type.d_1(s_boolean) from root.sg_type.d_0;",
+ "The data type of target path (root.sg_type.d_1.s_boolean[BOOLEAN]) is not compatible with the data type of source column (root.sg_type.d_0.s_double[DOUBLE]).");
+ assertTestFail(
+ "select s_double into root.sg_type.d_1(s_text) from root.sg_type.d_0;",
+ "The data type of target path (root.sg_type.d_1.s_text[TEXT]) is not compatible with the data type of source column (root.sg_type.d_0.s_double[DOUBLE]).");
+
+ // test BOOLEAN
+ assertTestFail(
+ "select s_boolean into root.sg_type.d_1(s_int32) from root.sg_type.d_0;",
+ "The data type of target path (root.sg_type.d_1.s_int32[INT32]) is not compatible with the data type of source column (root.sg_type.d_0.s_boolean[BOOLEAN]).");
+ assertTestFail(
+ "select s_boolean into root.sg_type.d_1(s_int64) from root.sg_type.d_0;",
+ "The data type of target path (root.sg_type.d_1.s_int64[INT64]) is not compatible with the data type of source column (root.sg_type.d_0.s_boolean[BOOLEAN]).");
+ assertTestFail(
+ "select s_boolean into root.sg_type.d_1(s_float) from root.sg_type.d_0;",
+ "The data type of target path (root.sg_type.d_1.s_float[FLOAT]) is not compatible with the data type of source column (root.sg_type.d_0.s_boolean[BOOLEAN]).");
+ assertTestFail(
+ "select s_boolean into root.sg_type.d_1(s_double) from root.sg_type.d_0;",
+ "The data type of target path (root.sg_type.d_1.s_double[DOUBLE]) is not compatible with the data type of source column (root.sg_type.d_0.s_boolean[BOOLEAN]).");
+ assertTestFail(
+ "select s_boolean into root.sg_type.d_1(s_text) from root.sg_type.d_0;",
+ "The data type of target path (root.sg_type.d_1.s_text[TEXT]) is not compatible with the data type of source column (root.sg_type.d_0.s_boolean[BOOLEAN]).");
+
+ // test TEXT
+ assertTestFail(
+ "select s_text into root.sg_type.d_1(s_int32) from root.sg_type.d_0;",
+ "The data type of target path (root.sg_type.d_1.s_int32[INT32]) is not compatible with the data type of source column (root.sg_type.d_0.s_text[TEXT]).");
+ assertTestFail(
+ "select s_text into root.sg_type.d_1(s_int64) from root.sg_type.d_0;",
+ "The data type of target path (root.sg_type.d_1.s_int64[INT64]) is not compatible with the data type of source column (root.sg_type.d_0.s_text[TEXT]).");
+ assertTestFail(
+ "select s_text into root.sg_type.d_1(s_float) from root.sg_type.d_0;",
+ "The data type of target path (root.sg_type.d_1.s_float[FLOAT]) is not compatible with the data type of source column (root.sg_type.d_0.s_text[TEXT]).");
+ assertTestFail(
+ "select s_text into root.sg_type.d_1(s_double) from root.sg_type.d_0;",
+ "The data type of target path (root.sg_type.d_1.s_double[DOUBLE]) is not compatible with the data type of source column (root.sg_type.d_0.s_text[TEXT]).");
+ assertTestFail(
+ "select s_text into root.sg_type.d_1(s_boolean) from root.sg_type.d_0;",
+ "The data type of target path (root.sg_type.d_1.s_boolean[BOOLEAN]) is not compatible with the data type of source column (root.sg_type.d_0.s_text[TEXT]).");
+ }
+
+ @Test
+ public void testDataTypeAutoCast() {
+ String[] intoRetArray =
+ new String[] {
+ "root.sg_type.d_0.s_int32,root.sg_type.d_1.s_int64,12,",
+ "root.sg_type.d_0.s_int32,root.sg_type.d_1.s_float,12,",
+ "root.sg_type.d_0.s_int32,root.sg_type.d_1.s_double,12,",
+ "root.sg_type.d_0.s_int64,root.sg_type.d_2.s_double,12,",
+ "root.sg_type.d_0.s_float,root.sg_type.d_3.s_double,12,",
+ };
+ resultSetEqualTest(
+ "select s_int32, s_int32, s_int32, s_int64, s_float "
+ + " into root.sg_type.d_1(s_int64, s_float, s_double), root.sg_type.d_2(s_double), root.sg_type.d_3(s_double) "
+ + " from root.sg_type.d_0;",
+ selectIntoHeader,
+ intoRetArray);
+
+ String expectedQueryHeader =
+ "Time,root.sg_type.d_1.s_int64,root.sg_type.d_1.s_float,root.sg_type.d_1.s_double,"
+ + "root.sg_type.d_2.s_double,root.sg_type.d_3.s_double,";
+ String[] queryRetArray =
+ new String[] {
+ "0,0,0.0,0.0,0.0,0.0,",
+ "1,1,1.0,1.0,1.0,1.0,",
+ "2,2,2.0,2.0,2.0,2.0,",
+ "3,3,3.0,3.0,3.0,3.0,",
+ "4,4,4.0,4.0,4.0,4.0,",
+ "5,5,5.0,5.0,5.0,5.0,",
+ "6,6,6.0,6.0,6.0,6.0,",
+ "7,7,7.0,7.0,7.0,7.0,",
+ "8,8,8.0,8.0,8.0,8.0,",
+ "9,9,9.0,9.0,9.0,9.0,",
+ "10,10,10.0,10.0,10.0,10.0,",
+ "11,11,11.0,11.0,11.0,11.0,"
+ };
+ resultSetEqualTest(
+ "select d_1.s_int64, d_1.s_float, d_1.s_double, d_2.s_double, d_3.s_double from root.sg_type;",
+ expectedQueryHeader,
+ queryRetArray);
+ }
}
diff --git a/integration-test/src/test/java/org/apache/iotdb/db/it/utils/TestUtils.java b/integration-test/src/test/java/org/apache/iotdb/db/it/utils/TestUtils.java
index 7aa09b3c52..0a3d5bb6b9 100644
--- a/integration-test/src/test/java/org/apache/iotdb/db/it/utils/TestUtils.java
+++ b/integration-test/src/test/java/org/apache/iotdb/db/it/utils/TestUtils.java
@@ -31,6 +31,7 @@ import java.sql.Statement;
import java.text.DateFormat;
import java.util.HashMap;
import java.util.HashSet;
+import java.util.List;
import java.util.Map;
import java.util.Set;
@@ -55,6 +56,19 @@ public class TestUtils {
}
}
+ public static void prepareData(List<String> sqls) {
+ try (Connection connection = EnvFactory.getEnv().getConnection();
+ Statement statement = connection.createStatement()) {
+ for (String sql : sqls) {
+ statement.addBatch(sql);
+ }
+ statement.executeBatch();
+ } catch (SQLException e) {
+ e.printStackTrace();
+ fail(e.getMessage());
+ }
+ }
+
public static void resultSetEqualTest(String sql, double[][] retArray, String[] columnNames) {
try (Connection connection = EnvFactory.getEnv().getConnection();
Statement statement = connection.createStatement()) {
diff --git a/server/src/main/java/org/apache/iotdb/db/mpp/execution/operator/process/AbstractIntoOperator.java b/server/src/main/java/org/apache/iotdb/db/mpp/execution/operator/process/AbstractIntoOperator.java
index 0d08a5467b..0b5c189c76 100644
--- a/server/src/main/java/org/apache/iotdb/db/mpp/execution/operator/process/AbstractIntoOperator.java
+++ b/server/src/main/java/org/apache/iotdb/db/mpp/execution/operator/process/AbstractIntoOperator.java
@@ -34,6 +34,8 @@ import org.apache.iotdb.tsfile.exception.write.UnSupportedDataTypeException;
import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
import org.apache.iotdb.tsfile.read.common.block.TsBlock;
import org.apache.iotdb.tsfile.read.common.block.column.Column;
+import org.apache.iotdb.tsfile.read.common.type.Type;
+import org.apache.iotdb.tsfile.read.common.type.TypeFactory;
import org.apache.iotdb.tsfile.utils.Binary;
import org.apache.iotdb.tsfile.utils.BitMap;
@@ -51,6 +53,7 @@ import java.util.Objects;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
import static com.google.common.util.concurrent.Futures.successfulAsList;
import static org.apache.iotdb.tsfile.read.common.block.TsBlockBuilderStatus.DEFAULT_MAX_TSBLOCK_SIZE_IN_BYTES;
@@ -78,16 +81,20 @@ public abstract class AbstractIntoOperator implements ProcessOperator {
private final long maxRetainedSize;
private final long maxReturnSize;
+ protected final List<Type> typeConvertors;
+
protected AbstractIntoOperator(
OperatorContext operatorContext,
Operator child,
- List<InsertTabletStatementGenerator> insertTabletStatementGenerators,
+ List<TSDataType> inputColumnTypes,
Map<String, InputLocation> sourceColumnToInputLocationMap,
ExecutorService intoOperationExecutor,
long maxStatementSize) {
this.operatorContext = operatorContext;
this.child = child;
- this.insertTabletStatementGenerators = insertTabletStatementGenerators;
+ this.typeConvertors =
+ inputColumnTypes.stream().map(TypeFactory::getType).collect(Collectors.toList());
+
this.sourceColumnToInputLocationMap = sourceColumnToInputLocationMap;
this.writeOperationExecutor = intoOperationExecutor;
@@ -202,7 +209,8 @@ public abstract class AbstractIntoOperator implements ProcessOperator {
protected static List<InsertTabletStatementGenerator> constructInsertTabletStatementGenerators(
Map<PartialPath, Map<String, InputLocation>> targetPathToSourceInputLocationMap,
Map<PartialPath, Map<String, TSDataType>> targetPathToDataTypeMap,
- Map<String, Boolean> targetDeviceToAlignedMap) {
+ Map<String, Boolean> targetDeviceToAlignedMap,
+ List<Type> sourceTypeConvertors) {
List<InsertTabletStatementGenerator> insertTabletStatementGenerators =
new ArrayList<>(targetPathToSourceInputLocationMap.size());
for (Map.Entry<PartialPath, Map<String, InputLocation>> entry :
@@ -213,7 +221,8 @@ public abstract class AbstractIntoOperator implements ProcessOperator {
targetDevice,
entry.getValue(),
targetPathToDataTypeMap.get(targetDevice),
- targetDeviceToAlignedMap.get(targetDevice.toString()));
+ targetDeviceToAlignedMap.get(targetDevice.toString()),
+ sourceTypeConvertors);
insertTabletStatementGenerators.add(generator);
}
return insertTabletStatementGenerators;
@@ -333,11 +342,14 @@ public abstract class AbstractIntoOperator implements ProcessOperator {
private final Map<String, AtomicInteger> writtenCounter;
+ private final List<Type> sourceTypeConvertors;
+
public InsertTabletStatementGenerator(
PartialPath devicePath,
Map<String, InputLocation> measurementToInputLocationMap,
Map<String, TSDataType> measurementToDataTypeMap,
- Boolean isAligned) {
+ Boolean isAligned,
+ List<Type> sourceTypeConvertors) {
this.devicePath = devicePath;
this.isAligned = isAligned;
this.measurements = measurementToInputLocationMap.keySet().toArray(new String[0]);
@@ -347,6 +359,7 @@ public abstract class AbstractIntoOperator implements ProcessOperator {
for (String measurement : measurements) {
writtenCounter.put(measurement, new AtomicInteger(0));
}
+ this.sourceTypeConvertors = sourceTypeConvertors;
this.reset();
}
@@ -393,7 +406,9 @@ public abstract class AbstractIntoOperator implements ProcessOperator {
times[rowCount] = tsBlock.getTimeByIndex(lastReadIndex);
for (int i = 0; i < measurements.length; ++i) {
- Column valueColumn = tsBlock.getValueColumns()[inputLocations[i].getValueColumnIndex()];
+ int valueColumnIndex = inputLocations[i].getValueColumnIndex();
+ Column valueColumn = tsBlock.getValueColumns()[valueColumnIndex];
+ Type sourceTypeConvertor = sourceTypeConvertors.get(valueColumnIndex);
// if the value is NULL
if (valueColumn.isNull(lastReadIndex)) {
@@ -403,24 +418,30 @@ public abstract class AbstractIntoOperator implements ProcessOperator {
bitMaps[i].unmark(rowCount);
writtenCounter.get(measurements[i]).getAndIncrement();
- switch (valueColumn.getDataType()) {
+ switch (dataTypes[i]) {
case INT32:
- ((int[]) columns[i])[rowCount] = valueColumn.getInt(lastReadIndex);
+ ((int[]) columns[i])[rowCount] =
+ sourceTypeConvertor.getInt(valueColumn, lastReadIndex);
break;
case INT64:
- ((long[]) columns[i])[rowCount] = valueColumn.getLong(lastReadIndex);
+ ((long[]) columns[i])[rowCount] =
+ sourceTypeConvertor.getLong(valueColumn, lastReadIndex);
break;
case FLOAT:
- ((float[]) columns[i])[rowCount] = valueColumn.getFloat(lastReadIndex);
+ ((float[]) columns[i])[rowCount] =
+ sourceTypeConvertor.getFloat(valueColumn, lastReadIndex);
break;
case DOUBLE:
- ((double[]) columns[i])[rowCount] = valueColumn.getDouble(lastReadIndex);
+ ((double[]) columns[i])[rowCount] =
+ sourceTypeConvertor.getDouble(valueColumn, lastReadIndex);
break;
case BOOLEAN:
- ((boolean[]) columns[i])[rowCount] = valueColumn.getBoolean(lastReadIndex);
+ ((boolean[]) columns[i])[rowCount] =
+ sourceTypeConvertor.getBoolean(valueColumn, lastReadIndex);
break;
case TEXT:
- ((Binary[]) columns[i])[rowCount] = valueColumn.getBinary(lastReadIndex);
+ ((Binary[]) columns[i])[rowCount] =
+ sourceTypeConvertor.getBinary(valueColumn, lastReadIndex);
break;
default:
throw new UnSupportedDataTypeException(
diff --git a/server/src/main/java/org/apache/iotdb/db/mpp/execution/operator/process/DeviceViewIntoOperator.java b/server/src/main/java/org/apache/iotdb/db/mpp/execution/operator/process/DeviceViewIntoOperator.java
index ebe8bbe64d..0ddd7ccc31 100644
--- a/server/src/main/java/org/apache/iotdb/db/mpp/execution/operator/process/DeviceViewIntoOperator.java
+++ b/server/src/main/java/org/apache/iotdb/db/mpp/execution/operator/process/DeviceViewIntoOperator.java
@@ -56,6 +56,7 @@ public class DeviceViewIntoOperator extends AbstractIntoOperator {
public DeviceViewIntoOperator(
OperatorContext operatorContext,
Operator child,
+ List<TSDataType> inputColumnTypes,
Map<String, Map<PartialPath, Map<String, InputLocation>>>
deviceToTargetPathSourceInputLocationMap,
Map<String, Map<PartialPath, Map<String, TSDataType>>> deviceToTargetPathDataTypeMap,
@@ -67,7 +68,7 @@ public class DeviceViewIntoOperator extends AbstractIntoOperator {
super(
operatorContext,
child,
- null,
+ inputColumnTypes,
sourceColumnToInputLocationMap,
intoOperationExecutor,
maxStatementSize);
@@ -144,7 +145,10 @@ public class DeviceViewIntoOperator extends AbstractIntoOperator {
Map<PartialPath, Map<String, TSDataType>> targetPathToDataTypeMap =
deviceToTargetPathDataTypeMap.get(currentDevice);
return constructInsertTabletStatementGenerators(
- targetPathToSourceInputLocationMap, targetPathToDataTypeMap, targetDeviceToAlignedMap);
+ targetPathToSourceInputLocationMap,
+ targetPathToDataTypeMap,
+ targetDeviceToAlignedMap,
+ typeConvertors);
}
private void updateResultTsBlock() {
diff --git a/server/src/main/java/org/apache/iotdb/db/mpp/execution/operator/process/IntoOperator.java b/server/src/main/java/org/apache/iotdb/db/mpp/execution/operator/process/IntoOperator.java
index 28488d5ecb..34a8edc88a 100644
--- a/server/src/main/java/org/apache/iotdb/db/mpp/execution/operator/process/IntoOperator.java
+++ b/server/src/main/java/org/apache/iotdb/db/mpp/execution/operator/process/IntoOperator.java
@@ -45,6 +45,7 @@ public class IntoOperator extends AbstractIntoOperator {
public IntoOperator(
OperatorContext operatorContext,
Operator child,
+ List<TSDataType> inputColumnTypes,
Map<PartialPath, Map<String, InputLocation>> targetPathToSourceInputLocationMap,
Map<PartialPath, Map<String, TSDataType>> targetPathToDataTypeMap,
Map<String, Boolean> targetDeviceToAlignedMap,
@@ -55,12 +56,17 @@ public class IntoOperator extends AbstractIntoOperator {
super(
operatorContext,
child,
- constructInsertTabletStatementGenerators(
- targetPathToSourceInputLocationMap, targetPathToDataTypeMap, targetDeviceToAlignedMap),
+ inputColumnTypes,
sourceColumnToInputLocationMap,
intoOperationExecutor,
maxStatementSize);
this.sourceTargetPathPairList = sourceTargetPathPairList;
+ insertTabletStatementGenerators =
+ constructInsertTabletStatementGenerators(
+ targetPathToSourceInputLocationMap,
+ targetPathToDataTypeMap,
+ targetDeviceToAlignedMap,
+ typeConvertors);
}
@Override
diff --git a/server/src/main/java/org/apache/iotdb/db/mpp/plan/analyze/AnalyzeVisitor.java b/server/src/main/java/org/apache/iotdb/db/mpp/plan/analyze/AnalyzeVisitor.java
index 4dd8e2cb23..f7b3fa75d3 100644
--- a/server/src/main/java/org/apache/iotdb/db/mpp/plan/analyze/AnalyzeVisitor.java
+++ b/server/src/main/java/org/apache/iotdb/db/mpp/plan/analyze/AnalyzeVisitor.java
@@ -1275,6 +1275,7 @@ public class AnalyzeVisitor extends StatementVisitor<Analysis, MPPQueryContext>
intoComponent.validate(sourceDevices, sourceColumns);
DeviceViewIntoPathDescriptor deviceViewIntoPathDescriptor = new DeviceViewIntoPathDescriptor();
+ PathPatternTree targetPathTree = new PathPatternTree();
IntoComponent.IntoDeviceMeasurementIterator intoDeviceMeasurementIterator =
intoComponent.getIntoDeviceMeasurementIterator();
for (PartialPath sourceDevice : sourceDevices) {
@@ -1295,12 +1296,24 @@ public class AnalyzeVisitor extends StatementVisitor<Analysis, MPPQueryContext>
}
deviceViewIntoPathDescriptor.specifyTargetDeviceMeasurement(
sourceDevice, targetDevice, sourceColumn.toString(), targetMeasurement);
+
+ targetPathTree.appendFullPath(targetDevice, targetMeasurement);
+ deviceViewIntoPathDescriptor.recordSourceColumnDataType(
+ sourceColumn.toString(), analysis.getType(sourceColumn));
+
intoDeviceMeasurementIterator.nextMeasurement();
}
intoDeviceMeasurementIterator.nextDevice();
}
deviceViewIntoPathDescriptor.validate();
+
+ // fetch schema of target paths
+ long startTime = System.nanoTime();
+ ISchemaTree targetSchemaTree = schemaFetcher.fetchSchema(targetPathTree);
+ QueryMetricsManager.getInstance().recordPlanCost(SCHEMA_FETCHER, System.nanoTime() - startTime);
+ deviceViewIntoPathDescriptor.bindType(targetSchemaTree);
+
analysis.setDeviceViewIntoPathDescriptor(deviceViewIntoPathDescriptor);
}
@@ -1322,6 +1335,7 @@ public class AnalyzeVisitor extends StatementVisitor<Analysis, MPPQueryContext>
intoComponent.validate(sourceColumns);
IntoPathDescriptor intoPathDescriptor = new IntoPathDescriptor();
+ PathPatternTree targetPathTree = new PathPatternTree();
IntoComponent.IntoPathIterator intoPathIterator = intoComponent.getIntoPathIterator();
for (Expression sourceColumn : sourceColumns) {
PartialPath deviceTemplate = intoPathIterator.getDeviceTemplate();
@@ -1338,9 +1352,21 @@ public class AnalyzeVisitor extends StatementVisitor<Analysis, MPPQueryContext>
intoPathDescriptor.specifyTargetPath(sourceColumn.toString(), targetPath);
intoPathDescriptor.specifyDeviceAlignment(
targetPath.getDevicePath().toString(), isAlignedDevice);
+
+ targetPathTree.appendFullPath(targetPath);
+ intoPathDescriptor.recordSourceColumnDataType(
+ sourceColumn.toString(), analysis.getType(sourceColumn));
+
intoPathIterator.next();
}
intoPathDescriptor.validate();
+
+ // fetch schema of target paths
+ long startTime = System.nanoTime();
+ ISchemaTree targetSchemaTree = schemaFetcher.fetchSchema(targetPathTree);
+ QueryMetricsManager.getInstance().recordPlanCost(SCHEMA_FETCHER, System.nanoTime() - startTime);
+ intoPathDescriptor.bindType(targetSchemaTree);
+
analysis.setIntoPathDescriptor(intoPathDescriptor);
}
diff --git a/server/src/main/java/org/apache/iotdb/db/mpp/plan/analyze/SelectIntoUtils.java b/server/src/main/java/org/apache/iotdb/db/mpp/plan/analyze/SelectIntoUtils.java
index 18107823e5..46b076bc8f 100644
--- a/server/src/main/java/org/apache/iotdb/db/mpp/plan/analyze/SelectIntoUtils.java
+++ b/server/src/main/java/org/apache/iotdb/db/mpp/plan/analyze/SelectIntoUtils.java
@@ -19,15 +19,23 @@
package org.apache.iotdb.db.mpp.plan.analyze;
+import org.apache.iotdb.commons.path.MeasurementPath;
import org.apache.iotdb.commons.path.PartialPath;
import org.apache.iotdb.db.exception.sql.SemanticException;
+import org.apache.iotdb.db.mpp.common.schematree.ISchemaTree;
import org.apache.iotdb.db.mpp.plan.expression.Expression;
import org.apache.iotdb.db.mpp.plan.expression.leaf.TimeSeriesOperand;
+import org.apache.iotdb.db.utils.TypeInferenceUtils;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.utils.Pair;
import java.util.ArrayList;
+import java.util.HashMap;
import java.util.List;
+import java.util.Map;
import java.util.regex.Matcher;
+import static com.google.common.base.Preconditions.checkState;
import static org.apache.iotdb.commons.conf.IoTDBConstant.DOUBLE_COLONS;
import static org.apache.iotdb.commons.conf.IoTDBConstant.LEVELED_PATH_TEMPLATE_PATTERN;
@@ -102,4 +110,59 @@ public class SelectIntoUtils {
}
return true;
}
+
+ public static List<Pair<String, PartialPath>> bindTypeForSourceTargetPathPairList(
+ List<Pair<String, PartialPath>> sourceTargetPathPairList,
+ Map<String, TSDataType> sourceToDataTypeMap,
+ ISchemaTree targetSchemaTree,
+ Map<String, Boolean> targetDeviceToAlignedMap) {
+ List<Pair<String, PartialPath>> sourceTypeBoundTargetPathPairList = new ArrayList<>();
+ for (Pair<String, PartialPath> sourceTargetPathPair : sourceTargetPathPairList) {
+ String sourceColumn = sourceTargetPathPair.left;
+ TSDataType sourceColumnType = sourceToDataTypeMap.get(sourceColumn);
+
+ MeasurementPath targetPathWithSchema;
+ PartialPath targetPath = sourceTargetPathPair.right;
+ List<MeasurementPath> actualTargetPaths =
+ targetSchemaTree.searchMeasurementPaths(targetPath).left;
+ if (actualTargetPaths.isEmpty()) {
+ targetPathWithSchema = new MeasurementPath(targetPath, sourceColumnType);
+ } else {
+ checkState(actualTargetPaths.size() == 1);
+ MeasurementPath actualTargetPath = actualTargetPaths.get(0);
+ if (!TypeInferenceUtils.canAutoCast(sourceColumnType, actualTargetPath.getSeriesType())) {
+ throw new SemanticException(
+ String.format(
+ "The data type of target path (%s[%s]) is not compatible with the data type of source column (%s[%s]).",
+ targetPath, actualTargetPath.getSeriesType(), sourceColumn, sourceColumnType));
+ }
+ boolean actualTargetPathAlignment = actualTargetPath.isUnderAlignedEntity();
+ String targetDevice = targetPath.getDevice();
+ if (targetDeviceToAlignedMap.get(targetDevice) != actualTargetPathAlignment) {
+ throw new SemanticException(
+ String.format(
+ "The specified alignment property of the target device (%s) conflicts with the actual (isAligned = %s).",
+ targetDevice, actualTargetPathAlignment));
+ }
+ targetPathWithSchema = actualTargetPath;
+ }
+ sourceTypeBoundTargetPathPairList.add(new Pair<>(sourceColumn, targetPathWithSchema));
+ }
+ return sourceTypeBoundTargetPathPairList;
+ }
+
+ public static Map<PartialPath, Map<String, TSDataType>>
+ convertSourceTargetPathPairListToTargetPathDataTypeMap(
+ List<Pair<String, PartialPath>> sourceTargetPathPairList) {
+ // targetDevice -> { targetMeasurement -> dataType }
+ Map<PartialPath, Map<String, TSDataType>> targetPathToDataTypeMap = new HashMap<>();
+
+ for (Pair<String, PartialPath> sourceTargetPathPair : sourceTargetPathPairList) {
+ PartialPath targetPath = sourceTargetPathPair.right;
+ targetPathToDataTypeMap
+ .computeIfAbsent(targetPath.getDevicePath(), key -> new HashMap<>())
+ .put(targetPath.getMeasurement(), targetPath.getSeriesType());
+ }
+ return targetPathToDataTypeMap;
+ }
}
diff --git a/server/src/main/java/org/apache/iotdb/db/mpp/plan/planner/OperatorTreeGenerator.java b/server/src/main/java/org/apache/iotdb/db/mpp/plan/planner/OperatorTreeGenerator.java
index a083eb15d4..ad6f8b29ca 100644
--- a/server/src/main/java/org/apache/iotdb/db/mpp/plan/planner/OperatorTreeGenerator.java
+++ b/server/src/main/java/org/apache/iotdb/db/mpp/plan/planner/OperatorTreeGenerator.java
@@ -1486,18 +1486,18 @@ public class OperatorTreeGenerator extends PlanVisitor<Operator, LocalExecutionP
IntoOperator.class.getSimpleName());
IntoPathDescriptor intoPathDescriptor = node.getIntoPathDescriptor();
+
Map<String, InputLocation> sourceColumnToInputLocationMap =
constructSourceColumnToInputLocationMap(node);
-
Map<PartialPath, Map<String, InputLocation>> targetPathToSourceInputLocationMap =
new HashMap<>();
- Map<PartialPath, Map<String, TSDataType>> targetPathToDataTypeMap = new HashMap<>();
processTargetPathToSourceMap(
intoPathDescriptor.getTargetPathToSourceMap(),
targetPathToSourceInputLocationMap,
- targetPathToDataTypeMap,
- sourceColumnToInputLocationMap,
- context.getTypeProvider());
+ sourceColumnToInputLocationMap);
+
+ Map<PartialPath, Map<String, TSDataType>> targetPathToDataTypeMap =
+ intoPathDescriptor.getTargetPathToDataTypeMap();
int rowLimit =
IoTDBDescriptor.getInstance().getConfig().getSelectIntoInsertTabletPlanRowLimit();
@@ -1508,6 +1508,7 @@ public class OperatorTreeGenerator extends PlanVisitor<Operator, LocalExecutionP
return new IntoOperator(
operatorContext,
child,
+ getInputColumnTypes(node, context.getTypeProvider()),
targetPathToSourceInputLocationMap,
targetPathToDataTypeMap,
intoPathDescriptor.getTargetDeviceToAlignedMap(),
@@ -1536,7 +1537,7 @@ public class OperatorTreeGenerator extends PlanVisitor<Operator, LocalExecutionP
Map<String, Map<PartialPath, Map<String, InputLocation>>>
deviceToTargetPathSourceInputLocationMap = new HashMap<>();
Map<String, Map<PartialPath, Map<String, TSDataType>>> deviceToTargetPathDataTypeMap =
- new HashMap<>();
+ deviceViewIntoPathDescriptor.getSourceDeviceToTargetPathDataTypeMap();
Map<String, Map<PartialPath, Map<String, String>>> sourceDeviceToTargetPathMap =
deviceViewIntoPathDescriptor.getSourceDeviceToTargetPathMap();
long statementSizePerLine = 0L;
@@ -1545,17 +1546,14 @@ public class OperatorTreeGenerator extends PlanVisitor<Operator, LocalExecutionP
String sourceDevice = deviceEntry.getKey();
Map<PartialPath, Map<String, InputLocation>> targetPathToSourceInputLocationMap =
new HashMap<>();
- Map<PartialPath, Map<String, TSDataType>> targetPathToDataTypeMap = new HashMap<>();
processTargetPathToSourceMap(
deviceEntry.getValue(),
targetPathToSourceInputLocationMap,
- targetPathToDataTypeMap,
- sourceColumnToInputLocationMap,
- context.getTypeProvider());
+ sourceColumnToInputLocationMap);
deviceToTargetPathSourceInputLocationMap.put(
sourceDevice, targetPathToSourceInputLocationMap);
- deviceToTargetPathDataTypeMap.put(sourceDevice, targetPathToDataTypeMap);
- statementSizePerLine += calculateStatementSizePerLine(targetPathToDataTypeMap);
+ statementSizePerLine +=
+ calculateStatementSizePerLine(deviceToTargetPathDataTypeMap.get(sourceDevice));
}
int rowLimit =
@@ -1566,6 +1564,7 @@ public class OperatorTreeGenerator extends PlanVisitor<Operator, LocalExecutionP
return new DeviceViewIntoOperator(
operatorContext,
child,
+ getInputColumnTypes(node, context.getTypeProvider()),
deviceToTargetPathSourceInputLocationMap,
deviceToTargetPathDataTypeMap,
deviceViewIntoPathDescriptor.getTargetDeviceToAlignedMap(),
@@ -1587,22 +1586,17 @@ public class OperatorTreeGenerator extends PlanVisitor<Operator, LocalExecutionP
private void processTargetPathToSourceMap(
Map<PartialPath, Map<String, String>> targetPathToSourceMap,
Map<PartialPath, Map<String, InputLocation>> targetPathToSourceInputLocationMap,
- Map<PartialPath, Map<String, TSDataType>> targetPathToDataTypeMap,
- Map<String, InputLocation> sourceColumnToInputLocationMap,
- TypeProvider typeProvider) {
+ Map<String, InputLocation> sourceColumnToInputLocationMap) {
for (Map.Entry<PartialPath, Map<String, String>> entry : targetPathToSourceMap.entrySet()) {
PartialPath targetDevice = entry.getKey();
Map<String, InputLocation> measurementToInputLocationMap = new HashMap<>();
- Map<String, TSDataType> measurementToDataTypeMap = new HashMap<>();
for (Map.Entry<String, String> measurementEntry : entry.getValue().entrySet()) {
String targetMeasurement = measurementEntry.getKey();
String sourceColumn = measurementEntry.getValue();
measurementToInputLocationMap.put(
targetMeasurement, sourceColumnToInputLocationMap.get(sourceColumn));
- measurementToDataTypeMap.put(targetMeasurement, typeProvider.getType(sourceColumn));
}
targetPathToSourceInputLocationMap.put(targetDevice, measurementToInputLocationMap);
- targetPathToDataTypeMap.put(targetDevice, measurementToDataTypeMap);
}
}
diff --git a/server/src/main/java/org/apache/iotdb/db/mpp/plan/planner/plan/parameter/DeviceViewIntoPathDescriptor.java b/server/src/main/java/org/apache/iotdb/db/mpp/plan/planner/plan/parameter/DeviceViewIntoPathDescriptor.java
index fc0486af0b..25689a7132 100644
--- a/server/src/main/java/org/apache/iotdb/db/mpp/plan/planner/plan/parameter/DeviceViewIntoPathDescriptor.java
+++ b/server/src/main/java/org/apache/iotdb/db/mpp/plan/planner/plan/parameter/DeviceViewIntoPathDescriptor.java
@@ -22,6 +22,9 @@ package org.apache.iotdb.db.mpp.plan.planner.plan.parameter;
import org.apache.iotdb.commons.path.PartialPath;
import org.apache.iotdb.commons.path.PathDeserializeUtil;
import org.apache.iotdb.db.exception.sql.SemanticException;
+import org.apache.iotdb.db.mpp.common.schematree.ISchemaTree;
+import org.apache.iotdb.db.mpp.plan.analyze.SelectIntoUtils;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
import org.apache.iotdb.tsfile.utils.Pair;
import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils;
@@ -42,14 +45,18 @@ import static org.apache.iotdb.db.mpp.plan.statement.component.IntoComponent.DUP
public class DeviceViewIntoPathDescriptor {
// device -> List<(sourceColumn, targetPath)>
- private final Map<String, List<Pair<String, PartialPath>>> deviceToSourceTargetPathPairListMap;
+ private Map<String, List<Pair<String, PartialPath>>> deviceToSourceTargetPathPairListMap;
// targetDevice -> isAlignedDevice
private final Map<String, Boolean> targetDeviceToAlignedMap;
+ // sourceColumn -> dataType (not serialize & deserialize)
+ private Map<String, TSDataType> sourceToDataTypeMap;
+
public DeviceViewIntoPathDescriptor() {
this.deviceToSourceTargetPathPairListMap = new HashMap<>();
this.targetDeviceToAlignedMap = new HashMap<>();
+ this.sourceToDataTypeMap = new HashMap<>();
}
public DeviceViewIntoPathDescriptor(
@@ -77,6 +84,10 @@ public class DeviceViewIntoPathDescriptor {
targetDeviceToAlignedMap.put(targetDevice, isAligned);
}
+ public void recordSourceColumnDataType(String sourceColumn, TSDataType dataType) {
+ sourceToDataTypeMap.put(sourceColumn, dataType);
+ }
+
public void validate() {
List<PartialPath> targetPaths =
deviceToSourceTargetPathPairListMap.values().stream()
@@ -88,6 +99,22 @@ public class DeviceViewIntoPathDescriptor {
}
}
+ public void bindType(ISchemaTree targetSchemaTree) {
+ Map<String, List<Pair<String, PartialPath>>> deviceToSourceTypeBoundTargetPathPairListMap =
+ new HashMap<>();
+ for (Map.Entry<String, List<Pair<String, PartialPath>>> sourceTargetEntry :
+ this.deviceToSourceTargetPathPairListMap.entrySet()) {
+ deviceToSourceTypeBoundTargetPathPairListMap.put(
+ sourceTargetEntry.getKey(),
+ SelectIntoUtils.bindTypeForSourceTargetPathPairList(
+ sourceTargetEntry.getValue(),
+ sourceToDataTypeMap,
+ targetSchemaTree,
+ targetDeviceToAlignedMap));
+ }
+ this.deviceToSourceTargetPathPairListMap = deviceToSourceTypeBoundTargetPathPairListMap;
+ }
+
public Map<String, List<Pair<String, PartialPath>>> getDeviceToSourceTargetPathPairListMap() {
return deviceToSourceTargetPathPairListMap;
}
@@ -117,6 +144,22 @@ public class DeviceViewIntoPathDescriptor {
return sourceDeviceToTargetPathMap;
}
+ public Map<String, Map<PartialPath, Map<String, TSDataType>>>
+ getSourceDeviceToTargetPathDataTypeMap() {
+ // sourceDevice -> targetPathToDataTypeMap (for each device)
+ // targetPathToSourceMap: targetDevice -> { targetMeasurement -> dataType }
+ Map<String, Map<PartialPath, Map<String, TSDataType>>> sourceDeviceToTargetPathDataTypeMap =
+ new HashMap<>();
+ for (Map.Entry<String, List<Pair<String, PartialPath>>> sourceTargetEntry :
+ deviceToSourceTargetPathPairListMap.entrySet()) {
+ sourceDeviceToTargetPathDataTypeMap.put(
+ sourceTargetEntry.getKey(),
+ SelectIntoUtils.convertSourceTargetPathPairListToTargetPathDataTypeMap(
+ sourceTargetEntry.getValue()));
+ }
+ return sourceDeviceToTargetPathDataTypeMap;
+ }
+
public Map<String, Boolean> getTargetDeviceToAlignedMap() {
return targetDeviceToAlignedMap;
}
diff --git a/server/src/main/java/org/apache/iotdb/db/mpp/plan/planner/plan/parameter/IntoPathDescriptor.java b/server/src/main/java/org/apache/iotdb/db/mpp/plan/planner/plan/parameter/IntoPathDescriptor.java
index b71e470f01..59208e3af1 100644
--- a/server/src/main/java/org/apache/iotdb/db/mpp/plan/planner/plan/parameter/IntoPathDescriptor.java
+++ b/server/src/main/java/org/apache/iotdb/db/mpp/plan/planner/plan/parameter/IntoPathDescriptor.java
@@ -22,6 +22,9 @@ package org.apache.iotdb.db.mpp.plan.planner.plan.parameter;
import org.apache.iotdb.commons.path.PartialPath;
import org.apache.iotdb.commons.path.PathDeserializeUtil;
import org.apache.iotdb.db.exception.sql.SemanticException;
+import org.apache.iotdb.db.mpp.common.schematree.ISchemaTree;
+import org.apache.iotdb.db.mpp.plan.analyze.SelectIntoUtils;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
import org.apache.iotdb.tsfile.utils.Pair;
import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils;
@@ -42,14 +45,18 @@ import static org.apache.iotdb.db.mpp.plan.statement.component.IntoComponent.DUP
public class IntoPathDescriptor {
// List<(sourceColumn, targetPath)>
- private final List<Pair<String, PartialPath>> sourceTargetPathPairList;
+ private List<Pair<String, PartialPath>> sourceTargetPathPairList;
// targetDevice -> isAlignedDevice
private final Map<String, Boolean> targetDeviceToAlignedMap;
+ // sourceColumn -> dataType (not serialize & deserialize)
+ private Map<String, TSDataType> sourceToDataTypeMap;
+
public IntoPathDescriptor() {
this.sourceTargetPathPairList = new ArrayList<>();
this.targetDeviceToAlignedMap = new HashMap<>();
+ this.sourceToDataTypeMap = new HashMap<>();
}
public IntoPathDescriptor(
@@ -71,6 +78,10 @@ public class IntoPathDescriptor {
targetDeviceToAlignedMap.put(targetDevice, isAligned);
}
+ public void recordSourceColumnDataType(String sourceColumn, TSDataType dataType) {
+ sourceToDataTypeMap.put(sourceColumn, dataType);
+ }
+
public void validate() {
List<PartialPath> targetPaths =
sourceTargetPathPairList.stream().map(Pair::getRight).collect(Collectors.toList());
@@ -79,6 +90,15 @@ public class IntoPathDescriptor {
}
}
+ public void bindType(ISchemaTree targetSchemaTree) {
+ this.sourceTargetPathPairList =
+ SelectIntoUtils.bindTypeForSourceTargetPathPairList(
+ sourceTargetPathPairList,
+ sourceToDataTypeMap,
+ targetSchemaTree,
+ targetDeviceToAlignedMap);
+ }
+
public List<Pair<String, PartialPath>> getSourceTargetPathPairList() {
return sourceTargetPathPairList;
}
@@ -103,6 +123,11 @@ public class IntoPathDescriptor {
return targetPathToSourceMap;
}
+ public Map<PartialPath, Map<String, TSDataType>> getTargetPathToDataTypeMap() {
+ return SelectIntoUtils.convertSourceTargetPathPairListToTargetPathDataTypeMap(
+ sourceTargetPathPairList);
+ }
+
public void serialize(ByteBuffer byteBuffer) {
ReadWriteIOUtils.write(sourceTargetPathPairList.size(), byteBuffer);
for (Pair<String, PartialPath> sourceTargetPathPair : sourceTargetPathPairList) {
diff --git a/server/src/main/java/org/apache/iotdb/db/utils/TypeInferenceUtils.java b/server/src/main/java/org/apache/iotdb/db/utils/TypeInferenceUtils.java
index 96c2b8292b..e4690d1260 100644
--- a/server/src/main/java/org/apache/iotdb/db/utils/TypeInferenceUtils.java
+++ b/server/src/main/java/org/apache/iotdb/db/utils/TypeInferenceUtils.java
@@ -29,19 +29,19 @@ import org.apache.commons.lang3.StringUtils;
public class TypeInferenceUtils {
- private static TSDataType booleanStringInferType =
+ private static final TSDataType booleanStringInferType =
IoTDBDescriptor.getInstance().getConfig().getBooleanStringInferType();
- private static TSDataType integerStringInferType =
+ private static final TSDataType integerStringInferType =
IoTDBDescriptor.getInstance().getConfig().getIntegerStringInferType();
- private static TSDataType longStringInferType =
+ private static final TSDataType longStringInferType =
IoTDBDescriptor.getInstance().getConfig().getLongStringInferType();
- private static TSDataType floatingStringInferType =
+ private static final TSDataType floatingStringInferType =
IoTDBDescriptor.getInstance().getConfig().getFloatingStringInferType();
- private static TSDataType nanStringInferType =
+ private static final TSDataType nanStringInferType =
IoTDBDescriptor.getInstance().getConfig().getNanStringInferType();
private TypeInferenceUtils() {}
@@ -157,4 +157,31 @@ public class TypeInferenceUtils {
throw new IllegalArgumentException("Invalid Aggregation function: " + aggrFuncName);
}
}
+
+ public static boolean canAutoCast(TSDataType fromType, TSDataType toType) {
+ if (fromType.equals(toType)) {
+ return true;
+ }
+
+ switch (fromType) {
+ case INT32:
+ switch (toType) {
+ case INT64:
+ case FLOAT:
+ case DOUBLE:
+ return true;
+ default:
+ return false;
+ }
+ case INT64:
+ case FLOAT:
+ return toType.equals(TSDataType.DOUBLE);
+ case DOUBLE:
+ case BOOLEAN:
+ case TEXT:
+ return false;
+ default:
+ throw new IllegalArgumentException("Unknown data type: " + fromType);
+ }
+ }
}