You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by GitBox <gi...@apache.org> on 2021/08/24 07:37:55 UTC
[GitHub] [iotdb] xuanronaldo opened a new pull request #3820: [IOTDB-xxx] Better Export/Import-CSV Tool
xuanronaldo opened a new pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820
## Description
Update the Export/Import-CSV tool.
### Import `common-csv`
Using `common-csv` to do CSV file reading and writing.
### Using `Java stream`
Using `Java stream` to simplify code.
### The New Import Tool
The new import tool support 4 data formats. Including the data aligned by time or device, with or without data type of headers. Besides, you can specify a directory to save file which save failed lines.
### The New Export Tool
The new export tool can print data type of timeseries in the head line of CSV file. Besides, you can directly specify a query comman that you want to execute by setting parameter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xuanronaldo commented on a change in pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
xuanronaldo commented on a change in pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#discussion_r696228026
##########
File path: cli/src/test/java/org/apache/iotdb/tool/integration/ExportCsvTestIT.java
##########
@@ -63,25 +68,20 @@ protected void testOnWindows() throws IOException {
"-h",
"127.0.0.1",
"-p",
- "6668",
+ "6667",
"-u",
"root",
"-pw",
"root",
"-td",
- "./");
- testOutput(builder, output);
+ ".\\",
+ "-q",
+ queryCommand);
+ testOutput(builder, null);
Review comment:
Ok, I got this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] HTHou commented on a change in pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
HTHou commented on a change in pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#discussion_r699778328
##########
File path: checkstyle.xml
##########
@@ -1,4 +1,5 @@
-<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE module PUBLIC "-//Checkstyle//DTD Checkstyle Configuration 1.3//EN" "https://checkstyle.org/dtds/configuration_1_3.dtd">
+<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE module PUBLIC "-//Checkstyle//DTD Checkstyle Configuration 1.3//EN"
Review comment:
Recover this file?
##########
File path: cli/src/main/java/org/apache/iotdb/tool/ExportCsv.java
##########
@@ -65,13 +63,23 @@
private static final String SQL_FILE_ARGS = "s";
private static final String SQL_FILE_NAME = "sqlfile";
+ private static final String DATA_TYPE_ARGS = "datatype";
+ private static final String DATA_TYPE_NAME = "datatype";
+
+ private static final String QUERY_COMMAND_ARGS = "q";
+ private static final String QUERY_COMMAND_NAME = "queryCommand";
+
private static final String TSFILEDB_CLI_PREFIX = "ExportCsv";
private static final String DUMP_FILE_NAME_DEFAULT = "dump";
private static String targetFile = DUMP_FILE_NAME_DEFAULT;
private static String targetDirectory;
+ private static Boolean dataType;
Review comment:
What is the meaning of dataType? why is a boolean?
##########
File path: pom.xml
##########
@@ -1434,4 +1434,4 @@
</properties>
</profile>
</profiles>
-</project>
+</project>
Review comment:
Don't change this.
##########
File path: cli/src/main/java/org/apache/iotdb/tool/ExportCsv.java
##########
@@ -65,13 +63,23 @@
private static final String SQL_FILE_ARGS = "s";
private static final String SQL_FILE_NAME = "sqlfile";
+ private static final String DATA_TYPE_ARGS = "datatype";
+ private static final String DATA_TYPE_NAME = "datatype";
+
+ private static final String QUERY_COMMAND_ARGS = "q";
+ private static final String QUERY_COMMAND_NAME = "queryCommand";
+
private static final String TSFILEDB_CLI_PREFIX = "ExportCsv";
private static final String DUMP_FILE_NAME_DEFAULT = "dump";
private static String targetFile = DUMP_FILE_NAME_DEFAULT;
private static String targetDirectory;
+ private static Boolean dataType;
Review comment:
How about choice another name? The current one is hard to understand...
##########
File path: cli/src/main/java/org/apache/iotdb/tool/ExportCsv.java
##########
@@ -65,13 +63,23 @@
private static final String SQL_FILE_ARGS = "s";
private static final String SQL_FILE_NAME = "sqlfile";
+ private static final String DATA_TYPE_ARGS = "datatype";
+ private static final String DATA_TYPE_NAME = "datatype";
+
+ private static final String QUERY_COMMAND_ARGS = "q";
+ private static final String QUERY_COMMAND_NAME = "queryCommand";
+
private static final String TSFILEDB_CLI_PREFIX = "ExportCsv";
private static final String DUMP_FILE_NAME_DEFAULT = "dump";
private static String targetFile = DUMP_FILE_NAME_DEFAULT;
private static String targetDirectory;
+ private static Boolean dataType;
Review comment:
Use needDataTypePrinted?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] HTHou merged pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
HTHou merged pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xuanronaldo commented on a change in pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
xuanronaldo commented on a change in pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#discussion_r699780837
##########
File path: cli/src/main/java/org/apache/iotdb/tool/ExportCsv.java
##########
@@ -65,13 +63,23 @@
private static final String SQL_FILE_ARGS = "s";
private static final String SQL_FILE_NAME = "sqlfile";
+ private static final String DATA_TYPE_ARGS = "datatype";
+ private static final String DATA_TYPE_NAME = "datatype";
+
+ private static final String QUERY_COMMAND_ARGS = "q";
+ private static final String QUERY_COMMAND_NAME = "queryCommand";
+
private static final String TSFILEDB_CLI_PREFIX = "ExportCsv";
private static final String DUMP_FILE_NAME_DEFAULT = "dump";
private static String targetFile = DUMP_FILE_NAME_DEFAULT;
private static String targetDirectory;
+ private static Boolean dataType;
Review comment:
It' a parameter to specify wheter print the data type if timeseries in the head line of CSV file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] HTHou commented on a change in pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
HTHou commented on a change in pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#discussion_r699779445
##########
File path: cli/src/main/java/org/apache/iotdb/tool/ExportCsv.java
##########
@@ -65,13 +63,23 @@
private static final String SQL_FILE_ARGS = "s";
private static final String SQL_FILE_NAME = "sqlfile";
+ private static final String DATA_TYPE_ARGS = "datatype";
+ private static final String DATA_TYPE_NAME = "datatype";
+
+ private static final String QUERY_COMMAND_ARGS = "q";
+ private static final String QUERY_COMMAND_NAME = "queryCommand";
+
private static final String TSFILEDB_CLI_PREFIX = "ExportCsv";
private static final String DUMP_FILE_NAME_DEFAULT = "dump";
private static String targetFile = DUMP_FILE_NAME_DEFAULT;
private static String targetDirectory;
+ private static Boolean dataType;
Review comment:
What is the meaning of dataType? why is a boolean?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#issuecomment-904432612
[](https://coveralls.io/builds/42395938)
Coverage decreased (-0.02%) to 67.264% when pulling **e23d17920371855ca5f632d3ec680e5b041396ef on xuanronaldo:xuanronaldo** into **8ec3e12865699064958e36a37a50083d85e11f9c on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] HTHou commented on a change in pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
HTHou commented on a change in pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#discussion_r708046231
##########
File path: docs/zh/UserGuide/System-Tools/CSV-Tool.md
##########
@@ -37,92 +129,81 @@ CREATE TIMESERIES root.fit.d2.s1 WITH DATATYPE=INT32,ENCODING=RLE;
CREATE TIMESERIES root.fit.d2.s3 WITH DATATYPE=INT32,ENCODING=RLE;
CREATE TIMESERIES root.fit.p.s1 WITH DATATYPE=INT32,ENCODING=RLE;
```
+
IoTDB 具有类型推断的能力,因此在数据导入前创建元数据不是必须的。但我们仍然推荐在使用 CSV 导入工具导入数据前创建元数据,因为这可以避免不必要的类型转换错误。
#### 待导入 CSV 文件示例
+通过时间对齐,并且header中不包含数据类型的数据。
+
```sql
-Time,root.fit.d1.s1,root.fit.d1.s2,root.fit.d2.s1,root.fit.d2.s3,root.fit.p.s1
-1,100,hello,200,300,400
-2,500,world,600,700,800
-3,900,"hello, \"world\"",1000,1100,1200
+Time,root.test.t1.str,root.test.t2.str,root.test.t2.int
+1970-01-01T08:00:00.001+08:00,"123hello world","123\,abc",100
+1970-01-01T08:00:00.002+08:00,"123",,
```
-#### 运行方法
-
-```shell
-# Unix/OS X
-> tools/import-csv.sh -h <ip> -p <port> -u <username> -pw <password> -f <xxx.csv>
+通过时间对齐,并且header中包含数据类型的数据。
-# Windows
-> tools\import-csv.bat -h <ip> -p <port> -u <username> -pw <password> -f <xxx.csv>
+```sql
+Time,root.test.t1.str(TEXT),root.test.t2.str(TEXT),root.test.t2.int(INT32)
+1970-01-01T08:00:00.001+08:00,"123hello world","123\,abc",100
+1970-01-01T08:00:00.002+08:00,"123",,
```
-#### 运行示例
-
-```shell
-# Unix/OS X
-> tools/import-csv.sh -h 127.0.0.1 -p 6667 -u root -pw root -f example-filename.csv
+通过设备对齐,并且header中不包含数据类型的数据。
-# Windows
-> tools\import-csv.bat -h 127.0.0.1 -p 6667 -u root -pw root -f example-filename.csv
+```sql
+Time,Device,str,int
+1970-01-01T08:00:00.001+08:00,root.test.t1,"123hello world",
+1970-01-01T08:00:00.002+08:00,root.test.t1,"123",
+1970-01-01T08:00:00.001+08:00,root.test.t2,"123\,abc",100
```
-#### 注意事项
-
-注意,在导入数据前,需要特殊处理下列的字符:
-
-1. `,` : 包含`,`的字段需要使用单引号或者双引号括起来
-2. `"` : "字段中的`"`需要被替换成转义字符`\"`或者用`\'`将字段括起来。
-3. `'` : "字段中的`'`需要被替换成转义字符`\'`或者用`\"`将字段括起来。
-4. 你可以输入时间格式像`yyyy-MM-dd'T'HH:mm:ss`, `yyy-MM-dd HH:mm:ss`, 或者`yyyy-MM-dd'T'HH:mm:ss.SSSZ`.
+通过设备对齐,并且header中包含数据类型的数据。
-### 使用 export-csv.sh
+```sql
+Time,Device,str(TEXT),int(INT32)
+1970-01-01T08:00:00.001+08:00,root.test.t1,"123hello world",
+1970-01-01T08:00:00.002+08:00,root.test.t1,"123",
+1970-01-01T08:00:00.001+08:00,root.test.t2,"123\,abc",100
+```
-#### 运行方法
+### 运行方法
```shell
-# Unix/OS X
-> tools/export-csv.sh -h <ip> -p <port> -u <username> -pw <password> -td <directory> [-tf <time-format> -s <sqlfile>]
+# Unix/OS X>
+tools/import-csv.sh -h <ip> -p <port> -u <username> -pw <password> -f <xxx.csv> [-fd <./failedDirectory>]
-# Windows
-> tools\export-csv.bat -h <ip> -p <port> -u <username> -pw <password> -td <directory> [-tf <time-format> -s <sqlfile>]
+# Windows>
+tools\import-csv.bat -h <ip> -p <port> -u <username> -pw <password> -f <xxx.csv> [-fd <./failedDirectory>]
```
-在运行导出脚本之后,您需要输入一些查询或指定一些 SQL 文件。如果在一个 SQL 文件中有多个 SQL,SQL 应该被换行符分割。
+参数:
-#### SQL 文件示例
+* `-f`:
+ - 指定你想要导入的数据。
+ - 例如: `-f filename.csv`
-```sql
-select * from root.fit.d1
-select * from root.sg1.d1
-```
+* `-fd`:
+ - 指定一个目录来存放保存失败的行的文件,如果你没有指定这个参数,失败的文件将会被保存到源数据的目录中,然后文件名是源文件名加上`.failed`的后缀。
+ - example: `-fd ./failed/`
-#### 运行示例
Review comment:
运行示例
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] HTHou commented on a change in pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
HTHou commented on a change in pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#discussion_r708779132
##########
File path: checkstyle.xml
##########
@@ -1,24 +1,20 @@
<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE module PUBLIC "-//Checkstyle//DTD Checkstyle Configuration 1.3//EN" "https://checkstyle.org/dtds/configuration_1_3.dtd">
<!--
-
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
Review comment:
```suggestion
with the License. You may obtain a copy of the License at
```
##########
File path: checkstyle.xml
##########
@@ -1,24 +1,20 @@
<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE module PUBLIC "-//Checkstyle//DTD Checkstyle Configuration 1.3//EN" "https://checkstyle.org/dtds/configuration_1_3.dtd">
<!--
Review comment:
```suggestion
<!--
```
##########
File path: cli/pom.xml
##########
@@ -75,6 +75,16 @@
<version>${project.version}</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-csv</artifactId>
+ <version>1.9.0</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-lang3</artifactId>
+ <version>3.0</version>
+ </dependency>
Review comment:
I see there is a commons-lang3 at Line 61. Is it duplicated?
##########
File path: checkstyle.xml
##########
@@ -1,24 +1,20 @@
<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE module PUBLIC "-//Checkstyle//DTD Checkstyle Configuration 1.3//EN" "https://checkstyle.org/dtds/configuration_1_3.dtd">
<!--
-
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
-
http://www.apache.org/licenses/LICENSE-2.0
Review comment:
```suggestion
http://www.apache.org/licenses/LICENSE-2.0
```
##########
File path: checkstyle.xml
##########
@@ -1,24 +1,20 @@
<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE module PUBLIC "-//Checkstyle//DTD Checkstyle Configuration 1.3//EN" "https://checkstyle.org/dtds/configuration_1_3.dtd">
<!--
-
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
-
http://www.apache.org/licenses/LICENSE-2.0
-
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
Review comment:
```suggestion
under the License.
```
##########
File path: cli/src/main/java/org/apache/iotdb/tool/ImportCsv.java
##########
@@ -16,53 +16,49 @@
* specific language governing permissions and limitations
* under the License.
*/
+
package org.apache.iotdb.tool;
import org.apache.iotdb.exception.ArgsErrorException;
import org.apache.iotdb.rpc.IoTDBConnectionException;
import org.apache.iotdb.rpc.StatementExecutionException;
import org.apache.iotdb.session.Session;
-import org.apache.iotdb.tsfile.common.constant.TsFileConstant;
+import org.apache.iotdb.session.SessionDataSet;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
import jline.console.ConsoleReader;
-import me.tongfei.progressbar.ProgressBar;
-import org.apache.commons.cli.CommandLine;
-import org.apache.commons.cli.CommandLineParser;
-import org.apache.commons.cli.DefaultParser;
-import org.apache.commons.cli.HelpFormatter;
-import org.apache.commons.cli.Option;
-import org.apache.commons.cli.Options;
-import org.apache.commons.cli.ParseException;
-
-import java.io.BufferedReader;
-import java.io.File;
-import java.io.FileInputStream;
-import java.io.FileNotFoundException;
-import java.io.IOException;
-import java.io.InputStreamReader;
-import java.io.LineNumberReader;
-import java.nio.charset.StandardCharsets;
+import org.apache.commons.cli.*;
+import org.apache.commons.csv.*;
Review comment:
We don't suggest to use the wildcard * here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] HTHou commented on a change in pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
HTHou commented on a change in pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#discussion_r699781229
##########
File path: pom.xml
##########
@@ -1434,4 +1434,4 @@
</properties>
</profile>
</profiles>
-</project>
+</project>
Review comment:
Don't change this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xuanronaldo commented on a change in pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
xuanronaldo commented on a change in pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#discussion_r696227587
##########
File path: cli/pom.xml
##########
@@ -75,6 +75,16 @@
<version>${project.version}</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-csv</artifactId>
+ <version>1.9.0</version>
+ </dependency>
+ <dependency>
+ <groupId>commons-lang</groupId>
Review comment:
Ok, I got it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#issuecomment-904432612
[](https://coveralls.io/builds/42370834)
Coverage decreased (-0.0005%) to 67.283% when pulling **c71619687609b584758e2a99a9eb2cafe6756acf on xuanronaldo:xuanronaldo** into **8ec3e12865699064958e36a37a50083d85e11f9c on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xuanronaldo commented on pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
xuanronaldo commented on pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#issuecomment-905684455
Ok, I got it.
发自我的iPhone
------------------ Original ------------------
From: Xiangdong Huang ***@***.***>
Date: Wed,Aug 25,2021 11:16 PM
To: apache/iotdb ***@***.***>
Cc: Xuan Ronaldo ***@***.***>, Author ***@***.***>
Subject: Re: [apache/iotdb] [IOTDB-842] Better Export/Import-CSV Tool (#3820)
@jixuan1989 commented on this pull request.
In cli/src/test/java/org/apache/iotdb/tool/integration/ImportCsvTestIT.java:
> @@ -16,9 +16,10 @@ * specific language governing permissions and limitations * under the License. */ -package org.apache.iotdb.tool; +package org.apache.iotdb.tool.integration;
this IT should be moved to cross-test module.
It is because cli module has no dependency of iotdb-server (in my memory).
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] jixuan1989 commented on a change in pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
jixuan1989 commented on a change in pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#discussion_r695854581
##########
File path: cli/src/test/java/org/apache/iotdb/tool/unit/WriteCsvFileTestUT.java
##########
@@ -16,18 +16,29 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.iotdb.tool;
-import org.junit.Assert;
+package org.apache.iotdb.tool.unit;
+
+import org.apache.iotdb.tool.AbstractCsvTool;
+
import org.junit.Test;
-public class CsvLineSplitTest {
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+public class WriteCsvFileTestUT {
@Test
- public void testSplit() {
- Assert.assertArrayEquals(
- new String[] {"", "a", "b", "c", "\\\""}, ImportCsv.splitCsvLine(",a,b,c,\"\\\"\""));
- Assert.assertArrayEquals(
- new String[] {"", "a", "b", "\\'"}, ImportCsv.splitCsvLine(",a,b,\"\\'\""));
+ public void writeCsvFileTest() {
+ List<String> headerNames =
+ new ArrayList<>(Arrays.asList("Time", "column1", "column2", "column3"));
+
+ List<Object> row1 = new ArrayList<>(Arrays.asList(1, null, "hello,world", true));
+ List<Object> row2 = new ArrayList<>(Arrays.asList(2, "", "hello,world", false));
+ List<Object> row3 = new ArrayList<>(Arrays.asList(3, "100", "hello world!!!", false));
+ ArrayList<List<Object>> records = new ArrayList<>(Arrays.asList(row1, row2, row3));
+
+ AbstractCsvTool.writeCsvFile(headerNames, records, "./test0.csv");
+ AbstractCsvTool.writeCsvFile(null, records, "./test1.csv");
Review comment:
no assert.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xuanronaldo commented on a change in pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
xuanronaldo commented on a change in pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#discussion_r709019955
##########
File path: cli/src/main/java/org/apache/iotdb/tool/ImportCsv.java
##########
@@ -16,53 +16,49 @@
* specific language governing permissions and limitations
* under the License.
*/
+
package org.apache.iotdb.tool;
import org.apache.iotdb.exception.ArgsErrorException;
import org.apache.iotdb.rpc.IoTDBConnectionException;
import org.apache.iotdb.rpc.StatementExecutionException;
import org.apache.iotdb.session.Session;
-import org.apache.iotdb.tsfile.common.constant.TsFileConstant;
+import org.apache.iotdb.session.SessionDataSet;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
import jline.console.ConsoleReader;
-import me.tongfei.progressbar.ProgressBar;
-import org.apache.commons.cli.CommandLine;
-import org.apache.commons.cli.CommandLineParser;
-import org.apache.commons.cli.DefaultParser;
-import org.apache.commons.cli.HelpFormatter;
-import org.apache.commons.cli.Option;
-import org.apache.commons.cli.Options;
-import org.apache.commons.cli.ParseException;
-
-import java.io.BufferedReader;
-import java.io.File;
-import java.io.FileInputStream;
-import java.io.FileNotFoundException;
-import java.io.IOException;
-import java.io.InputStreamReader;
-import java.io.LineNumberReader;
-import java.nio.charset.StandardCharsets;
+import org.apache.commons.cli.*;
+import org.apache.commons.csv.*;
Review comment:
Yeah, you are right. I will delete it suddenly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] jixuan1989 commented on a change in pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
jixuan1989 commented on a change in pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#discussion_r695856226
##########
File path: cli/src/test/java/org/apache/iotdb/tool/integration/ImportCsvTestIT.java
##########
@@ -16,9 +16,10 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.iotdb.tool;
+package org.apache.iotdb.tool.integration;
Review comment:
this IT should be moved to cross-test module.
It is because cli module has no dependency of iotdb-server (in my memory).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xuanronaldo commented on a change in pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
xuanronaldo commented on a change in pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#discussion_r699782435
##########
File path: cli/src/main/java/org/apache/iotdb/tool/ExportCsv.java
##########
@@ -65,13 +63,23 @@
private static final String SQL_FILE_ARGS = "s";
private static final String SQL_FILE_NAME = "sqlfile";
+ private static final String DATA_TYPE_ARGS = "datatype";
+ private static final String DATA_TYPE_NAME = "datatype";
+
+ private static final String QUERY_COMMAND_ARGS = "q";
+ private static final String QUERY_COMMAND_NAME = "queryCommand";
+
private static final String TSFILEDB_CLI_PREFIX = "ExportCsv";
private static final String DUMP_FILE_NAME_DEFAULT = "dump";
private static String targetFile = DUMP_FILE_NAME_DEFAULT;
private static String targetDirectory;
+ private static Boolean dataType;
Review comment:
How about "isDataTypePrinted"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] jixuan1989 commented on a change in pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
jixuan1989 commented on a change in pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#discussion_r695848358
##########
File path: cli/pom.xml
##########
@@ -75,6 +75,16 @@
<version>${project.version}</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-csv</artifactId>
+ <version>1.9.0</version>
+ </dependency>
+ <dependency>
+ <groupId>commons-lang</groupId>
Review comment:
it is better to use commons-lang3, as we have used it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#issuecomment-904432612
[](https://coveralls.io/builds/42365069)
Coverage decreased (-0.03%) to 67.258% when pulling **0b61f38d61262a1862fb5bf611cee066f01342ec on xuanronaldo:xuanronaldo** into **8ec3e12865699064958e36a37a50083d85e11f9c on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls commented on pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
coveralls commented on pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#issuecomment-904432612
[](https://coveralls.io/builds/42362821)
Coverage decreased (-0.04%) to 67.245% when pulling **02729fbc4adf2d7fd1383935d4ed903b5289e067 on xuanronaldo:xuanronaldo** into **8ec3e12865699064958e36a37a50083d85e11f9c on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#issuecomment-904432612
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xuanronaldo commented on a change in pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
xuanronaldo commented on a change in pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#discussion_r708061989
##########
File path: docs/zh/UserGuide/System-Tools/CSV-Tool.md
##########
@@ -37,92 +129,81 @@ CREATE TIMESERIES root.fit.d2.s1 WITH DATATYPE=INT32,ENCODING=RLE;
CREATE TIMESERIES root.fit.d2.s3 WITH DATATYPE=INT32,ENCODING=RLE;
CREATE TIMESERIES root.fit.p.s1 WITH DATATYPE=INT32,ENCODING=RLE;
```
+
IoTDB 具有类型推断的能力,因此在数据导入前创建元数据不是必须的。但我们仍然推荐在使用 CSV 导入工具导入数据前创建元数据,因为这可以避免不必要的类型转换错误。
#### 待导入 CSV 文件示例
+通过时间对齐,并且header中不包含数据类型的数据。
+
```sql
-Time,root.fit.d1.s1,root.fit.d1.s2,root.fit.d2.s1,root.fit.d2.s3,root.fit.p.s1
-1,100,hello,200,300,400
-2,500,world,600,700,800
-3,900,"hello, \"world\"",1000,1100,1200
+Time,root.test.t1.str,root.test.t2.str,root.test.t2.int
+1970-01-01T08:00:00.001+08:00,"123hello world","123\,abc",100
+1970-01-01T08:00:00.002+08:00,"123",,
```
-#### 运行方法
-
-```shell
-# Unix/OS X
-> tools/import-csv.sh -h <ip> -p <port> -u <username> -pw <password> -f <xxx.csv>
+通过时间对齐,并且header中包含数据类型的数据。
-# Windows
-> tools\import-csv.bat -h <ip> -p <port> -u <username> -pw <password> -f <xxx.csv>
+```sql
+Time,root.test.t1.str(TEXT),root.test.t2.str(TEXT),root.test.t2.int(INT32)
+1970-01-01T08:00:00.001+08:00,"123hello world","123\,abc",100
+1970-01-01T08:00:00.002+08:00,"123",,
```
-#### 运行示例
-
-```shell
-# Unix/OS X
-> tools/import-csv.sh -h 127.0.0.1 -p 6667 -u root -pw root -f example-filename.csv
+通过设备对齐,并且header中不包含数据类型的数据。
-# Windows
-> tools\import-csv.bat -h 127.0.0.1 -p 6667 -u root -pw root -f example-filename.csv
+```sql
+Time,Device,str,int
+1970-01-01T08:00:00.001+08:00,root.test.t1,"123hello world",
+1970-01-01T08:00:00.002+08:00,root.test.t1,"123",
+1970-01-01T08:00:00.001+08:00,root.test.t2,"123\,abc",100
```
-#### 注意事项
-
-注意,在导入数据前,需要特殊处理下列的字符:
-
-1. `,` : 包含`,`的字段需要使用单引号或者双引号括起来
-2. `"` : "字段中的`"`需要被替换成转义字符`\"`或者用`\'`将字段括起来。
-3. `'` : "字段中的`'`需要被替换成转义字符`\'`或者用`\"`将字段括起来。
-4. 你可以输入时间格式像`yyyy-MM-dd'T'HH:mm:ss`, `yyy-MM-dd HH:mm:ss`, 或者`yyyy-MM-dd'T'HH:mm:ss.SSSZ`.
+通过设备对齐,并且header中包含数据类型的数据。
-### 使用 export-csv.sh
+```sql
+Time,Device,str(TEXT),int(INT32)
+1970-01-01T08:00:00.001+08:00,root.test.t1,"123hello world",
+1970-01-01T08:00:00.002+08:00,root.test.t1,"123",
+1970-01-01T08:00:00.001+08:00,root.test.t2,"123\,abc",100
+```
-#### 运行方法
+### 运行方法
```shell
-# Unix/OS X
-> tools/export-csv.sh -h <ip> -p <port> -u <username> -pw <password> -td <directory> [-tf <time-format> -s <sqlfile>]
+# Unix/OS X>
+tools/import-csv.sh -h <ip> -p <port> -u <username> -pw <password> -f <xxx.csv> [-fd <./failedDirectory>]
-# Windows
-> tools\export-csv.bat -h <ip> -p <port> -u <username> -pw <password> -td <directory> [-tf <time-format> -s <sqlfile>]
+# Windows>
+tools\import-csv.bat -h <ip> -p <port> -u <username> -pw <password> -f <xxx.csv> [-fd <./failedDirectory>]
```
-在运行导出脚本之后,您需要输入一些查询或指定一些 SQL 文件。如果在一个 SQL 文件中有多个 SQL,SQL 应该被换行符分割。
+参数:
-#### SQL 文件示例
+* `-f`:
+ - 指定你想要导入的数据。
+ - 例如: `-f filename.csv`
-```sql
-select * from root.fit.d1
-select * from root.sg1.d1
-```
+* `-fd`:
+ - 指定一个目录来存放保存失败的行的文件,如果你没有指定这个参数,失败的文件将会被保存到源数据的目录中,然后文件名是源文件名加上`.failed`的后缀。
+ - example: `-fd ./failed/`
-#### 运行示例
+### Example
Review comment:
got it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] HTHou commented on a change in pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
HTHou commented on a change in pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#discussion_r699785738
##########
File path: cli/src/main/java/org/apache/iotdb/tool/ExportCsv.java
##########
@@ -65,13 +63,23 @@
private static final String SQL_FILE_ARGS = "s";
private static final String SQL_FILE_NAME = "sqlfile";
+ private static final String DATA_TYPE_ARGS = "datatype";
+ private static final String DATA_TYPE_NAME = "datatype";
+
+ private static final String QUERY_COMMAND_ARGS = "q";
+ private static final String QUERY_COMMAND_NAME = "queryCommand";
+
private static final String TSFILEDB_CLI_PREFIX = "ExportCsv";
private static final String DUMP_FILE_NAME_DEFAULT = "dump";
private static String targetFile = DUMP_FILE_NAME_DEFAULT;
private static String targetDirectory;
+ private static Boolean dataType;
Review comment:
Use needDataTypePrinted?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] HTHou commented on a change in pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
HTHou commented on a change in pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#discussion_r707921760
##########
File path: checkstyle.xml
##########
@@ -1,4 +1,5 @@
-<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE module PUBLIC "-//Checkstyle//DTD Checkstyle Configuration 1.3//EN" "https://checkstyle.org/dtds/configuration_1_3.dtd">
+<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE module PUBLIC "-//Checkstyle//DTD Checkstyle Configuration 1.3//EN"
+ "https://checkstyle.org/dtds/configuration_1_3.dtd">
Review comment:
Hi, can you please recover the changes?
##########
File path: docs/UserGuide/System-Tools/CSV-Tool.md
##########
@@ -19,126 +19,192 @@
-->
-## CSV Tool
+# CSV Tool
The CSV tool can help you import data in CSV format to IoTDB or export data from IoTDB to a CSV file.
+## Usage of export-csv.sh
-### Usage of import-csv.sh
+### Syntax
Review comment:
Update the Chinese doc?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] HTHou commented on a change in pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
HTHou commented on a change in pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#discussion_r699781613
##########
File path: cli/src/main/java/org/apache/iotdb/tool/ExportCsv.java
##########
@@ -65,13 +63,23 @@
private static final String SQL_FILE_ARGS = "s";
private static final String SQL_FILE_NAME = "sqlfile";
+ private static final String DATA_TYPE_ARGS = "datatype";
+ private static final String DATA_TYPE_NAME = "datatype";
+
+ private static final String QUERY_COMMAND_ARGS = "q";
+ private static final String QUERY_COMMAND_NAME = "queryCommand";
+
private static final String TSFILEDB_CLI_PREFIX = "ExportCsv";
private static final String DUMP_FILE_NAME_DEFAULT = "dump";
private static String targetFile = DUMP_FILE_NAME_DEFAULT;
private static String targetDirectory;
+ private static Boolean dataType;
Review comment:
How about choice another name? The current one is hard to understand...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#issuecomment-904432612
[](https://coveralls.io/builds/42397754)
Coverage decreased (-0.0008%) to 67.283% when pulling **a7017db864aec310ab3ef68af2866a70ad8080f8 on xuanronaldo:xuanronaldo** into **8ec3e12865699064958e36a37a50083d85e11f9c on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xuanronaldo commented on a change in pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
xuanronaldo commented on a change in pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#discussion_r699786012
##########
File path: cli/src/main/java/org/apache/iotdb/tool/ExportCsv.java
##########
@@ -65,13 +63,23 @@
private static final String SQL_FILE_ARGS = "s";
private static final String SQL_FILE_NAME = "sqlfile";
+ private static final String DATA_TYPE_ARGS = "datatype";
+ private static final String DATA_TYPE_NAME = "datatype";
+
+ private static final String QUERY_COMMAND_ARGS = "q";
+ private static final String QUERY_COMMAND_NAME = "queryCommand";
+
private static final String TSFILEDB_CLI_PREFIX = "ExportCsv";
private static final String DUMP_FILE_NAME_DEFAULT = "dump";
private static String targetFile = DUMP_FILE_NAME_DEFAULT;
private static String targetDirectory;
+ private static Boolean dataType;
Review comment:
Ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xuanronaldo commented on a change in pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
xuanronaldo commented on a change in pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#discussion_r709020215
##########
File path: cli/pom.xml
##########
@@ -75,6 +75,16 @@
<version>${project.version}</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-csv</artifactId>
+ <version>1.9.0</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-lang3</artifactId>
+ <version>3.0</version>
+ </dependency>
Review comment:
Yeah, you are right. I will delete it suddenly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] HTHou commented on a change in pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
HTHou commented on a change in pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#discussion_r699778328
##########
File path: checkstyle.xml
##########
@@ -1,4 +1,5 @@
-<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE module PUBLIC "-//Checkstyle//DTD Checkstyle Configuration 1.3//EN" "https://checkstyle.org/dtds/configuration_1_3.dtd">
+<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE module PUBLIC "-//Checkstyle//DTD Checkstyle Configuration 1.3//EN"
Review comment:
Recover this file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xuanronaldo commented on a change in pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
xuanronaldo commented on a change in pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#discussion_r699781671
##########
File path: pom.xml
##########
@@ -1434,4 +1434,4 @@
</properties>
</profile>
</profiles>
-</project>
+</project>
Review comment:
Sorry, it's my fault. I didn't realize that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#issuecomment-904432612
[](https://coveralls.io/builds/42367547)
Coverage increased (+0.002%) to 67.286% when pulling **c94565f63d2065de1010b9bcb45a3037b2ed6c94 on xuanronaldo:xuanronaldo** into **8ec3e12865699064958e36a37a50083d85e11f9c on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xuanronaldo commented on a change in pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
xuanronaldo commented on a change in pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#discussion_r707975989
##########
File path: docs/UserGuide/System-Tools/CSV-Tool.md
##########
@@ -19,126 +19,192 @@
-->
-## CSV Tool
+# CSV Tool
The CSV tool can help you import data in CSV format to IoTDB or export data from IoTDB to a CSV file.
+## Usage of export-csv.sh
-### Usage of import-csv.sh
+### Syntax
Review comment:
I just updated the Chinese doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xuanronaldo commented on a change in pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
xuanronaldo commented on a change in pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#discussion_r707975610
##########
File path: checkstyle.xml
##########
@@ -1,4 +1,5 @@
-<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE module PUBLIC "-//Checkstyle//DTD Checkstyle Configuration 1.3//EN" "https://checkstyle.org/dtds/configuration_1_3.dtd">
+<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE module PUBLIC "-//Checkstyle//DTD Checkstyle Configuration 1.3//EN"
+ "https://checkstyle.org/dtds/configuration_1_3.dtd">
Review comment:
Yeah, I just recovered the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xuanronaldo commented on a change in pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
xuanronaldo commented on a change in pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#discussion_r696229081
##########
File path: cli/src/test/java/org/apache/iotdb/tool/integration/ImportCsvTestIT.java
##########
@@ -16,9 +16,10 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.iotdb.tool;
+package org.apache.iotdb.tool.integration;
Review comment:
I just checked it out,there is the dependency of iotdb-server in cli module. But I will do it as you say, if it is necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#issuecomment-904432612
[](https://coveralls.io/builds/42391229)
Coverage decreased (-0.05%) to 67.239% when pulling **3b45e103bd481a723cc54a781415258a15549941 on xuanronaldo:xuanronaldo** into **8ec3e12865699064958e36a37a50083d85e11f9c on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] jixuan1989 commented on a change in pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
jixuan1989 commented on a change in pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#discussion_r695847049
##########
File path: checkstyle.xml
##########
@@ -1,4 +1,5 @@
-<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE module PUBLIC "-//Checkstyle//DTD Checkstyle Configuration 1.3//EN" "https://checkstyle.org/dtds/configuration_1_3.dtd">
+<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE module PUBLIC "-//Checkstyle//DTD Checkstyle Configuration 1.3//EN"
Review comment:
why you change checkstyle.xml??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] jixuan1989 commented on a change in pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
jixuan1989 commented on a change in pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#discussion_r695853336
##########
File path: cli/src/test/java/org/apache/iotdb/tool/integration/ExportCsvTestIT.java
##########
@@ -63,25 +68,20 @@ protected void testOnWindows() throws IOException {
"-h",
"127.0.0.1",
"-p",
- "6668",
+ "6667",
"-u",
"root",
"-pw",
"root",
"-td",
- "./");
- testOutput(builder, output);
+ ".\\",
+ "-q",
+ queryCommand);
+ testOutput(builder, null);
Review comment:
.....what is the meaning of this test???
You need:
1. write some data into iotdb in setup()
2. export and check whether the result is correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xuanronaldo commented on a change in pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
xuanronaldo commented on a change in pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#discussion_r699780837
##########
File path: cli/src/main/java/org/apache/iotdb/tool/ExportCsv.java
##########
@@ -65,13 +63,23 @@
private static final String SQL_FILE_ARGS = "s";
private static final String SQL_FILE_NAME = "sqlfile";
+ private static final String DATA_TYPE_ARGS = "datatype";
+ private static final String DATA_TYPE_NAME = "datatype";
+
+ private static final String QUERY_COMMAND_ARGS = "q";
+ private static final String QUERY_COMMAND_NAME = "queryCommand";
+
private static final String TSFILEDB_CLI_PREFIX = "ExportCsv";
private static final String DUMP_FILE_NAME_DEFAULT = "dump";
private static String targetFile = DUMP_FILE_NAME_DEFAULT;
private static String targetDirectory;
+ private static Boolean dataType;
Review comment:
It' a parameter to specify wheter print the data type if timeseries in the head line of CSV file.
##########
File path: pom.xml
##########
@@ -1434,4 +1434,4 @@
</properties>
</profile>
</profiles>
-</project>
+</project>
Review comment:
Sorry, it's my fault. I didn't realize that.
##########
File path: cli/src/main/java/org/apache/iotdb/tool/ExportCsv.java
##########
@@ -65,13 +63,23 @@
private static final String SQL_FILE_ARGS = "s";
private static final String SQL_FILE_NAME = "sqlfile";
+ private static final String DATA_TYPE_ARGS = "datatype";
+ private static final String DATA_TYPE_NAME = "datatype";
+
+ private static final String QUERY_COMMAND_ARGS = "q";
+ private static final String QUERY_COMMAND_NAME = "queryCommand";
+
private static final String TSFILEDB_CLI_PREFIX = "ExportCsv";
private static final String DUMP_FILE_NAME_DEFAULT = "dump";
private static String targetFile = DUMP_FILE_NAME_DEFAULT;
private static String targetDirectory;
+ private static Boolean dataType;
Review comment:
Ok, I got it.
##########
File path: cli/src/main/java/org/apache/iotdb/tool/ExportCsv.java
##########
@@ -65,13 +63,23 @@
private static final String SQL_FILE_ARGS = "s";
private static final String SQL_FILE_NAME = "sqlfile";
+ private static final String DATA_TYPE_ARGS = "datatype";
+ private static final String DATA_TYPE_NAME = "datatype";
+
+ private static final String QUERY_COMMAND_ARGS = "q";
+ private static final String QUERY_COMMAND_NAME = "queryCommand";
+
private static final String TSFILEDB_CLI_PREFIX = "ExportCsv";
private static final String DUMP_FILE_NAME_DEFAULT = "dump";
private static String targetFile = DUMP_FILE_NAME_DEFAULT;
private static String targetDirectory;
+ private static Boolean dataType;
Review comment:
How about "isDataTypePrinted"?
##########
File path: cli/src/main/java/org/apache/iotdb/tool/ExportCsv.java
##########
@@ -65,13 +63,23 @@
private static final String SQL_FILE_ARGS = "s";
private static final String SQL_FILE_NAME = "sqlfile";
+ private static final String DATA_TYPE_ARGS = "datatype";
+ private static final String DATA_TYPE_NAME = "datatype";
+
+ private static final String QUERY_COMMAND_ARGS = "q";
+ private static final String QUERY_COMMAND_NAME = "queryCommand";
+
private static final String TSFILEDB_CLI_PREFIX = "ExportCsv";
private static final String DUMP_FILE_NAME_DEFAULT = "dump";
private static String targetFile = DUMP_FILE_NAME_DEFAULT;
private static String targetDirectory;
+ private static Boolean dataType;
Review comment:
Ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] HTHou commented on a change in pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
HTHou commented on a change in pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#discussion_r708047714
##########
File path: docs/zh/UserGuide/System-Tools/CSV-Tool.md
##########
@@ -37,92 +129,81 @@ CREATE TIMESERIES root.fit.d2.s1 WITH DATATYPE=INT32,ENCODING=RLE;
CREATE TIMESERIES root.fit.d2.s3 WITH DATATYPE=INT32,ENCODING=RLE;
CREATE TIMESERIES root.fit.p.s1 WITH DATATYPE=INT32,ENCODING=RLE;
```
+
IoTDB 具有类型推断的能力,因此在数据导入前创建元数据不是必须的。但我们仍然推荐在使用 CSV 导入工具导入数据前创建元数据,因为这可以避免不必要的类型转换错误。
#### 待导入 CSV 文件示例
+通过时间对齐,并且header中不包含数据类型的数据。
+
```sql
-Time,root.fit.d1.s1,root.fit.d1.s2,root.fit.d2.s1,root.fit.d2.s3,root.fit.p.s1
-1,100,hello,200,300,400
-2,500,world,600,700,800
-3,900,"hello, \"world\"",1000,1100,1200
+Time,root.test.t1.str,root.test.t2.str,root.test.t2.int
+1970-01-01T08:00:00.001+08:00,"123hello world","123\,abc",100
+1970-01-01T08:00:00.002+08:00,"123",,
```
-#### 运行方法
-
-```shell
-# Unix/OS X
-> tools/import-csv.sh -h <ip> -p <port> -u <username> -pw <password> -f <xxx.csv>
+通过时间对齐,并且header中包含数据类型的数据。
-# Windows
-> tools\import-csv.bat -h <ip> -p <port> -u <username> -pw <password> -f <xxx.csv>
+```sql
+Time,root.test.t1.str(TEXT),root.test.t2.str(TEXT),root.test.t2.int(INT32)
+1970-01-01T08:00:00.001+08:00,"123hello world","123\,abc",100
+1970-01-01T08:00:00.002+08:00,"123",,
```
-#### 运行示例
-
-```shell
-# Unix/OS X
-> tools/import-csv.sh -h 127.0.0.1 -p 6667 -u root -pw root -f example-filename.csv
+通过设备对齐,并且header中不包含数据类型的数据。
-# Windows
-> tools\import-csv.bat -h 127.0.0.1 -p 6667 -u root -pw root -f example-filename.csv
+```sql
+Time,Device,str,int
+1970-01-01T08:00:00.001+08:00,root.test.t1,"123hello world",
+1970-01-01T08:00:00.002+08:00,root.test.t1,"123",
+1970-01-01T08:00:00.001+08:00,root.test.t2,"123\,abc",100
```
-#### 注意事项
-
-注意,在导入数据前,需要特殊处理下列的字符:
-
-1. `,` : 包含`,`的字段需要使用单引号或者双引号括起来
-2. `"` : "字段中的`"`需要被替换成转义字符`\"`或者用`\'`将字段括起来。
-3. `'` : "字段中的`'`需要被替换成转义字符`\'`或者用`\"`将字段括起来。
-4. 你可以输入时间格式像`yyyy-MM-dd'T'HH:mm:ss`, `yyy-MM-dd HH:mm:ss`, 或者`yyyy-MM-dd'T'HH:mm:ss.SSSZ`.
+通过设备对齐,并且header中包含数据类型的数据。
-### 使用 export-csv.sh
+```sql
+Time,Device,str(TEXT),int(INT32)
+1970-01-01T08:00:00.001+08:00,root.test.t1,"123hello world",
+1970-01-01T08:00:00.002+08:00,root.test.t1,"123",
+1970-01-01T08:00:00.001+08:00,root.test.t2,"123\,abc",100
+```
-#### 运行方法
+### 运行方法
```shell
-# Unix/OS X
-> tools/export-csv.sh -h <ip> -p <port> -u <username> -pw <password> -td <directory> [-tf <time-format> -s <sqlfile>]
+# Unix/OS X>
+tools/import-csv.sh -h <ip> -p <port> -u <username> -pw <password> -f <xxx.csv> [-fd <./failedDirectory>]
-# Windows
-> tools\export-csv.bat -h <ip> -p <port> -u <username> -pw <password> -td <directory> [-tf <time-format> -s <sqlfile>]
+# Windows>
+tools\import-csv.bat -h <ip> -p <port> -u <username> -pw <password> -f <xxx.csv> [-fd <./failedDirectory>]
```
-在运行导出脚本之后,您需要输入一些查询或指定一些 SQL 文件。如果在一个 SQL 文件中有多个 SQL,SQL 应该被换行符分割。
+参数:
-#### SQL 文件示例
+* `-f`:
+ - 指定你想要导入的数据。
+ - 例如: `-f filename.csv`
-```sql
-select * from root.fit.d1
-select * from root.sg1.d1
-```
+* `-fd`:
+ - 指定一个目录来存放保存失败的行的文件,如果你没有指定这个参数,失败的文件将会被保存到源数据的目录中,然后文件名是源文件名加上`.failed`的后缀。
+ - example: `-fd ./failed/`
-#### 运行示例
+### Example
Review comment:
运行示例
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xuanronaldo commented on a change in pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
xuanronaldo commented on a change in pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#discussion_r699782036
##########
File path: cli/src/main/java/org/apache/iotdb/tool/ExportCsv.java
##########
@@ -65,13 +63,23 @@
private static final String SQL_FILE_ARGS = "s";
private static final String SQL_FILE_NAME = "sqlfile";
+ private static final String DATA_TYPE_ARGS = "datatype";
+ private static final String DATA_TYPE_NAME = "datatype";
+
+ private static final String QUERY_COMMAND_ARGS = "q";
+ private static final String QUERY_COMMAND_NAME = "queryCommand";
+
private static final String TSFILEDB_CLI_PREFIX = "ExportCsv";
private static final String DUMP_FILE_NAME_DEFAULT = "dump";
private static String targetFile = DUMP_FILE_NAME_DEFAULT;
private static String targetDirectory;
+ private static Boolean dataType;
Review comment:
Ok, I got it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#issuecomment-904432612
[](https://coveralls.io/builds/42395941)
Coverage decreased (-0.02%) to 67.263% when pulling **e23d17920371855ca5f632d3ec680e5b041396ef on xuanronaldo:xuanronaldo** into **8ec3e12865699064958e36a37a50083d85e11f9c on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#issuecomment-904432612
[](https://coveralls.io/builds/42389651)
Coverage decreased (-0.01%) to 67.269% when pulling **ae5f27f656aea19405c8179893a70f5787c38f94 on xuanronaldo:xuanronaldo** into **8ec3e12865699064958e36a37a50083d85e11f9c on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xuanronaldo commented on a change in pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
xuanronaldo commented on a change in pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#discussion_r696227476
##########
File path: checkstyle.xml
##########
@@ -1,4 +1,5 @@
-<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE module PUBLIC "-//Checkstyle//DTD Checkstyle Configuration 1.3//EN" "https://checkstyle.org/dtds/configuration_1_3.dtd">
+<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE module PUBLIC "-//Checkstyle//DTD Checkstyle Configuration 1.3//EN"
Review comment:
Sorry, it's my fault. I didn't realize that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3820: [IOTDB-842] Better Export/Import-CSV Tool
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3820:
URL: https://github.com/apache/iotdb/pull/3820#issuecomment-904432612
[](https://coveralls.io/builds/42364100)
Coverage increased (+0.009%) to 67.293% when pulling **90a07e9fadd758825f9c2ff59be5a8c0384888a6 on xuanronaldo:xuanronaldo** into **8ec3e12865699064958e36a37a50083d85e11f9c on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org