You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by GitBox <gi...@apache.org> on 2021/07/15 07:49:45 UTC
[GitHub] [iotdb] yschengzi opened a new pull request #3576: [IOTDB-1504]Fix rename bug in loading tsfile
yschengzi opened a new pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576
Problem:
a new tsfile will not be loaded if existing a tsfile in data which has the same name with the new tsfile.
Steps:
step1: execute sql following
set storage group to root.group_1
create timeseries root.group_1.d_1.s_0 with datatype = boolean
create timeseries root.group_1.d_1.s_1 with datatype = float
create timeseries root.group_1.d_1.s_2 with datatype = float
insert into root.group_1.d_1(timestamp, s_1, s_0, s_2) values(2018-07-18T00:00:00.000+08:00, 18, false, 18.18)
flush
insert into root.group_1.d_1(timestamp, s_1, s_0, s_2) values(2018-07-19T00:00:00.000+08:00, 19, false, 19.18)
flush
step2: move a tsfile in data/sequence/root.group_1/../ to another folder, and change the name to be the same with the left one
step3: load the moving tsfile
and we can find the tsfile will be not in the database.
Solution:
Rewrite the code for renaming tsfile in sequence and unsequence folder.
When loading a tsfile, the version will be updated to the largest. The renaming rule for establish time in sequence tsfile not change, and the establish time of unsequence tsfile will be renamed to the nearest time stamp to the loading tsfile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] yschengzi commented on a change in pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
yschengzi commented on a change in pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#discussion_r677296500
##########
File path: docs/zh/UserGuide/System-Tools/Load-External-Tsfile.md
##########
@@ -30,31 +30,45 @@
用户通过Cli工具或JDBC向Apache IoTDB系统发送指定命令实现文件加载的功能。
#### 加载tsfile文件
-加载tsfile文件的指令为：`load "<path/dir>" [true/false] [storage group level]`
+加载tsfile文件的指令为：`load "<path/dir>" [autoregister=true/false][,sglevel=int][,verify=true/false]`
该指令有两种用法：
1. 通过指定文件路径(绝对路径)加载单tsfile文件。
-第二个参数表示待加载的tsfile文件的路径，其中文件名称需要符合tsfile的命名规范，即`{systemTime}-{versionNum}-{mergeNum}.tsfile`。第三、四个参数为可选项。当待加载的tsfile文件中时间序列对应的元数据不存在时，用户可以选择是否自动创建schema，参数为true表示自动创建schema，相反false表示不创建，缺省时默认创建schema。当tsfile对应的存储组不存在时，用户可以通过第四个参数来制定存储组的级别，默认为`iotdb-engine.properties`中设置的级别。若待加载的tsfile文件对应的`.resource`文件存在，会被一并加载至Apache IoTDB数据文件的目录和引擎中，否则将通过tsfile文件重新生成对应的`.resource`文件，即加载的tsfile文件所对应的`.resource`文件不是必要的。
+第二个参数表示待加载的tsfile文件的路径，其中文件名称需要符合tsfile的命名规范，即`{systemTime}-{versionNum}-{mergeNum}.tsfile`或者`{systemTime}-{versionNum}-{mergeNum}-{unsequenceMergeNum}.tsfile`。load命令有三个可选项，分别是schema，值域为true/false，level，值域为正整数，metadatat，值域为true/false。不同选项之间用逗号连接，选项之间无顺序要求。
Review comment:
already renaming
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#issuecomment-880523918
[](https://coveralls.io/builds/41561171)
Coverage increased (+0.01%) to 68.146% when pulling **2f7a84d085cd06d90bc7a90f18bdd883d0d9f26c on yschengzi:loadrename** into **6e9f3404442108c08493351d1447e6d2c1ec1347 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] wangchao316 commented on a change in pull request #3576: [IOTDB-1504]Fix rename bug in loading tsfile
Posted by GitBox <gi...@apache.org>.
wangchao316 commented on a change in pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#discussion_r670938112
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -90,20 +90,8 @@
import java.nio.ByteBuffer;
import java.nio.MappedByteBuffer;
import java.nio.file.Files;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collection;
-import java.util.Collections;
-import java.util.Date;
-import java.util.Deque;
-import java.util.HashMap;
-import java.util.Iterator;
-import java.util.LinkedList;
-import java.util.List;
-import java.util.Map;
+import java.util.*;
Review comment:
generate, we do not use import *.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#issuecomment-880523918
[](https://coveralls.io/builds/41541621)
Coverage increased (+0.02%) to 68.151% when pulling **f3721bdff767e59f49cb429f79b4cc3ce0143bea on yschengzi:loadrename** into **6e9f3404442108c08493351d1447e6d2c1ec1347 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] yschengzi commented on a change in pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
yschengzi commented on a change in pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#discussion_r677298086
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -2538,59 +2524,57 @@ private void removeFullyOverlapFile(
/**
* Get an appropriate filename to ensure the order between files. The tsfile is named after
- * ({systemTime}-{versionNum}-{mergeNum}.tsfile).
+ * ({systemTime}-{versionNum}-{mergeNum}-{unsequenceMergeNum}.tsfile).
Review comment:
already fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#issuecomment-880523918
[](https://coveralls.io/builds/41561993)
Coverage increased (+0.02%) to 68.153% when pulling **2f7a84d085cd06d90bc7a90f18bdd883d0d9f26c on yschengzi:loadrename** into **6e9f3404442108c08493351d1447e6d2c1ec1347 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] yschengzi commented on a change in pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
yschengzi commented on a change in pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#discussion_r677298340
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -2538,59 +2524,57 @@ private void removeFullyOverlapFile(
/**
* Get an appropriate filename to ensure the order between files. The tsfile is named after
- * ({systemTime}-{versionNum}-{mergeNum}.tsfile).
+ * ({systemTime}-{versionNum}-{mergeNum}-{unsequenceMergeNum}.tsfile).
*
* <p>The sorting rules for tsfile names @see {@link this#compareFileName}, we can restore the
* list based on the file name and ensure the correctness of the order, so there are three cases.
*
- * <p>1. The tsfile is to be inserted in the first place of the list. If the timestamp in the file
- * name is less than the timestamp in the file name of the first tsfile in the list, then the file
- * name is legal and the file name is returned directly. Otherwise, its timestamp can be set to
- * half of the timestamp value in the file name of the first tsfile in the list , and the version
- * number is the version number in the file name of the first tsfile in the list.
+ * <p>1. The tsfile is to be inserted in the first place of the list. Timestamp can be set to half
+ * of the timestamp value in the file name of the first tsfile in the list , and the version
+ * number will be updated to the largest number in this time partition.
*
- * <p>2. The tsfile is to be inserted in the last place of the list. If the timestamp in the file
- * name is lager than the timestamp in the file name of the last tsfile in the list, then the file
- * name is legal and the file name is returned directly. Otherwise, the file name is generated by
+ * <p>2. The tsfile is to be inserted in the last place of the list. The file name is generated by
* the system according to the naming rules and returned.
*
- * <p>3. This file is inserted between two files. If the timestamp in the name of the file
- * satisfies the timestamp between the timestamps in the name of the two files, then it is a legal
- * name and returns directly; otherwise, the time stamp is the mean of the timestamps of the two
- * files, the version number is the version number in the tsfile with a larger timestamp.
+ * <p>3. This file is inserted between two files. The time stamp is the mean of the timestamps of
+ * the two files, the version number will be updated to the largest number in this time partition.
*
- * @param tsfileName origin tsfile name
* @param insertIndex the new file will be inserted between the files [insertIndex, insertIndex +
* 1]
* @return appropriate filename
*/
- private String getFileNameForLoadingFile(
- String tsfileName, int insertIndex, long timePartitionId, List<TsFileResource> sequenceList) {
- long currentTsFileTime = Long.parseLong(tsfileName.split(FILE_NAME_SEPARATOR)[0]);
- long preTime;
+ private String getFileNameForSequenceLoadingFile(
+ int insertIndex, TsFileResource newTsFileResource, List<TsFileResource> sequenceList)
+ throws LoadFileException {
+ long timePartitionId = newTsFileResource.getTimePartition();
+ long preTime, subsequenceTime;
+
if (insertIndex == -1) {
preTime = 0L;
} else {
String preName = sequenceList.get(insertIndex).getTsFile().getName();
preTime = Long.parseLong(preName.split(FILE_NAME_SEPARATOR)[0]);
}
if (insertIndex == tsFileManagement.size(true) - 1) {
- if (preTime < currentTsFileTime) {
- return tsfileName;
- } else {
- return getNewTsFileName(timePartitionId);
- }
+ subsequenceTime = preTime + ((System.currentTimeMillis() - preTime) << 1);
+ } else {
+ String subsequenceName = sequenceList.get(insertIndex + 1).getTsFile().getName();
+ subsequenceTime = Long.parseLong(subsequenceName.split(FILE_NAME_SEPARATOR)[0]);
}
- String subsequenceName = sequenceList.get(insertIndex + 1).getTsFile().getName();
- long subsequenceTime = Long.parseLong(subsequenceName.split(FILE_NAME_SEPARATOR)[0]);
- long subsequenceVersion = Long.parseLong(subsequenceName.split(FILE_NAME_SEPARATOR)[1]);
- if (preTime < currentTsFileTime && currentTsFileTime < subsequenceTime) {
- return tsfileName;
+ long meanTime = preTime + ((subsequenceTime - preTime) >> 1);
+ if (insertIndex != tsFileManagement.size(true) - 1 && meanTime == subsequenceTime) {
+ throw new LoadFileException("can not load tsfile with the same establish time");
Review comment:
change info
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] qiaojialin commented on a change in pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
qiaojialin commented on a change in pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#discussion_r677142582
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -2293,42 +2293,35 @@ public void loadNewTsFile(TsFileResource newTsFileResource) throws LoadFileExcep
List<TsFileResource> sequenceList = tsFileManagement.getTsFileList(true);
int insertPos = findInsertionPosition(newTsFileResource, newFilePartitionId, sequenceList);
- if (insertPos == POS_ALREADY_EXIST) {
- return;
- }
+ String newFileName, renameInfo;
+ LoadTsFileType tsFileType;
// loading tsfile by type
if (insertPos == POS_OVERLAP) {
- loadTsFileByType(
- LoadTsFileType.LOAD_UNSEQUENCE,
- tsfileToBeInserted,
- newTsFileResource,
- newFilePartitionId);
+ newFileName =
+ getNewTsFileName(
+ System.currentTimeMillis(),
+ getAndSetNewVersion(newFilePartitionId, newTsFileResource),
+ 0);
+ renameInfo = "unsequence";
Review comment:
use a constant, do not hard code
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -2293,42 +2293,35 @@ public void loadNewTsFile(TsFileResource newTsFileResource) throws LoadFileExcep
List<TsFileResource> sequenceList = tsFileManagement.getTsFileList(true);
int insertPos = findInsertionPosition(newTsFileResource, newFilePartitionId, sequenceList);
- if (insertPos == POS_ALREADY_EXIST) {
- return;
- }
+ String newFileName, renameInfo;
+ LoadTsFileType tsFileType;
// loading tsfile by type
if (insertPos == POS_OVERLAP) {
- loadTsFileByType(
- LoadTsFileType.LOAD_UNSEQUENCE,
- tsfileToBeInserted,
- newTsFileResource,
- newFilePartitionId);
+ newFileName =
+ getNewTsFileName(
+ System.currentTimeMillis(),
+ getAndSetNewVersion(newFilePartitionId, newTsFileResource),
+ 0);
+ renameInfo = "unsequence";
+ tsFileType = LoadTsFileType.LOAD_UNSEQUENCE;
} else {
-
// check whether the file name needs to be renamed.
- if (!tsFileManagement.isEmpty(true)) {
- String newFileName =
- getFileNameForLoadingFile(
- tsfileToBeInserted.getName(),
- insertPos,
- newTsFileResource.getTimePartition(),
- sequenceList);
- if (!newFileName.equals(tsfileToBeInserted.getName())) {
- logger.info(
- "Tsfile {} must be renamed to {} for loading into the sequence list.",
- tsfileToBeInserted.getName(),
- newFileName);
- newTsFileResource.setFile(
- fsFactory.getFile(tsfileToBeInserted.getParentFile(), newFileName));
- }
- }
- loadTsFileByType(
- LoadTsFileType.LOAD_SEQUENCE,
- tsfileToBeInserted,
- newTsFileResource,
- newFilePartitionId);
+ newFileName = getFileNameForSequenceLoadingFile(insertPos, newTsFileResource, sequenceList);
+ renameInfo = "sequence";
Review comment:
change to a constant
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -2293,42 +2293,35 @@ public void loadNewTsFile(TsFileResource newTsFileResource) throws LoadFileExcep
List<TsFileResource> sequenceList = tsFileManagement.getTsFileList(true);
int insertPos = findInsertionPosition(newTsFileResource, newFilePartitionId, sequenceList);
- if (insertPos == POS_ALREADY_EXIST) {
- return;
- }
+ String newFileName, renameInfo;
+ LoadTsFileType tsFileType;
// loading tsfile by type
if (insertPos == POS_OVERLAP) {
- loadTsFileByType(
- LoadTsFileType.LOAD_UNSEQUENCE,
- tsfileToBeInserted,
- newTsFileResource,
- newFilePartitionId);
+ newFileName =
+ getNewTsFileName(
+ System.currentTimeMillis(),
+ getAndSetNewVersion(newFilePartitionId, newTsFileResource),
+ 0);
+ renameInfo = "unsequence";
+ tsFileType = LoadTsFileType.LOAD_UNSEQUENCE;
Review comment:
newTsFileResource.setSeq(false); ?
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/physical/sys/OperateFilePlan.java
##########
@@ -32,18 +32,33 @@
private File targetDir;
private boolean autoCreateSchema;
private int sgLevel;
+ private boolean metadataCheck;
Review comment:
```suggestion
private boolean verifyMetadata;
```
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -2538,59 +2524,57 @@ private void removeFullyOverlapFile(
/**
* Get an appropriate filename to ensure the order between files. The tsfile is named after
- * ({systemTime}-{versionNum}-{mergeNum}.tsfile).
+ * ({systemTime}-{versionNum}-{mergeNum}-{unsequenceMergeNum}.tsfile).
*
* <p>The sorting rules for tsfile names @see {@link this#compareFileName}, we can restore the
* list based on the file name and ensure the correctness of the order, so there are three cases.
*
- * <p>1. The tsfile is to be inserted in the first place of the list. If the timestamp in the file
- * name is less than the timestamp in the file name of the first tsfile in the list, then the file
- * name is legal and the file name is returned directly. Otherwise, its timestamp can be set to
- * half of the timestamp value in the file name of the first tsfile in the list , and the version
- * number is the version number in the file name of the first tsfile in the list.
+ * <p>1. The tsfile is to be inserted in the first place of the list. Timestamp can be set to half
+ * of the timestamp value in the file name of the first tsfile in the list , and the version
+ * number will be updated to the largest number in this time partition.
*
- * <p>2. The tsfile is to be inserted in the last place of the list. If the timestamp in the file
- * name is lager than the timestamp in the file name of the last tsfile in the list, then the file
- * name is legal and the file name is returned directly. Otherwise, the file name is generated by
+ * <p>2. The tsfile is to be inserted in the last place of the list. The file name is generated by
* the system according to the naming rules and returned.
*
- * <p>3. This file is inserted between two files. If the timestamp in the name of the file
- * satisfies the timestamp between the timestamps in the name of the two files, then it is a legal
- * name and returns directly; otherwise, the time stamp is the mean of the timestamps of the two
- * files, the version number is the version number in the tsfile with a larger timestamp.
+ * <p>3. This file is inserted between two files. The time stamp is the mean of the timestamps of
+ * the two files, the version number will be updated to the largest number in this time partition.
*
- * @param tsfileName origin tsfile name
* @param insertIndex the new file will be inserted between the files [insertIndex, insertIndex +
* 1]
* @return appropriate filename
*/
- private String getFileNameForLoadingFile(
- String tsfileName, int insertIndex, long timePartitionId, List<TsFileResource> sequenceList) {
- long currentTsFileTime = Long.parseLong(tsfileName.split(FILE_NAME_SEPARATOR)[0]);
- long preTime;
+ private String getFileNameForSequenceLoadingFile(
+ int insertIndex, TsFileResource newTsFileResource, List<TsFileResource> sequenceList)
+ throws LoadFileException {
+ long timePartitionId = newTsFileResource.getTimePartition();
+ long preTime, subsequenceTime;
+
if (insertIndex == -1) {
preTime = 0L;
} else {
String preName = sequenceList.get(insertIndex).getTsFile().getName();
preTime = Long.parseLong(preName.split(FILE_NAME_SEPARATOR)[0]);
}
if (insertIndex == tsFileManagement.size(true) - 1) {
- if (preTime < currentTsFileTime) {
- return tsfileName;
- } else {
- return getNewTsFileName(timePartitionId);
- }
+ subsequenceTime = preTime + ((System.currentTimeMillis() - preTime) << 1);
+ } else {
+ String subsequenceName = sequenceList.get(insertIndex + 1).getTsFile().getName();
+ subsequenceTime = Long.parseLong(subsequenceName.split(FILE_NAME_SEPARATOR)[0]);
}
- String subsequenceName = sequenceList.get(insertIndex + 1).getTsFile().getName();
- long subsequenceTime = Long.parseLong(subsequenceName.split(FILE_NAME_SEPARATOR)[0]);
- long subsequenceVersion = Long.parseLong(subsequenceName.split(FILE_NAME_SEPARATOR)[1]);
- if (preTime < currentTsFileTime && currentTsFileTime < subsequenceTime) {
- return tsfileName;
+ long meanTime = preTime + ((subsequenceTime - preTime) >> 1);
+ if (insertIndex != tsFileManagement.size(true) - 1 && meanTime == subsequenceTime) {
+ throw new LoadFileException("can not load tsfile with the same establish time");
Review comment:
can not load TsFile because of can not find suitable location
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/physical/sys/OperateFilePlan.java
##########
@@ -84,6 +103,8 @@ public String toString() {
+ autoCreateSchema
+ ", sgLevel="
+ sgLevel
+ + ", metadataCheck="
Review comment:
```suggestion
+ ", verify="
```
This should be consistent to user
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -2538,59 +2524,57 @@ private void removeFullyOverlapFile(
/**
* Get an appropriate filename to ensure the order between files. The tsfile is named after
- * ({systemTime}-{versionNum}-{mergeNum}.tsfile).
+ * ({systemTime}-{versionNum}-{mergeNum}-{unsequenceMergeNum}.tsfile).
Review comment:
remember to update
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/executor/PlanExecutor.java
##########
@@ -1097,6 +1098,13 @@ private void loadFile(File file, OperateFilePlan plan) throws QueryProcessExcept
List<ChunkGroupMetadata> chunkGroupMetadataList = new ArrayList<>();
try (TsFileSequenceReader reader = new TsFileSequenceReader(file.getAbsolutePath(), false)) {
reader.selfCheck(schemaMap, chunkGroupMetadataList, false);
+ if (plan.getMetadataCheck() && !loadNewTsFileCheckMetadata(reader)) {
+ throw new QueryProcessException(
+ "loading tsfile metadata is inconsistent with existing timeseries.");
Review comment:
print which timeseries is not consistent:
root.sg.d.s is INT32 in the loading TsFile but is FLOAT in IoTDB.
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -2293,42 +2293,35 @@ public void loadNewTsFile(TsFileResource newTsFileResource) throws LoadFileExcep
List<TsFileResource> sequenceList = tsFileManagement.getTsFileList(true);
int insertPos = findInsertionPosition(newTsFileResource, newFilePartitionId, sequenceList);
- if (insertPos == POS_ALREADY_EXIST) {
- return;
- }
+ String newFileName, renameInfo;
+ LoadTsFileType tsFileType;
// loading tsfile by type
if (insertPos == POS_OVERLAP) {
- loadTsFileByType(
- LoadTsFileType.LOAD_UNSEQUENCE,
- tsfileToBeInserted,
- newTsFileResource,
- newFilePartitionId);
+ newFileName =
+ getNewTsFileName(
+ System.currentTimeMillis(),
+ getAndSetNewVersion(newFilePartitionId, newTsFileResource),
+ 0);
+ renameInfo = "unsequence";
+ tsFileType = LoadTsFileType.LOAD_UNSEQUENCE;
} else {
-
// check whether the file name needs to be renamed.
- if (!tsFileManagement.isEmpty(true)) {
- String newFileName =
- getFileNameForLoadingFile(
- tsfileToBeInserted.getName(),
- insertPos,
- newTsFileResource.getTimePartition(),
- sequenceList);
- if (!newFileName.equals(tsfileToBeInserted.getName())) {
- logger.info(
- "Tsfile {} must be renamed to {} for loading into the sequence list.",
- tsfileToBeInserted.getName(),
- newFileName);
- newTsFileResource.setFile(
- fsFactory.getFile(tsfileToBeInserted.getParentFile(), newFileName));
- }
- }
- loadTsFileByType(
- LoadTsFileType.LOAD_SEQUENCE,
- tsfileToBeInserted,
- newTsFileResource,
- newFilePartitionId);
+ newFileName = getFileNameForSequenceLoadingFile(insertPos, newTsFileResource, sequenceList);
+ renameInfo = "sequence";
+ tsFileType = LoadTsFileType.LOAD_SEQUENCE;
+ newTsFileResource.setSeq(true);
+ }
+
+ if (!newFileName.equals(tsfileToBeInserted.getName())) {
+ logger.info(
+ "Tsfile {} must be renamed to {} for loading into the " + renameInfo + " list.",
Review comment:
```suggestion
"TsFile {} must be renamed to {} for loading into the " + renameInfo + " list.",
```
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/physical/sys/OperateFilePlan.java
##########
@@ -73,6 +88,10 @@ public int getSgLevel() {
return sgLevel;
}
+ public boolean getMetadataCheck() {
Review comment:
```suggestion
public boolean getVerifMetadata() {
```
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/executor/PlanExecutor.java
##########
@@ -1097,6 +1098,13 @@ private void loadFile(File file, OperateFilePlan plan) throws QueryProcessExcept
List<ChunkGroupMetadata> chunkGroupMetadataList = new ArrayList<>();
try (TsFileSequenceReader reader = new TsFileSequenceReader(file.getAbsolutePath(), false)) {
reader.selfCheck(schemaMap, chunkGroupMetadataList, false);
+ if (plan.getMetadataCheck() && !loadNewTsFileCheckMetadata(reader)) {
+ throw new QueryProcessException(
+ "loading tsfile metadata is inconsistent with existing timeseries.");
+ }
+ } catch (IOException e) {
+ logger.error("can not get timeseries metadata from {}.", file.getAbsoluteFile());
Review comment:
change to warn
Only system crash is error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#issuecomment-880523918
[](https://coveralls.io/builds/41541902)
Coverage increased (+0.02%) to 68.152% when pulling **f3721bdff767e59f49cb429f79b4cc3ce0143bea on yschengzi:loadrename** into **6e9f3404442108c08493351d1447e6d2c1ec1347 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] yschengzi commented on a change in pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
yschengzi commented on a change in pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#discussion_r677294548
##########
File path: docs/zh/UserGuide/System-Tools/Load-External-Tsfile.md
##########
@@ -30,31 +30,45 @@
用户通过Cli工具或JDBC向Apache IoTDB系统发送指定命令实现文件加载的功能。
#### 加载tsfile文件
-加载tsfile文件的指令为：`load "<path/dir>" [true/false] [storage group level]`
+加载tsfile文件的指令为：`load "<path/dir>" [schema=true/false][,level=int][,metadata=true/false]`
该指令有两种用法：
1. 通过指定文件路径(绝对路径)加载单tsfile文件。
-第二个参数表示待加载的tsfile文件的路径，其中文件名称需要符合tsfile的命名规范，即`{systemTime}-{versionNum}-{mergeNum}.tsfile`。第三、四个参数为可选项。当待加载的tsfile文件中时间序列对应的元数据不存在时，用户可以选择是否自动创建schema，参数为true表示自动创建schema，相反false表示不创建，缺省时默认创建schema。当tsfile对应的存储组不存在时，用户可以通过第四个参数来制定存储组的级别，默认为`iotdb-engine.properties`中设置的级别。若待加载的tsfile文件对应的`.resource`文件存在，会被一并加载至Apache IoTDB数据文件的目录和引擎中，否则将通过tsfile文件重新生成对应的`.resource`文件，即加载的tsfile文件所对应的`.resource`文件不是必要的。
+第二个参数表示待加载的tsfile文件的路径，其中文件名称需要符合tsfile的命名规范，即`{systemTime}-{versionNum}-{mergeNum}.tsfile`或者`{systemTime}-{versionNum}-{mergeNum}-{unsequenceMergeNum}.tsfile`。load命令有三个可选项，分别是schema，值域为true/false，level，值域为正整数，metadatat，值域为true/false。不同选项之间用逗号连接，选项之间无顺序要求。
+
+schema选项表示当待加载的tsfile文件中时间序列对应的元数据不存在时，用户可以选择是否自动创建schema，参数为true表示自动创建schema，相反false表示不创建，缺省时默认创建schema。
Review comment:
already fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] yschengzi commented on a change in pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
yschengzi commented on a change in pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#discussion_r677296345
##########
File path: docs/zh/UserGuide/System-Tools/Load-External-Tsfile.md
##########
@@ -30,31 +30,45 @@
用户通过Cli工具或JDBC向Apache IoTDB系统发送指定命令实现文件加载的功能。
#### 加载tsfile文件
-加载tsfile文件的指令为：`load "<path/dir>" [true/false] [storage group level]`
+加载tsfile文件的指令为：`load "<path/dir>" [autoregister=true/false][,sglevel=int][,verify=true/false]`
该指令有两种用法：
1. 通过指定文件路径(绝对路径)加载单tsfile文件。
-第二个参数表示待加载的tsfile文件的路径，其中文件名称需要符合tsfile的命名规范，即`{systemTime}-{versionNum}-{mergeNum}.tsfile`。第三、四个参数为可选项。当待加载的tsfile文件中时间序列对应的元数据不存在时，用户可以选择是否自动创建schema，参数为true表示自动创建schema，相反false表示不创建，缺省时默认创建schema。当tsfile对应的存储组不存在时，用户可以通过第四个参数来制定存储组的级别，默认为`iotdb-engine.properties`中设置的级别。若待加载的tsfile文件对应的`.resource`文件存在，会被一并加载至Apache IoTDB数据文件的目录和引擎中，否则将通过tsfile文件重新生成对应的`.resource`文件，即加载的tsfile文件所对应的`.resource`文件不是必要的。
+第二个参数表示待加载的tsfile文件的路径，其中文件名称需要符合tsfile的命名规范，即`{systemTime}-{versionNum}-{mergeNum}.tsfile`或者`{systemTime}-{versionNum}-{mergeNum}-{unsequenceMergeNum}.tsfile`。load命令有三个可选项，分别是schema，值域为true/false，level，值域为正整数，metadatat，值域为true/false。不同选项之间用逗号连接，选项之间无顺序要求。
+
+AUTOREGISTER选项表示当待加载的tsfile文件中时间序列对应的元数据不存在时，用户可以选择是否自动创建schema，参数为true表示自动创建schema，相反false表示不创建，缺省时默认创建schema。
+
+SGLEVEL选项，当tsfile对应的存储组不存在时，用户可以通过level参数的值来制定存储组的级别，默认为`iotdb-engine.properties`中设置的级别。例如当设置level参数为1时表明此tsfile中所有时间序列中层级为1的前缀路径是存储组，即若存在设备root.sg.d1.s1，此时root.sg被指定为存储组。
+
+VERIFY选项表示是否对载入的tsfile中的所有时间序列进行元数据检查，默认为true。开启时，若载入的tsfile中的时间序列在当前iotdb中也存在，则会比较该时间序列的所有Measurement的数据类型是否一致，如果出现不一致将会导致载入失败，关闭该选项会跳过检查，载入更快。
+
+若待加载的tsfile文件对应的`.resource`文件存在，会被一并加载至Apache IoTDB数据文件的目录和引擎中，否则将通过tsfile文件重新生成对应的`.resource`文件，即加载的tsfile文件所对应的`.resource`文件不是必要的。
示例：
-* load `"/Users/Desktop/data/1575028885956-101-0.tsfile"`
-* load `"/Users/Desktop/data/1575028885956-101-0.tsfile" false`
-* load `"/Users/Desktop/data/1575028885956-101-0.tsfile" true`
-* load `"/Users/Desktop/data/1575028885956-101-0.tsfile" true 1`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile"`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" autoregister=false`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" autoregister=true`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" sglevel=1`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" verify=true`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" autoregister=true,sglevel=1`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" verify=false,sglevel=1`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" autoregister=false,verify=true`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" autoregister=false,sglevel=1,verify=true`
2. 通过指定文件夹路径(绝对路径)批量加载文件。
-第二个参数表示待加载的tsfile文件的路径，其中文件名称需要符合tsfile的命名规范，即`{systemTime}-{versionNum}-{mergeNum}.tsfile`。第三、四个参数为可选项。当待加载的tsfile文件中时间序列对应的元数据不存在时，用户可以选择是否自动创建schema，参数为true表示自动创建schema，相反false表示不创建，缺省时默认创建schema。当tsfile对应的存储组不存在时，用户可以通过第四个参数来制定存储组的级别，默认为`iotdb-engine.properties`中设置的级别。若待加载文件对应的`.resource`文件存在，则会一并加载至Apache IoTDB数据文件目录和引擎中，否则将通过tsfile文件重新生成对应的`.resource`文件，即加载的tsfile文件所对应的`.resource`文件不是必要的。
+第二个参数表示待加载的tsfile文件夹的路径，其中文件夹内所有文件名称需要符合tsfile的命名规范，即`{systemTime}-{versionNum}-{mergeNum}.tsfile`或者`{systemTime}-{versionNum}-{mergeNum}-{unsequenceMergeNum}.tsfile`。选项意义与加载单个tsfile文件相同。
Review comment:
already renaming
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] yschengzi commented on a change in pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
yschengzi commented on a change in pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#discussion_r677296712
##########
File path: docs/UserGuide/System-Tools/Load-External-Tsfile.md
##########
@@ -31,33 +31,45 @@ The user sends specified commands to the Apache IoTDB system through the Cli too
#### load tsfiles
-The command to load tsfiles is `load "<path/dir>" [true/false] [storage group level]`.
+The command to load tsfiles is `load "<path/dir>" [autoregister=true/false][,sglevel=int][,verify=true/false]`.
This command has two usages:
1. Load a single tsfile by specifying a file path (absolute path).
-The second parameter indicates the path of the tsfile to be loaded and the name of the tsfile needs to conform to the tsfile naming convention, that is, `{systemTime}-{versionNum}-{mergeNum} .tsfile`. The third and fourth parameters are optional. If the metadata correspond to the timeseries in the tsfile to be loaded does not exist, you can choose whether to create the schema automatically. If the third parameter is true, the schema is created automatically. If the third parameter is false, the schema will not be created. By default, the schema will be created. If the storage group correspond to the tsfile does not exist, the user can set the level of storage group through the fourth parameter. By default, it uses the storage group level which is set in `iotdb-engine.properties`.
+The second parameter indicates the path of the tsfile to be loaded and the name of the tsfile needs to conform to the tsfile naming convention, that is, `{systemTime}-{versionNum}-{mergeNum} .tsfile` or `{systemTime}-{versionNum}-{mergeNum}-{unsequenceMergeNum}.tsfile`. This command has three options: schema, level and metadata.
+
+AUTOREGISTER option. If the metadata correspond to the timeseries in the tsfile to be loaded does not exist, you can choose whether to create the schema automatically. If this parameter is true, the schema is created automatically. If it is false, the schema will not be created. By default, the schema will be created.
+
+SGLEVEL option. If the storage group correspond to the tsfile does not exist, the user can set the level of storage group through the fourth parameter. By default, it uses the storage group level which is set in `iotdb-engine.properties`.
+
+VERIFY option. If this parameter is true, All timeseries in this loading tsfile will be compared with the timeseries in IoTDB. If existing a measurement which has different datatype with the measurement in IoTDB, the loading process will be stopped and exit. If consistence can be promised, setting false for this parameter will be a better choice.
+
If the `.resource` file corresponding to the file exists, it will be loaded into the data directory and engine of the Apache IoTDB. Otherwise, the corresponding `.resource` file will be regenerated from the tsfile file.
Examples:
* `load "/Users/Desktop/data/1575028885956-101-0.tsfile"`
-* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" false`
-* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" true`
-* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" true 1`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" autoregister=false`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" autoregister=true`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" sglevel=1`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" verify=true`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" autoregister=true,sglevel=1`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" verify=false,sglevel=1`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" autoregister=false,verify=true`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" autoregister=false,sglevel=1,verify=true`
2. Load a batch of files by specifying a folder path (absolute path).
-The second parameter indicates the path of the tsfile to be loaded and the name of the tsfiles need to conform to the tsfile naming convention, that is, `{systemTime}-{versionNum}-{mergeNum} .tsfile`. The third and fourth parameters are optional. When the metadata corresponding to the time series in the tsfile to be loaded does not exist, you can choose whether to create the schema automatically. If the third parameter is true, the schema will be created automatically. If the third parameter is false, the schema will not be created. By default, the schema will be created. When the storage group corresponding to tsfile does not exist, the user can set the level of storage group through the fourth parameter. By default, it'll use the storage group level which is set in `iotdb-engine.properties`.
-If the `.resource` file corresponding to the file exists, they will be loaded into the data directory and engine of the Apache IoTDB. Otherwise, the corresponding` .resource` files will be regenerated from the tsfile sfile.
+The second parameter indicates the path of the tsfile to be loaded and the name of the tsfiles need to conform to the tsfile naming convention, that is, `{systemTime}-{versionNum}-{mergeNum} .tsfile` or `{systemTime}-{versionNum}-{mergeNum}-{unsequenceMergeNum}.tsfile`. The options above also works for this command.
Review comment:
already renaming
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3576: [IOTDB-1504]Fix rename bug in loading tsfile
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#issuecomment-880523918
[](https://coveralls.io/builds/41538535)
Coverage increased (+0.003%) to 68.16% when pulling **b0f5d8693f6a1b8b1f0c2963ba7d7abcaa01cd29 on yschengzi:loadrename** into **5102f7f672e7e15f60b5095230a75db1e44712c2 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] qiaojialin commented on a change in pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
qiaojialin commented on a change in pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#discussion_r677109300
##########
File path: docs/UserGuide/System-Tools/Load-External-Tsfile.md
##########
@@ -31,33 +31,45 @@ The user sends specified commands to the Apache IoTDB system through the Cli too
#### load tsfiles
-The command to load tsfiles is `load "<path/dir>" [true/false] [storage group level]`.
+The command to load tsfiles is `load "<path/dir>" [autoregister=true/false][,sglevel=int][,verify=true/false]`.
This command has two usages:
1. Load a single tsfile by specifying a file path (absolute path).
-The second parameter indicates the path of the tsfile to be loaded and the name of the tsfile needs to conform to the tsfile naming convention, that is, `{systemTime}-{versionNum}-{mergeNum} .tsfile`. The third and fourth parameters are optional. If the metadata correspond to the timeseries in the tsfile to be loaded does not exist, you can choose whether to create the schema automatically. If the third parameter is true, the schema is created automatically. If the third parameter is false, the schema will not be created. By default, the schema will be created. If the storage group correspond to the tsfile does not exist, the user can set the level of storage group through the fourth parameter. By default, it uses the storage group level which is set in `iotdb-engine.properties`.
+The second parameter indicates the path of the tsfile to be loaded and the name of the tsfile needs to conform to the tsfile naming convention, that is, `{systemTime}-{versionNum}-{mergeNum} .tsfile` or `{systemTime}-{versionNum}-{mergeNum}-{unsequenceMergeNum}.tsfile`. This command has three options: schema, level and metadata.
Review comment:
```suggestion
The second parameter indicates the path of the tsfile to be loaded and the name of the tsfile needs to conform to the tsfile naming convention, that is, `{systemTime}-{versionNum}-{in_space_compaction_num}-{cross_space_compaction_num}.tsfile`. This command has three options: schema, level and metadata.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] qiaojialin merged pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
qiaojialin merged pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] yschengzi commented on pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
yschengzi commented on pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#issuecomment-884158780
update:
add a option for loading tsfile sql statement.
Now the statement for loading tsfile looks like this:
{code:sql}
load "<path/dir>" [schema=true/false] [level=int] [metadata=true/false]
{code}
for example,
we want to load tsfiles with auto creating schema, and storage group level is 1, and need metadata check, we can input sql:
{code:sql}
load "/Users/Desktop/data/1575028885956-101-0.tsfile" schema=true,level=1,meatadata=true
{code}
if the options is missing, the default value is "schema=true,level=1,metadata=true".
for example, when we input sql:
{code:sql}
load "/Users/Desktop/data/1575028885956-101-0.tsfile"
{code}
means:
{code:sql}
load "/Users/Desktop/data/1575028885956-101-0.tsfile" schema=true,level=1,meatadata=true
{code}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] yschengzi commented on a change in pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
yschengzi commented on a change in pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#discussion_r677297437
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1308,7 +1308,7 @@ private String getNewTsFileName(long timePartitionId) {
}
private String getNewTsFileName(long time, long version, int mergeCnt) {
Review comment:
change method parameter to (time, version, mergeCnt, unSeqMergeCnt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] yschengzi commented on a change in pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
yschengzi commented on a change in pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#discussion_r677299709
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/executor/PlanExecutor.java
##########
@@ -1097,6 +1098,13 @@ private void loadFile(File file, OperateFilePlan plan) throws QueryProcessExcept
List<ChunkGroupMetadata> chunkGroupMetadataList = new ArrayList<>();
try (TsFileSequenceReader reader = new TsFileSequenceReader(file.getAbsolutePath(), false)) {
reader.selfCheck(schemaMap, chunkGroupMetadataList, false);
+ if (plan.getMetadataCheck() && !loadNewTsFileCheckMetadata(reader)) {
+ throw new QueryProcessException(
+ "loading tsfile metadata is inconsistent with existing timeseries.");
Review comment:
i have moved this info to method "loadNewTsFileVerifyMetadata"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#issuecomment-880523918
[](https://coveralls.io/builds/41675558)
Coverage increased (+0.01%) to 68.08% when pulling **d59403358035375bede63821c3aed8c927fa32ab on yschengzi:loadrename** into **47fa594403d5bb0c79d8414886b5a9fe6a24b778 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls commented on pull request #3576: [IOTDB-1504]Fix rename bug in loading tsfile
Posted by GitBox <gi...@apache.org>.
coveralls commented on pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#issuecomment-880523918
[](https://coveralls.io/builds/41391358)
Coverage decreased (-0.001%) to 68.155% when pulling **ffa49b8e7db15ebeb31ab20be87bedd86119f599 on yschengzi:loadrename** into **5102f7f672e7e15f60b5095230a75db1e44712c2 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] yschengzi commented on a change in pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
yschengzi commented on a change in pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#discussion_r677300187
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/physical/sys/OperateFilePlan.java
##########
@@ -73,6 +88,10 @@ public int getSgLevel() {
return sgLevel;
}
+ public boolean getMetadataCheck() {
Review comment:
renaming it already
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] yschengzi commented on a change in pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
yschengzi commented on a change in pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#discussion_r677297042
##########
File path: docs/UserGuide/System-Tools/Load-External-Tsfile.md
##########
@@ -31,33 +31,45 @@ The user sends specified commands to the Apache IoTDB system through the Cli too
#### load tsfiles
-The command to load tsfiles is `load "<path/dir>" [true/false] [storage group level]`.
+The command to load tsfiles is `load "<path/dir>" [autoregister=true/false][,sglevel=int][,verify=true/false]`.
This command has two usages:
1. Load a single tsfile by specifying a file path (absolute path).
-The second parameter indicates the path of the tsfile to be loaded and the name of the tsfile needs to conform to the tsfile naming convention, that is, `{systemTime}-{versionNum}-{mergeNum} .tsfile`. The third and fourth parameters are optional. If the metadata correspond to the timeseries in the tsfile to be loaded does not exist, you can choose whether to create the schema automatically. If the third parameter is true, the schema is created automatically. If the third parameter is false, the schema will not be created. By default, the schema will be created. If the storage group correspond to the tsfile does not exist, the user can set the level of storage group through the fourth parameter. By default, it uses the storage group level which is set in `iotdb-engine.properties`.
+The second parameter indicates the path of the tsfile to be loaded and the name of the tsfile needs to conform to the tsfile naming convention, that is, `{systemTime}-{versionNum}-{mergeNum} .tsfile` or `{systemTime}-{versionNum}-{mergeNum}-{unsequenceMergeNum}.tsfile`. This command has three options: schema, level and metadata.
Review comment:
already renaming
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] yschengzi commented on a change in pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
yschengzi commented on a change in pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#discussion_r677297756
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -2293,42 +2293,35 @@ public void loadNewTsFile(TsFileResource newTsFileResource) throws LoadFileExcep
List<TsFileResource> sequenceList = tsFileManagement.getTsFileList(true);
int insertPos = findInsertionPosition(newTsFileResource, newFilePartitionId, sequenceList);
- if (insertPos == POS_ALREADY_EXIST) {
- return;
- }
+ String newFileName, renameInfo;
+ LoadTsFileType tsFileType;
// loading tsfile by type
if (insertPos == POS_OVERLAP) {
- loadTsFileByType(
- LoadTsFileType.LOAD_UNSEQUENCE,
- tsfileToBeInserted,
- newTsFileResource,
- newFilePartitionId);
+ newFileName =
+ getNewTsFileName(
+ System.currentTimeMillis(),
+ getAndSetNewVersion(newFilePartitionId, newTsFileResource),
+ 0);
+ renameInfo = "unsequence";
+ tsFileType = LoadTsFileType.LOAD_UNSEQUENCE;
} else {
-
// check whether the file name needs to be renamed.
- if (!tsFileManagement.isEmpty(true)) {
- String newFileName =
- getFileNameForLoadingFile(
- tsfileToBeInserted.getName(),
- insertPos,
- newTsFileResource.getTimePartition(),
- sequenceList);
- if (!newFileName.equals(tsfileToBeInserted.getName())) {
- logger.info(
- "Tsfile {} must be renamed to {} for loading into the sequence list.",
- tsfileToBeInserted.getName(),
- newFileName);
- newTsFileResource.setFile(
- fsFactory.getFile(tsfileToBeInserted.getParentFile(), newFileName));
- }
- }
- loadTsFileByType(
- LoadTsFileType.LOAD_SEQUENCE,
- tsfileToBeInserted,
- newTsFileResource,
- newFilePartitionId);
+ newFileName = getFileNameForSequenceLoadingFile(insertPos, newTsFileResource, sequenceList);
+ renameInfo = "sequence";
Review comment:
use constant from IoTDBConstant
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -2293,42 +2293,35 @@ public void loadNewTsFile(TsFileResource newTsFileResource) throws LoadFileExcep
List<TsFileResource> sequenceList = tsFileManagement.getTsFileList(true);
int insertPos = findInsertionPosition(newTsFileResource, newFilePartitionId, sequenceList);
- if (insertPos == POS_ALREADY_EXIST) {
- return;
- }
+ String newFileName, renameInfo;
+ LoadTsFileType tsFileType;
// loading tsfile by type
if (insertPos == POS_OVERLAP) {
- loadTsFileByType(
- LoadTsFileType.LOAD_UNSEQUENCE,
- tsfileToBeInserted,
- newTsFileResource,
- newFilePartitionId);
+ newFileName =
+ getNewTsFileName(
+ System.currentTimeMillis(),
+ getAndSetNewVersion(newFilePartitionId, newTsFileResource),
+ 0);
+ renameInfo = "unsequence";
+ tsFileType = LoadTsFileType.LOAD_UNSEQUENCE;
Review comment:
already set
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -2293,42 +2293,35 @@ public void loadNewTsFile(TsFileResource newTsFileResource) throws LoadFileExcep
List<TsFileResource> sequenceList = tsFileManagement.getTsFileList(true);
int insertPos = findInsertionPosition(newTsFileResource, newFilePartitionId, sequenceList);
- if (insertPos == POS_ALREADY_EXIST) {
- return;
- }
+ String newFileName, renameInfo;
+ LoadTsFileType tsFileType;
// loading tsfile by type
if (insertPos == POS_OVERLAP) {
- loadTsFileByType(
- LoadTsFileType.LOAD_UNSEQUENCE,
- tsfileToBeInserted,
- newTsFileResource,
- newFilePartitionId);
+ newFileName =
+ getNewTsFileName(
+ System.currentTimeMillis(),
+ getAndSetNewVersion(newFilePartitionId, newTsFileResource),
+ 0);
+ renameInfo = "unsequence";
+ tsFileType = LoadTsFileType.LOAD_UNSEQUENCE;
} else {
-
// check whether the file name needs to be renamed.
- if (!tsFileManagement.isEmpty(true)) {
- String newFileName =
- getFileNameForLoadingFile(
- tsfileToBeInserted.getName(),
- insertPos,
- newTsFileResource.getTimePartition(),
- sequenceList);
- if (!newFileName.equals(tsfileToBeInserted.getName())) {
- logger.info(
- "Tsfile {} must be renamed to {} for loading into the sequence list.",
- tsfileToBeInserted.getName(),
- newFileName);
- newTsFileResource.setFile(
- fsFactory.getFile(tsfileToBeInserted.getParentFile(), newFileName));
- }
- }
- loadTsFileByType(
- LoadTsFileType.LOAD_SEQUENCE,
- tsfileToBeInserted,
- newTsFileResource,
- newFilePartitionId);
+ newFileName = getFileNameForSequenceLoadingFile(insertPos, newTsFileResource, sequenceList);
+ renameInfo = "sequence";
+ tsFileType = LoadTsFileType.LOAD_SEQUENCE;
+ newTsFileResource.setSeq(true);
+ }
+
+ if (!newFileName.equals(tsfileToBeInserted.getName())) {
+ logger.info(
+ "Tsfile {} must be renamed to {} for loading into the " + renameInfo + " list.",
Review comment:
already fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#issuecomment-880523918
[](https://coveralls.io/builds/41563492)
Coverage increased (+0.03%) to 68.165% when pulling **cc1b1c58640bb85d493942f1fd1c3d54f73ef2a1 on yschengzi:loadrename** into **6e9f3404442108c08493351d1447e6d2c1ec1347 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] yschengzi commented on a change in pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
yschengzi commented on a change in pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#discussion_r677299961
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/executor/PlanExecutor.java
##########
@@ -1097,6 +1098,13 @@ private void loadFile(File file, OperateFilePlan plan) throws QueryProcessExcept
List<ChunkGroupMetadata> chunkGroupMetadataList = new ArrayList<>();
try (TsFileSequenceReader reader = new TsFileSequenceReader(file.getAbsolutePath(), false)) {
reader.selfCheck(schemaMap, chunkGroupMetadataList, false);
+ if (plan.getMetadataCheck() && !loadNewTsFileCheckMetadata(reader)) {
+ throw new QueryProcessException(
+ "loading tsfile metadata is inconsistent with existing timeseries.");
+ }
+ } catch (IOException e) {
+ logger.error("can not get timeseries metadata from {}.", file.getAbsoluteFile());
Review comment:
ok, already change it to warn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] yschengzi commented on a change in pull request #3576: [IOTDB-1504]Fix rename bug in loading tsfile
Posted by GitBox <gi...@apache.org>.
yschengzi commented on a change in pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#discussion_r670980102
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -90,20 +90,8 @@
import java.nio.ByteBuffer;
import java.nio.MappedByteBuffer;
import java.nio.file.Files;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collection;
-import java.util.Collections;
-import java.util.Date;
-import java.util.Deque;
-import java.util.HashMap;
-import java.util.Iterator;
-import java.util.LinkedList;
-import java.util.List;
-import java.util.Map;
+import java.util.*;
Review comment:
i see, thanks for reminding😀
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#issuecomment-880523918
[](https://coveralls.io/builds/41669006)
Coverage increased (+0.004%) to 68.073% when pulling **668c037da96d0b7e2059efe197d60d36ad6ae11c on yschengzi:loadrename** into **47fa594403d5bb0c79d8414886b5a9fe6a24b778 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] yschengzi commented on a change in pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
yschengzi commented on a change in pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#discussion_r677300369
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/physical/sys/OperateFilePlan.java
##########
@@ -32,18 +32,33 @@
private File targetDir;
private boolean autoCreateSchema;
private int sgLevel;
+ private boolean metadataCheck;
Review comment:
renming done
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/physical/sys/OperateFilePlan.java
##########
@@ -84,6 +103,8 @@ public String toString() {
+ autoCreateSchema
+ ", sgLevel="
+ sgLevel
+ + ", metadataCheck="
Review comment:
already fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] qiaojialin commented on a change in pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
qiaojialin commented on a change in pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#discussion_r677141302
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1308,7 +1308,7 @@ private String getNewTsFileName(long timePartitionId) {
}
private String getNewTsFileName(long time, long version, int mergeCnt) {
Review comment:
add a parameter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] qiaojialin commented on a change in pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
qiaojialin commented on a change in pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#discussion_r677109142
##########
File path: docs/UserGuide/System-Tools/Load-External-Tsfile.md
##########
@@ -31,33 +31,45 @@ The user sends specified commands to the Apache IoTDB system through the Cli too
#### load tsfiles
-The command to load tsfiles is `load "<path/dir>" [true/false] [storage group level]`.
+The command to load tsfiles is `load "<path/dir>" [autoregister=true/false][,sglevel=int][,verify=true/false]`.
This command has two usages:
1. Load a single tsfile by specifying a file path (absolute path).
-The second parameter indicates the path of the tsfile to be loaded and the name of the tsfile needs to conform to the tsfile naming convention, that is, `{systemTime}-{versionNum}-{mergeNum} .tsfile`. The third and fourth parameters are optional. If the metadata correspond to the timeseries in the tsfile to be loaded does not exist, you can choose whether to create the schema automatically. If the third parameter is true, the schema is created automatically. If the third parameter is false, the schema will not be created. By default, the schema will be created. If the storage group correspond to the tsfile does not exist, the user can set the level of storage group through the fourth parameter. By default, it uses the storage group level which is set in `iotdb-engine.properties`.
+The second parameter indicates the path of the tsfile to be loaded and the name of the tsfile needs to conform to the tsfile naming convention, that is, `{systemTime}-{versionNum}-{mergeNum} .tsfile` or `{systemTime}-{versionNum}-{mergeNum}-{unsequenceMergeNum}.tsfile`. This command has three options: schema, level and metadata.
+
+AUTOREGISTER option. If the metadata correspond to the timeseries in the tsfile to be loaded does not exist, you can choose whether to create the schema automatically. If this parameter is true, the schema is created automatically. If it is false, the schema will not be created. By default, the schema will be created.
+
+SGLEVEL option. If the storage group correspond to the tsfile does not exist, the user can set the level of storage group through the fourth parameter. By default, it uses the storage group level which is set in `iotdb-engine.properties`.
+
+VERIFY option. If this parameter is true, All timeseries in this loading tsfile will be compared with the timeseries in IoTDB. If existing a measurement which has different datatype with the measurement in IoTDB, the loading process will be stopped and exit. If consistence can be promised, setting false for this parameter will be a better choice.
+
If the `.resource` file corresponding to the file exists, it will be loaded into the data directory and engine of the Apache IoTDB. Otherwise, the corresponding `.resource` file will be regenerated from the tsfile file.
Examples:
* `load "/Users/Desktop/data/1575028885956-101-0.tsfile"`
-* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" false`
-* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" true`
-* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" true 1`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" autoregister=false`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" autoregister=true`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" sglevel=1`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" verify=true`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" autoregister=true,sglevel=1`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" verify=false,sglevel=1`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" autoregister=false,verify=true`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" autoregister=false,sglevel=1,verify=true`
2. Load a batch of files by specifying a folder path (absolute path).
-The second parameter indicates the path of the tsfile to be loaded and the name of the tsfiles need to conform to the tsfile naming convention, that is, `{systemTime}-{versionNum}-{mergeNum} .tsfile`. The third and fourth parameters are optional. When the metadata corresponding to the time series in the tsfile to be loaded does not exist, you can choose whether to create the schema automatically. If the third parameter is true, the schema will be created automatically. If the third parameter is false, the schema will not be created. By default, the schema will be created. When the storage group corresponding to tsfile does not exist, the user can set the level of storage group through the fourth parameter. By default, it'll use the storage group level which is set in `iotdb-engine.properties`.
-If the `.resource` file corresponding to the file exists, they will be loaded into the data directory and engine of the Apache IoTDB. Otherwise, the corresponding` .resource` files will be regenerated from the tsfile sfile.
+The second parameter indicates the path of the tsfile to be loaded and the name of the tsfiles need to conform to the tsfile naming convention, that is, `{systemTime}-{versionNum}-{mergeNum} .tsfile` or `{systemTime}-{versionNum}-{mergeNum}-{unsequenceMergeNum}.tsfile`. The options above also works for this command.
Review comment:
```suggestion
The second parameter indicates the path of the tsfile to be loaded and the name of the tsfiles need to conform to the tsfile naming convention, that is, `{systemTime}-{versionNum}-{mergeNum}-{unsequenceMergeNum}.tsfile`. The options above also works for this command.
```
##########
File path: docs/UserGuide/System-Tools/Load-External-Tsfile.md
##########
@@ -31,33 +31,45 @@ The user sends specified commands to the Apache IoTDB system through the Cli too
#### load tsfiles
-The command to load tsfiles is `load "<path/dir>" [true/false] [storage group level]`.
+The command to load tsfiles is `load "<path/dir>" [autoregister=true/false][,sglevel=int][,verify=true/false]`.
This command has two usages:
1. Load a single tsfile by specifying a file path (absolute path).
-The second parameter indicates the path of the tsfile to be loaded and the name of the tsfile needs to conform to the tsfile naming convention, that is, `{systemTime}-{versionNum}-{mergeNum} .tsfile`. The third and fourth parameters are optional. If the metadata correspond to the timeseries in the tsfile to be loaded does not exist, you can choose whether to create the schema automatically. If the third parameter is true, the schema is created automatically. If the third parameter is false, the schema will not be created. By default, the schema will be created. If the storage group correspond to the tsfile does not exist, the user can set the level of storage group through the fourth parameter. By default, it uses the storage group level which is set in `iotdb-engine.properties`.
+The second parameter indicates the path of the tsfile to be loaded and the name of the tsfile needs to conform to the tsfile naming convention, that is, `{systemTime}-{versionNum}-{mergeNum} .tsfile` or `{systemTime}-{versionNum}-{mergeNum}-{unsequenceMergeNum}.tsfile`. This command has three options: schema, level and metadata.
Review comment:
```suggestion
The second parameter indicates the path of the tsfile to be loaded and the name of the tsfile needs to conform to the tsfile naming convention, that is, `{systemTime}-{versionNum}-{mergeNum}-{unsequenceMergeNum}.tsfile`. This command has three options: schema, level and metadata.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] yschengzi commented on a change in pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
yschengzi commented on a change in pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#discussion_r677298086
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -2538,59 +2524,57 @@ private void removeFullyOverlapFile(
/**
* Get an appropriate filename to ensure the order between files. The tsfile is named after
- * ({systemTime}-{versionNum}-{mergeNum}.tsfile).
+ * ({systemTime}-{versionNum}-{mergeNum}-{unsequenceMergeNum}.tsfile).
Review comment:
fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] yschengzi edited a comment on pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
yschengzi edited a comment on pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#issuecomment-884158780
update:
add a option for loading tsfile sql statement.
Now the statement for loading tsfile looks like this:
load "<path/dir>" [schema=true/false] [level=int] [metadata=true/false]
for example,
we want to load tsfiles with auto creating schema, and storage group level is 1, and need metadata check, we can input sql:
load "/Users/Desktop/data/1575028885956-101-0.tsfile" schema=true,level=1,meatadata=true
if the options is missing, the default value is "schema=true,level=1,metadata=true".
for example, when we input sql:
load "/Users/Desktop/data/1575028885956-101-0.tsfile"
means:
load "/Users/Desktop/data/1575028885956-101-0.tsfile" schema=true,level=1,meatadata=true
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] qiaojialin commented on a change in pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
qiaojialin commented on a change in pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#discussion_r676253033
##########
File path: docs/zh/UserGuide/System-Tools/Load-External-Tsfile.md
##########
@@ -30,31 +30,45 @@
用户通过Cli工具或JDBC向Apache IoTDB系统发送指定命令实现文件加载的功能。
#### 加载tsfile文件
-加载tsfile文件的指令为：`load "<path/dir>" [true/false] [storage group level]`
+加载tsfile文件的指令为：`load "<path/dir>" [schema=true/false][,level=int][,metadata=true/false]`
该指令有两种用法：
1. 通过指定文件路径(绝对路径)加载单tsfile文件。
-第二个参数表示待加载的tsfile文件的路径，其中文件名称需要符合tsfile的命名规范，即`{systemTime}-{versionNum}-{mergeNum}.tsfile`。第三、四个参数为可选项。当待加载的tsfile文件中时间序列对应的元数据不存在时，用户可以选择是否自动创建schema，参数为true表示自动创建schema，相反false表示不创建，缺省时默认创建schema。当tsfile对应的存储组不存在时，用户可以通过第四个参数来制定存储组的级别，默认为`iotdb-engine.properties`中设置的级别。若待加载的tsfile文件对应的`.resource`文件存在，会被一并加载至Apache IoTDB数据文件的目录和引擎中，否则将通过tsfile文件重新生成对应的`.resource`文件，即加载的tsfile文件所对应的`.resource`文件不是必要的。
+第二个参数表示待加载的tsfile文件的路径，其中文件名称需要符合tsfile的命名规范，即`{systemTime}-{versionNum}-{mergeNum}.tsfile`或者`{systemTime}-{versionNum}-{mergeNum}-{unsequenceMergeNum}.tsfile`。load命令有三个可选项，分别是schema，值域为true/false，level，值域为正整数，metadatat，值域为true/false。不同选项之间用逗号连接，选项之间无顺序要求。
+
+schema选项表示当待加载的tsfile文件中时间序列对应的元数据不存在时，用户可以选择是否自动创建schema，参数为true表示自动创建schema，相反false表示不创建，缺省时默认创建schema。
Review comment:
autoregister
verify
sglevel
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] yschengzi commented on a change in pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
yschengzi commented on a change in pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#discussion_r677297656
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -2293,42 +2293,35 @@ public void loadNewTsFile(TsFileResource newTsFileResource) throws LoadFileExcep
List<TsFileResource> sequenceList = tsFileManagement.getTsFileList(true);
int insertPos = findInsertionPosition(newTsFileResource, newFilePartitionId, sequenceList);
- if (insertPos == POS_ALREADY_EXIST) {
- return;
- }
+ String newFileName, renameInfo;
+ LoadTsFileType tsFileType;
// loading tsfile by type
if (insertPos == POS_OVERLAP) {
- loadTsFileByType(
- LoadTsFileType.LOAD_UNSEQUENCE,
- tsfileToBeInserted,
- newTsFileResource,
- newFilePartitionId);
+ newFileName =
+ getNewTsFileName(
+ System.currentTimeMillis(),
+ getAndSetNewVersion(newFilePartitionId, newTsFileResource),
+ 0);
+ renameInfo = "unsequence";
Review comment:
use constant from IoTDBConstant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] yschengzi commented on a change in pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
yschengzi commented on a change in pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#discussion_r677298086
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -2538,59 +2524,57 @@ private void removeFullyOverlapFile(
/**
* Get an appropriate filename to ensure the order between files. The tsfile is named after
- * ({systemTime}-{versionNum}-{mergeNum}.tsfile).
+ * ({systemTime}-{versionNum}-{mergeNum}-{unsequenceMergeNum}.tsfile).
Review comment:
fixed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] qiaojialin commented on a change in pull request #3576: [IOTDB-1519]add metadata checking option and change grammar for loading tsfile
Posted by GitBox <gi...@apache.org>.
qiaojialin commented on a change in pull request #3576:
URL: https://github.com/apache/iotdb/pull/3576#discussion_r677108977
##########
File path: docs/zh/UserGuide/System-Tools/Load-External-Tsfile.md
##########
@@ -30,31 +30,45 @@
用户通过Cli工具或JDBC向Apache IoTDB系统发送指定命令实现文件加载的功能。
#### 加载tsfile文件
-加载tsfile文件的指令为：`load "<path/dir>" [true/false] [storage group level]`
+加载tsfile文件的指令为：`load "<path/dir>" [autoregister=true/false][,sglevel=int][,verify=true/false]`
该指令有两种用法：
1. 通过指定文件路径(绝对路径)加载单tsfile文件。
-第二个参数表示待加载的tsfile文件的路径，其中文件名称需要符合tsfile的命名规范，即`{systemTime}-{versionNum}-{mergeNum}.tsfile`。第三、四个参数为可选项。当待加载的tsfile文件中时间序列对应的元数据不存在时，用户可以选择是否自动创建schema，参数为true表示自动创建schema，相反false表示不创建，缺省时默认创建schema。当tsfile对应的存储组不存在时，用户可以通过第四个参数来制定存储组的级别，默认为`iotdb-engine.properties`中设置的级别。若待加载的tsfile文件对应的`.resource`文件存在，会被一并加载至Apache IoTDB数据文件的目录和引擎中，否则将通过tsfile文件重新生成对应的`.resource`文件，即加载的tsfile文件所对应的`.resource`文件不是必要的。
+第二个参数表示待加载的tsfile文件的路径，其中文件名称需要符合tsfile的命名规范，即`{systemTime}-{versionNum}-{mergeNum}.tsfile`或者`{systemTime}-{versionNum}-{mergeNum}-{unsequenceMergeNum}.tsfile`。load命令有三个可选项，分别是schema，值域为true/false，level，值域为正整数，metadatat，值域为true/false。不同选项之间用逗号连接，选项之间无顺序要求。
+
+AUTOREGISTER选项表示当待加载的tsfile文件中时间序列对应的元数据不存在时，用户可以选择是否自动创建schema，参数为true表示自动创建schema，相反false表示不创建，缺省时默认创建schema。
+
+SGLEVEL选项，当tsfile对应的存储组不存在时，用户可以通过level参数的值来制定存储组的级别，默认为`iotdb-engine.properties`中设置的级别。例如当设置level参数为1时表明此tsfile中所有时间序列中层级为1的前缀路径是存储组，即若存在设备root.sg.d1.s1，此时root.sg被指定为存储组。
+
+VERIFY选项表示是否对载入的tsfile中的所有时间序列进行元数据检查，默认为true。开启时，若载入的tsfile中的时间序列在当前iotdb中也存在，则会比较该时间序列的所有Measurement的数据类型是否一致，如果出现不一致将会导致载入失败，关闭该选项会跳过检查，载入更快。
+
+若待加载的tsfile文件对应的`.resource`文件存在，会被一并加载至Apache IoTDB数据文件的目录和引擎中，否则将通过tsfile文件重新生成对应的`.resource`文件，即加载的tsfile文件所对应的`.resource`文件不是必要的。
示例：
-* load `"/Users/Desktop/data/1575028885956-101-0.tsfile"`
-* load `"/Users/Desktop/data/1575028885956-101-0.tsfile" false`
-* load `"/Users/Desktop/data/1575028885956-101-0.tsfile" true`
-* load `"/Users/Desktop/data/1575028885956-101-0.tsfile" true 1`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile"`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" autoregister=false`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" autoregister=true`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" sglevel=1`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" verify=true`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" autoregister=true,sglevel=1`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" verify=false,sglevel=1`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" autoregister=false,verify=true`
+* `load "/Users/Desktop/data/1575028885956-101-0.tsfile" autoregister=false,sglevel=1,verify=true`
2. 通过指定文件夹路径(绝对路径)批量加载文件。
-第二个参数表示待加载的tsfile文件的路径，其中文件名称需要符合tsfile的命名规范，即`{systemTime}-{versionNum}-{mergeNum}.tsfile`。第三、四个参数为可选项。当待加载的tsfile文件中时间序列对应的元数据不存在时，用户可以选择是否自动创建schema，参数为true表示自动创建schema，相反false表示不创建，缺省时默认创建schema。当tsfile对应的存储组不存在时，用户可以通过第四个参数来制定存储组的级别，默认为`iotdb-engine.properties`中设置的级别。若待加载文件对应的`.resource`文件存在，则会一并加载至Apache IoTDB数据文件目录和引擎中，否则将通过tsfile文件重新生成对应的`.resource`文件，即加载的tsfile文件所对应的`.resource`文件不是必要的。
+第二个参数表示待加载的tsfile文件夹的路径，其中文件夹内所有文件名称需要符合tsfile的命名规范，即`{systemTime}-{versionNum}-{mergeNum}.tsfile`或者`{systemTime}-{versionNum}-{mergeNum}-{unsequenceMergeNum}.tsfile`。选项意义与加载单个tsfile文件相同。
Review comment:
```suggestion
第二个参数表示待加载的tsfile文件夹的路径，其中文件夹内所有文件名称需要符合tsfile的命名规范，即`{systemTime}-{versionNum}-{mergeNum}-{unsequenceMergeNum}.tsfile`。选项意义与加载单个tsfile文件相同。
```
##########
File path: docs/zh/UserGuide/System-Tools/Load-External-Tsfile.md
##########
@@ -30,31 +30,45 @@
用户通过Cli工具或JDBC向Apache IoTDB系统发送指定命令实现文件加载的功能。
#### 加载tsfile文件
-加载tsfile文件的指令为：`load "<path/dir>" [true/false] [storage group level]`
+加载tsfile文件的指令为：`load "<path/dir>" [autoregister=true/false][,sglevel=int][,verify=true/false]`
该指令有两种用法：
1. 通过指定文件路径(绝对路径)加载单tsfile文件。
-第二个参数表示待加载的tsfile文件的路径，其中文件名称需要符合tsfile的命名规范，即`{systemTime}-{versionNum}-{mergeNum}.tsfile`。第三、四个参数为可选项。当待加载的tsfile文件中时间序列对应的元数据不存在时，用户可以选择是否自动创建schema，参数为true表示自动创建schema，相反false表示不创建，缺省时默认创建schema。当tsfile对应的存储组不存在时，用户可以通过第四个参数来制定存储组的级别，默认为`iotdb-engine.properties`中设置的级别。若待加载的tsfile文件对应的`.resource`文件存在，会被一并加载至Apache IoTDB数据文件的目录和引擎中，否则将通过tsfile文件重新生成对应的`.resource`文件，即加载的tsfile文件所对应的`.resource`文件不是必要的。
+第二个参数表示待加载的tsfile文件的路径，其中文件名称需要符合tsfile的命名规范，即`{systemTime}-{versionNum}-{mergeNum}.tsfile`或者`{systemTime}-{versionNum}-{mergeNum}-{unsequenceMergeNum}.tsfile`。load命令有三个可选项，分别是schema，值域为true/false，level，值域为正整数，metadatat，值域为true/false。不同选项之间用逗号连接，选项之间无顺序要求。
Review comment:
```suggestion
第二个参数表示待加载的tsfile文件的路径，其中文件名称需要符合tsfile的命名规范，即`{systemTime}-{versionNum}-{mergeNum}-{unsequenceMergeNum}.tsfile`。load命令有三个可选项，分别是schema，值域为true/false，level，值域为正整数，metadatat，值域为true/false。不同选项之间用逗号连接，选项之间无顺序要求。
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org