[GitHub] [incubator-iotdb] LeiRui commented on a change in pull request #597: [IOTDB-292]Add load external tsfile feature

[GitBox <gi...@apache.org>]

LeiRui commented on a change in pull request #597: [IOTDB-292]Add load external tsfile feature
URL: https://github.com/apache/incubator-iotdb/pull/597#discussion_r355191216
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/qp/executor/QueryProcessExecutor.java
 ##########
 @@ -135,12 +152,136 @@ public boolean processNonQuery(PhysicalPlan plan) throws QueryProcessException {
       case LOAD_CONFIGURATION:
         IoTDBDescriptor.getInstance().loadHotModifiedProps();
         return true;
+      case LOAD_FILES:
+        operateLoadFiles((OperateFilePlan) plan);
+        return true;
+      case REMOVE_FILE:
+        operateRemoveFile((OperateFilePlan) plan);
+        return true;
+      case MOVE_FILE:
+        operateMoveFile((OperateFilePlan) plan);
+        return true;
       default:
         throw new UnsupportedOperationException(
             String.format("operation %s is not supported", plan.getOperatorType()));
     }
   }
 
+  private void operateLoadFiles(OperateFilePlan plan) throws QueryProcessException {
+    File file = plan.getFile();
+    if (!file.exists()) {
+      throw new QueryProcessException(
+          String.format("File path %s doesn't exists.", file.getPath()));
+    }
+    if (file.isDirectory()) {
+      recursionFileDir(file, plan);
+    } else {
+      loadFile(file, plan);
+    }
+  }
+
+  private void recursionFileDir(File curFile, OperateFilePlan plan) throws QueryProcessException {
+    File[] files = curFile.listFiles();
+    for (File file : files) {
+      if (file.isDirectory()) {
+        recursionFileDir(file, plan);
+      } else {
+        loadFile(file, plan);
+      }
+    }
+  }
+
+  private void loadFile(File file, OperateFilePlan plan) throws QueryProcessException {
+    if (!file.getName().endsWith(TSFILE_SUFFIX)) {
+      return;
+    }
+    TsFileResource tsFileResource = new TsFileResource(file);
+    try {
+      // check file
+      RestorableTsFileIOWriter restorableTsFileIOWriter = new RestorableTsFileIOWriter(file);
+      if (restorableTsFileIOWriter.hasCrashed()){
+        restorableTsFileIOWriter.close();
+        throw new QueryProcessException(
+            String.format("Cannot load file %s because the file has crashed.", file.getAbsolutePath()));
+      }
+      Map<String, MeasurementSchema> schemaMap = new HashMap<>();
+      List<ChunkGroupMetaData> chunkGroupMetaData = new ArrayList<>();
+      try (TsFileSequenceReader reader = new TsFileSequenceReader(file.getAbsolutePath(), false)) {
+        reader.selfCheck(schemaMap, chunkGroupMetaData, false);
 
 Review comment:
   From my understanding, the `reader.selfCheck` method with the parameter `fastFinish` assigned false will sequentially read the whole data part of the tsfile.
   If the method here is only used to generate the schemaMap and chunkGroupMetaData, why not just read the metadata at the end of the tsfile, since the tsfile is always complete here?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services