[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request #983: Online Upgrade Tool

[GitBox <gi...@apache.org>]

jt2594838 commented on a change in pull request #983:
URL: https://github.com/apache/incubator-iotdb/pull/983#discussion_r422752029



##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -298,6 +319,46 @@ private void updatePartitionFileVersion(long partitionNum, long fileVersion) {
     }
   }
 
+  /**
+   * use old seq file to update latestTimeForEachDevice, globalLatestFlushedTimeForEachDevice,
+   * partitionLatestFlushedTimeForEachDevice and timePartitionIdVersionControllerMap
+   *
+   */
+  private void doUpdate() throws IOException {

Review comment:
       I think a name like `updateLastestFlushedTime` suits better.

##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -316,40 +377,93 @@ private VersionController getVersionControllerByTimePartitionId(long timePartiti
         });
   }
 
-  private List<TsFileResource> getAllFiles(List<String> folders) {
+  private Pair<List<TsFileResource>, List<TsFileResource>> getAllFiles(List<String> folders) throws IOException {
     List<File> tsFiles = new ArrayList<>();
+    List<File> upgradeFiles = new ArrayList<>();
     for (String baseDir : folders) {
       File fileFolder = fsFactory.getFile(baseDir, storageGroupName);
       if (!fileFolder.exists()) {
         continue;
       }
 
+      // old version
+      // some TsFileResource may be being persisted when the system crashed, try recovering such
+      // resources
+      continueFailedRenames(fileFolder, TEMP_SUFFIX);
+
+      // some TsFiles were going to be replaced by the merged files when the system crashed and
+      // the process was interrupted before the merged files could be named
+      continueFailedRenames(fileFolder, MERGE_SUFFIX);
+
+      File[] oldTsfileArray = fsFactory.listFilesBySuffix(fileFolder.getAbsolutePath(), TSFILE_SUFFIX);
+      File[] oldResourceFileArray = fsFactory.listFilesBySuffix(fileFolder.getAbsolutePath(), TsFileResource.RESOURCE_SUFFIX);
+      File upgradeFolder = fsFactory.getFile(fileFolder, IoTDBConstant.UPGRADE_FOLDER_NAME);
+      // move the old files to upgrade folder if exists
+      if (oldTsfileArray.length != 0 || oldResourceFileArray.length != 0) {
+        // create upgrade directory if not exist
+        if (upgradeFolder.mkdirs()) {
+          logger.info("Upgrade Directory {} doesn't exist, create it",
+              upgradeFolder.getPath());
+        } else if (!upgradeFolder.exists()) {
+          logger.error("Create upgrade Directory {} failed",
+              upgradeFolder.getPath());
+        }
+        // move .tsfile to upgrade folder
+        for (File file : oldTsfileArray) {
+          if (!file.renameTo(fsFactory.getFile(upgradeFolder, file.getName()))) {
+            logger.error("Failed to move {} to upgrade folder", file);
+          }
+        }
+        // move .resource to upgrade folder
+        for (File file : oldResourceFileArray) {
+          if (!file.renameTo(fsFactory.getFile(upgradeFolder, file.getName()))) {
+            logger.error("Failed to move {} to upgrade folder", file);
+          }
+        }
+
+        Collections.addAll(upgradeFiles,
+            fsFactory.listFilesBySuffix(upgradeFolder.getAbsolutePath(), TSFILE_SUFFIX));
+      }
+      // if already move old files to upgradeFolder 
+      else if (upgradeFolder.exists()) {
+        Collections.addAll(upgradeFiles,
+            fsFactory.listFilesBySuffix(upgradeFolder.getAbsolutePath(), TSFILE_SUFFIX));
+      }
+
       File[] subFiles = fileFolder.listFiles();
       if (subFiles != null) {
         for (File partitionFolder : subFiles) {
-          // some TsFileResource may be being persisted when the system crashed, try recovering such
-          // resources
-          continueFailedRenames(partitionFolder, TEMP_SUFFIX);
-
-          // some TsFiles were going to be replaced by the merged files when the system crashed and
-          // the process was interrupted before the merged files could be named
-          continueFailedRenames(partitionFolder, MERGE_SUFFIX);
-
-        if (!partitionFolder.isDirectory()) {
-          logger.warn("{} is not a directory.", partitionFolder.getAbsolutePath());
-          continue;
-        }
-
-        Collections.addAll(tsFiles,
-            fsFactory.listFilesBySuffix(partitionFolder.getAbsolutePath(), TSFILE_SUFFIX));
+          if (!partitionFolder.isDirectory()) {
+            logger.warn("{} is not a directory.", partitionFolder.getAbsolutePath());
+          } else if (!partitionFolder.getName().equals(IoTDBConstant.UPGRADE_FOLDER_NAME)) {
+            // some TsFileResource may be being persisted when the system crashed, try recovering such
+            // resources
+            continueFailedRenames(partitionFolder, TEMP_SUFFIX);
+
+            // some TsFiles were going to be replaced by the merged files when the system crashed and
+            // the process was interrupted before the merged files could be named
+            continueFailedRenames(partitionFolder, MERGE_SUFFIX);
+
+            Collections.addAll(tsFiles,
+                fsFactory.listFilesBySuffix(partitionFolder.getAbsolutePath(), TSFILE_SUFFIX));
+          }
         }
       }
 
     }
     tsFiles.sort(this::compareFileName);
     List<TsFileResource> ret = new ArrayList<>();
     tsFiles.forEach(f -> ret.add(new TsFileResource(f)));
-    return ret;
+    upgradeFiles.sort(this::compareFileName);
+    List<TsFileResource> upgradeRet = new ArrayList<>();
+    for (File f : upgradeFiles) {
+      TsFileResource fileResource = new TsFileResource(f);
+      fileResource.setClosed(true);
+      // make sure the flush command is called before IoTDB is down.
+      fileResource.deserialize();
+      upgradeRet.add(fileResource);
+    }
+    return new Pair<>(ret, upgradeRet);

Review comment:
       This is my understanding:
   Files that are already in the partitioned folders (like data/sg1/0/1-1-1.tsfile) will not be upgraded, but files that are not in the partitioned folders  (like data/sg1/1-1-1.tsfile) will be moved to an upgrading folder (like data/sg1/upgrade/1-1-1.tsfile), upgraded and reloaded into IoTDB.
   
   But what if the files in the partitioned folders also need upgrading?
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org