You are viewing a plain text version of this content. The canonical link for it is here.

Posted to notifications@asterixdb.apache.org by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu> on 2021/09/14 01:03:44 UTC

Change in asterixdb[master]: [NO ISSUE][REP] Ensure files delete order from replicas

From Murtadha Hubail <mh...@apache.org>:

Murtadha Hubail has uploaded this change for review. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/13224 )


Change subject: [NO ISSUE][REP] Ensure files delete order from replicas
......................................................................

[NO ISSUE][REP] Ensure files delete order from replicas

- user model changes: no
- storage format changes: no
- interface changes: no

Details:

- When deleting files from a replica, delete the resource
  files last to ensure no orphaned component files if
  the replica fails after deleting the resource file.
- When replicating a resource file, ensure the index
  directory is empty.

Change-Id: I7dbb9960cb644c063de49dd0920a1709964de2f8
---
M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/task/LocalStorageCleanupTask.java
M asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/messaging/ReplicateFileTask.java
M asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/sync/ReplicaFilesSynchronizer.java
3 files changed, 10 insertions(+), 1 deletion(-)



  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/24/13224/1

diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/task/LocalStorageCleanupTask.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/task/LocalStorageCleanupTask.java
index a926c66..dd7a663 100644
--- a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/task/LocalStorageCleanupTask.java
+++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/task/LocalStorageCleanupTask.java
@@ -46,7 +46,6 @@
         deleteInvalidMetadataIndexes(localResourceRepository);
         final Set<Integer> nodePartitions = appContext.getReplicaManager().getPartitions();
         localResourceRepository.deleteCorruptedResources();
-        //TODO optimize this to cleanup all active partitions at once
         for (Integer partition : nodePartitions) {
             localResourceRepository.cleanup(partition);
         }
diff --git a/asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/messaging/ReplicateFileTask.java b/asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/messaging/ReplicateFileTask.java
index 7f26b96..f645850 100644
--- a/asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/messaging/ReplicateFileTask.java
+++ b/asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/messaging/ReplicateFileTask.java
@@ -39,6 +39,7 @@
 import org.apache.asterix.replication.api.IReplicaTask;
 import org.apache.asterix.replication.api.IReplicationWorker;
 import org.apache.asterix.replication.management.NetworkingUtil;
+import org.apache.commons.io.FileUtils;
 import org.apache.hyracks.api.exceptions.HyracksDataException;
 import org.apache.hyracks.api.io.FileReference;
 import org.apache.hyracks.api.io.IIOManager;
@@ -72,6 +73,10 @@
             // resolve path
             final FileReference localPath = ioManager.resolve(file);
             final Path resourceDir = Files.createDirectories(localPath.getFile().getParentFile().toPath());
+            if (indexMetadata) {
+                // ensure clean index directory
+                FileUtils.deleteQuietly(resourceDir.toFile());
+            }
             // create mask
             final Path maskPath = Paths.get(resourceDir.toString(),
                     StorageConstants.MASK_FILE_PREFIX + localPath.getFile().getName());
diff --git a/asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/sync/ReplicaFilesSynchronizer.java b/asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/sync/ReplicaFilesSynchronizer.java
index faf3f54..bb8dc7b 100644
--- a/asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/sync/ReplicaFilesSynchronizer.java
+++ b/asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/sync/ReplicaFilesSynchronizer.java
@@ -21,6 +21,7 @@
 import java.io.IOException;
 import java.nio.ByteBuffer;
 import java.util.ArrayList;
+import java.util.Collections;
 import java.util.HashMap;
 import java.util.HashSet;
 import java.util.List;
@@ -115,6 +116,10 @@
 
     private void deleteInvalidFiles(List<String> files) {
         final FileSynchronizer sync = new FileSynchronizer(appCtx, replica);
+        // sort files to ensure index metadata files starting with "." are deleted last
+        files.sort(String::compareTo);
+        Collections.reverse(files);
+        LOGGER.info("deleting {}", files);
         files.forEach(sync::delete);
     }
 

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/13224
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: I7dbb9960cb644c063de49dd0920a1709964de2f8
Gerrit-Change-Number: 13224
Gerrit-PatchSet: 1
Gerrit-Owner: Murtadha Hubail <mh...@apache.org>
Gerrit-MessageType: newchange