You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@asterixdb.apache.org by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu> on 2021/09/08 01:05:58 UTC
Change in asterixdb[master]: [NO ISSUE][STO] Add utility to clean up partition directory
From Ali Alsuliman <al...@gmail.com>:
Ali Alsuliman has uploaded this change for review. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/13105 )
Change subject: [NO ISSUE][STO] Add utility to clean up partition directory
......................................................................
[NO ISSUE][STO] Add utility to clean up partition directory
- user model changes: no
- storage format changes: no
- interface changes: no
Details:
Change-Id: Ic78864d3f8f7c3bda5e4711db00d3be0ae8f7aea
---
M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StoragePathUtil.java
M asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/resource/PersistentLocalResourceRepository.java
2 files changed, 29 insertions(+), 2 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/05/13105/1
diff --git a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StoragePathUtil.java b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StoragePathUtil.java
index 32a226e..f84472e 100644
--- a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StoragePathUtil.java
+++ b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StoragePathUtil.java
@@ -90,9 +90,11 @@
}
public static int getPartitionNumFromRelativePath(String relativePath) {
- int startIdx = relativePath.indexOf(StorageConstants.PARTITION_DIR_PREFIX)
+ int startIdx = relativePath.lastIndexOf(StorageConstants.PARTITION_DIR_PREFIX)
+ StorageConstants.PARTITION_DIR_PREFIX.length();
- String partition = relativePath.substring(startIdx, relativePath.indexOf(File.separatorChar, startIdx));
+ int partitionEndIdx = relativePath.indexOf(File.separatorChar, startIdx);
+ int idxEnd = partitionEndIdx != -1 ? partitionEndIdx : relativePath.length();
+ String partition = relativePath.substring(startIdx, idxEnd);
return Integer.parseInt(partition);
}
diff --git a/asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/resource/PersistentLocalResourceRepository.java b/asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/resource/PersistentLocalResourceRepository.java
index 3b192aa..4d15385 100644
--- a/asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/resource/PersistentLocalResourceRepository.java
+++ b/asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/resource/PersistentLocalResourceRepository.java
@@ -33,6 +33,7 @@
import java.nio.file.Paths;
import java.text.ParseException;
import java.util.ArrayList;
+import java.util.Arrays;
import java.util.Collection;
import java.util.HashMap;
import java.util.HashSet;
@@ -646,4 +647,28 @@
public Path[] getStorageRoots() {
return storageRoots;
}
+
+ public void keepPartitions(Set<Integer> keepPartitions) {
+ List<File> onDiskPartitions = getOnDiskPartitions();
+ for (File onDiskPartition : onDiskPartitions) {
+ int partitionNum = StoragePathUtil.getPartitionNumFromRelativePath(onDiskPartition.getAbsolutePath());
+ if (!keepPartitions.contains(partitionNum)) {
+ LOGGER.warn("deleting partition {} since it is not on partitions to keep {}", partitionNum,
+ keepPartitions);
+ FileUtils.deleteQuietly(onDiskPartition);
+ }
+ }
+ }
+
+ public List<File> getOnDiskPartitions() {
+ List<File> onDiskPartitions = new ArrayList<>();
+ for (Path root : storageRoots) {
+ File[] partitions = root.toFile().listFiles(
+ (dir, name) -> dir.isDirectory() && name.startsWith(StorageConstants.PARTITION_DIR_PREFIX));
+ if (partitions != null) {
+ onDiskPartitions.addAll(Arrays.asList(partitions));
+ }
+ }
+ return onDiskPartitions;
+ }
}
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/13105
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Ic78864d3f8f7c3bda5e4711db00d3be0ae8f7aea
Gerrit-Change-Number: 13105
Gerrit-PatchSet: 1
Gerrit-Owner: Ali Alsuliman <al...@gmail.com>
Gerrit-MessageType: newchange
Change in asterixdb[master]: [NO ISSUE][STO] Add utility to clean up partition directory
Posted by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu>.
Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/13105 )
Change subject: [NO ISSUE][STO] Add utility to clean up partition directory
......................................................................
Patch Set 1:
Analytics Compatibility Compilation Successful
https://cbjenkins.page.link/xaPboyEwVYYwMMA17 : SUCCESS
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/13105
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Ic78864d3f8f7c3bda5e4711db00d3be0ae8f7aea
Gerrit-Change-Number: 13105
Gerrit-PatchSet: 1
Gerrit-Owner: Ali Alsuliman <al...@gmail.com>
Gerrit-Reviewer: Michael Blow <mb...@apache.org>
Gerrit-Reviewer: Murtadha Hubail <mh...@apache.org>
Gerrit-CC: Anon. E. Moose #1000171
Gerrit-CC: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Comment-Date: Wed, 08 Sep 2021 01:16:49 +0000
Gerrit-HasComments: No
Gerrit-Has-Labels: No
Gerrit-MessageType: comment
Change in asterixdb[master]: [NO ISSUE][STO] Add utility to clean up partition directory
Posted by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu>.
From Ali Alsuliman <al...@gmail.com>:
Ali Alsuliman has uploaded this change for review. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/13105 )
Change subject: [NO ISSUE][STO] Add utility to clean up partition directory
......................................................................
[NO ISSUE][STO] Add utility to clean up partition directory
- user model changes: no
- storage format changes: no
- interface changes: no
Details:
Change-Id: Ic78864d3f8f7c3bda5e4711db00d3be0ae8f7aea
---
M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StoragePathUtil.java
M asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/resource/PersistentLocalResourceRepository.java
2 files changed, 29 insertions(+), 2 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/05/13105/1
diff --git a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StoragePathUtil.java b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StoragePathUtil.java
index 32a226e..f84472e 100644
--- a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StoragePathUtil.java
+++ b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StoragePathUtil.java
@@ -90,9 +90,11 @@
}
public static int getPartitionNumFromRelativePath(String relativePath) {
- int startIdx = relativePath.indexOf(StorageConstants.PARTITION_DIR_PREFIX)
+ int startIdx = relativePath.lastIndexOf(StorageConstants.PARTITION_DIR_PREFIX)
+ StorageConstants.PARTITION_DIR_PREFIX.length();
- String partition = relativePath.substring(startIdx, relativePath.indexOf(File.separatorChar, startIdx));
+ int partitionEndIdx = relativePath.indexOf(File.separatorChar, startIdx);
+ int idxEnd = partitionEndIdx != -1 ? partitionEndIdx : relativePath.length();
+ String partition = relativePath.substring(startIdx, idxEnd);
return Integer.parseInt(partition);
}
diff --git a/asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/resource/PersistentLocalResourceRepository.java b/asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/resource/PersistentLocalResourceRepository.java
index 3b192aa..4d15385 100644
--- a/asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/resource/PersistentLocalResourceRepository.java
+++ b/asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/resource/PersistentLocalResourceRepository.java
@@ -33,6 +33,7 @@
import java.nio.file.Paths;
import java.text.ParseException;
import java.util.ArrayList;
+import java.util.Arrays;
import java.util.Collection;
import java.util.HashMap;
import java.util.HashSet;
@@ -646,4 +647,28 @@
public Path[] getStorageRoots() {
return storageRoots;
}
+
+ public void keepPartitions(Set<Integer> keepPartitions) {
+ List<File> onDiskPartitions = getOnDiskPartitions();
+ for (File onDiskPartition : onDiskPartitions) {
+ int partitionNum = StoragePathUtil.getPartitionNumFromRelativePath(onDiskPartition.getAbsolutePath());
+ if (!keepPartitions.contains(partitionNum)) {
+ LOGGER.warn("deleting partition {} since it is not on partitions to keep {}", partitionNum,
+ keepPartitions);
+ FileUtils.deleteQuietly(onDiskPartition);
+ }
+ }
+ }
+
+ public List<File> getOnDiskPartitions() {
+ List<File> onDiskPartitions = new ArrayList<>();
+ for (Path root : storageRoots) {
+ File[] partitions = root.toFile().listFiles(
+ (dir, name) -> dir.isDirectory() && name.startsWith(StorageConstants.PARTITION_DIR_PREFIX));
+ if (partitions != null) {
+ onDiskPartitions.addAll(Arrays.asList(partitions));
+ }
+ }
+ return onDiskPartitions;
+ }
}
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/13105
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Ic78864d3f8f7c3bda5e4711db00d3be0ae8f7aea
Gerrit-Change-Number: 13105
Gerrit-PatchSet: 1
Gerrit-Owner: Ali Alsuliman <al...@gmail.com>
Gerrit-MessageType: newchange
Change in asterixdb[master]: [NO ISSUE][STO] Add utility to clean up partition directory
Posted by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu>.
Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/13105 )
Change subject: [NO ISSUE][STO] Add utility to clean up partition directory
......................................................................
Patch Set 1: Contrib-2
Analytics Compatibility Tests Failed
https://cbjenkins.page.link/KQjZZ2HJmkp7yYKp8 : UNSTABLE
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/13105
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Ic78864d3f8f7c3bda5e4711db00d3be0ae8f7aea
Gerrit-Change-Number: 13105
Gerrit-PatchSet: 1
Gerrit-Owner: Ali Alsuliman <al...@gmail.com>
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Michael Blow <mb...@apache.org>
Gerrit-Reviewer: Murtadha Hubail <mh...@apache.org>
Gerrit-Comment-Date: Wed, 08 Sep 2021 04:10:04 +0000
Gerrit-HasComments: No
Gerrit-Has-Labels: Yes
Gerrit-MessageType: comment
Change in asterixdb[master]: [NO ISSUE][STO] Add utility to clean up partition directory
Posted by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu>.
From Jenkins <je...@fulliautomatix.ics.uci.edu>:
Jenkins has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/13105 )
Change subject: [NO ISSUE][STO] Add utility to clean up partition directory
......................................................................
Patch Set 1: Integration-Tests+1
Integration Tests Successful
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/12449/ : SUCCESS
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/13105
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Ic78864d3f8f7c3bda5e4711db00d3be0ae8f7aea
Gerrit-Change-Number: 13105
Gerrit-PatchSet: 1
Gerrit-Owner: Ali Alsuliman <al...@gmail.com>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Michael Blow <mb...@apache.org>
Gerrit-Reviewer: Murtadha Hubail <mh...@apache.org>
Gerrit-CC: Anon. E. Moose #1000171
Gerrit-Comment-Date: Wed, 08 Sep 2021 02:57:11 +0000
Gerrit-HasComments: No
Gerrit-Has-Labels: Yes
Gerrit-MessageType: comment