You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/08/07 17:00:32 UTC
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #1286: HDDS-4040. OFS should use deleteObjects when deleting directory
xiaoyuyao commented on a change in pull request #1286:
URL: https://github.com/apache/hadoop-ozone/pull/1286#discussion_r467156037
##########
File path: hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicRootedOzoneFileSystem.java
##########
@@ -255,9 +258,11 @@ public FSDataOutputStream append(Path f, int bufferSize,
}
@Override
- boolean processKeyPath(String keyPath) throws IOException {
- String newPath = dstPath.concat(keyPath.substring(srcPath.length()));
- adapterImpl.rename(this.bucket, keyPath, newPath);
+ boolean processKeyPath(List<String> keyPathList) throws IOException {
+ for (String keyPath : keyPathList) {
+ String newPath = dstPath.concat(keyPath.substring(srcPath.length()));
+ adapterImpl.rename(this.bucket, keyPath, newPath);
Review comment:
Do we have a follow up JIRA to fix rename once the batch rename is ready?
##########
File path: hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicRootedOzoneClientAdapterImpl.java
##########
@@ -457,38 +458,79 @@ public boolean deleteObject(String path) {
}
}
+ /**
+ * Helper function to check if the list of key paths are in the same volume
+ * and same bucket.
+ */
+ private boolean areInSameBucket(List<String> keyNameList) {
+ if (keyNameList.size() == 0) {
+ return true;
+ }
+ String firstKeyPath = keyNameList.get(0);
+ final String volAndBucket = new OFSPath(firstKeyPath).getNonKeyPath();
+ // If any key path's volume and bucket from the second element and on
+ // in the list doesn't match the first element's, hasDifferentVolAndBucket
+ // would be true
+ boolean hasDifferentVolAndBucket = keyNameList.stream().skip(1)
+ .anyMatch(p -> !(new OFSPath(p).getNonKeyPath().equals(volAndBucket)));
+ return !hasDifferentVolAndBucket;
+ }
+
/**
* Helper method to delete an object specified by key name in bucket.
*
- * @param pathList key name list to be deleted
+ * Only supports deleting keys in the same bucket in one call.
+ *
+ * Each item in the given list should be the String of an OFS path:
+ * e.g. ofs://om/vol1/buck1/k1
+ *
+ * @param keyNameList key name list to be deleted
* @return true if the key is deleted, false otherwise
*/
@Override
- public boolean deleteObjects(List<String> pathList) {
- // TODO: we will support deleteObjects in ofs.
- LOG.error("ofs currently does not support deleteObjects");
- return false;
+ public boolean deleteObjects(List<String> keyNameList) {
+ LOG.trace("issuing delete for keys: {}", keyNameList);
Review comment:
LOG.trace a keyNameList address does not provide much useful info. I would suggest either remove or expand the list.
##########
File path: hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicRootedOzoneClientAdapterImpl.java
##########
@@ -457,38 +458,79 @@ public boolean deleteObject(String path) {
}
}
+ /**
+ * Helper function to check if the list of key paths are in the same volume
+ * and same bucket.
+ */
+ private boolean areInSameBucket(List<String> keyNameList) {
+ if (keyNameList.size() == 0) {
+ return true;
+ }
+ String firstKeyPath = keyNameList.get(0);
+ final String volAndBucket = new OFSPath(firstKeyPath).getNonKeyPath();
+ // If any key path's volume and bucket from the second element and on
+ // in the list doesn't match the first element's, hasDifferentVolAndBucket
+ // would be true
+ boolean hasDifferentVolAndBucket = keyNameList.stream().skip(1)
+ .anyMatch(p -> !(new OFSPath(p).getNonKeyPath().equals(volAndBucket)));
+ return !hasDifferentVolAndBucket;
+ }
+
/**
* Helper method to delete an object specified by key name in bucket.
*
- * @param pathList key name list to be deleted
+ * Only supports deleting keys in the same bucket in one call.
+ *
+ * Each item in the given list should be the String of an OFS path:
+ * e.g. ofs://om/vol1/buck1/k1
+ *
+ * @param keyNameList key name list to be deleted
* @return true if the key is deleted, false otherwise
*/
@Override
- public boolean deleteObjects(List<String> pathList) {
- // TODO: we will support deleteObjects in ofs.
- LOG.error("ofs currently does not support deleteObjects");
- return false;
+ public boolean deleteObjects(List<String> keyNameList) {
+ LOG.trace("issuing delete for keys: {}", keyNameList);
+ if (keyNameList.size() == 0) {
+ return false;
+ }
+ // Sanity check. Support only deleting a list of keys in the same bucket
+ if (!areInSameBucket(keyNameList)) {
+ LOG.error("Deleting keys from different buckets in a single batch "
+ + "is not supported.");
+ return false;
+ }
+ try {
+ OFSPath firstKeyPath = new OFSPath(keyNameList.get(0));
+ OzoneBucket bucket = getBucket(firstKeyPath, false);
+ return deleteObjects(bucket, keyNameList);
+ } catch (IOException ioe) {
+ LOG.error("delete key failed: {}", ioe.getMessage());
+ return false;
+ }
}
/**
* Package-private helper function to reduce calls to getBucket().
+ *
+ * This will be faster than the public variant of the method since this
+ * doesn't verify the same-bucket condition.
+ *
* @param bucket Bucket to operate in.
- * @param path Path to delete.
- * @return true if operation succeeded, false upon IOException.
+ * @param keyNameList key name list to be deleted.
+ * @return true if operation succeeded, false on IOException.
*/
- boolean deleteObject(OzoneBucket bucket, String path) {
- LOG.trace("issuing delete for path to key: {}", path);
- incrementCounter(Statistic.OBJECTS_DELETED);
- OFSPath ofsPath = new OFSPath(path);
- String keyName = ofsPath.getKeyName();
- if (keyName.length() == 0) {
- return false;
- }
+ boolean deleteObjects(OzoneBucket bucket, List<String> keyNameList) {
+ LOG.trace("issuing delete in volume: {}, bucket: {} for keys: {}",
+ bucket.getVolumeName(), bucket.getName(), keyNameList);
+ List<String> keyList = keyNameList.stream()
+ .map(p -> new OFSPath(p).getKeyName())
+ .collect(Collectors.toList());
try {
- bucket.deleteKey(keyName);
+ incrementCounter(Statistic.OBJECTS_DELETED);
Review comment:
should incrementCount for deletes by the number of keys in the list instead of 1?
Also, should we move the counter update if deleteKeys succeed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org