You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "ashishkumar50 (via GitHub)" <gi...@apache.org> on 2023/05/08 08:40:03 UTC
[GitHub] [ozone] ashishkumar50 opened a new pull request, #4675: HDDS-8547. Support Trash for FSO bucket using ozone sh command.
ashishkumar50 opened a new pull request, #4675:
URL: https://github.com/apache/ozone/pull/4675
## What changes were proposed in this pull request?
When trash is enabled and user runs ozone sh key delete, the key should be moved inside trash instead of permanent delete immediately.
Currently using fs command when trash is enabled and user deletes the file for FSO bucket it goes inside Trash.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-8547
## How was this patch tested?
Added new integration and robot test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ashishkumar50 commented on pull request #4675: HDDS-8547. Support Trash for FSO bucket using ozone sh command.
Posted by "ashishkumar50 (via GitHub)" <gi...@apache.org>.
ashishkumar50 commented on PR #4675:
URL: https://github.com/apache/ozone/pull/4675#issuecomment-1537988305
@sadanand48 , Can you please help to review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ashishkumar50 commented on pull request #4675: HDDS-8547. Support Trash for FSO bucket using ozone sh command.
Posted by "ashishkumar50 (via GitHub)" <gi...@apache.org>.
ashishkumar50 commented on PR #4675:
URL: https://github.com/apache/ozone/pull/4675#issuecomment-1539968644
> Thanks @ashishkumar50 for the patch. Left few comments inline.
Hi @sadanand48, Thanks for the review, Handled the comments and replied. Can you pls look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sadanand48 commented on a diff in pull request #4675: HDDS-8547. Support Trash for FSO bucket using ozone sh command.
Posted by "sadanand48 (via GitHub)" <gi...@apache.org>.
sadanand48 commented on code in PR #4675:
URL: https://github.com/apache/ozone/pull/4675#discussion_r1187319544
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/DeleteKeyHandler.java:
##########
@@ -45,6 +64,81 @@ protected void execute(OzoneClient client, OzoneAddress address)
OzoneVolume vol = client.getObjectStore().getVolume(volumeName);
OzoneBucket bucket = vol.getBucket(bucketName);
- bucket.deleteKey(keyName);
+
+ float hadoopTrashInterval = getConf().getFloat(
+ FS_TRASH_INTERVAL_KEY, FS_TRASH_INTERVAL_DEFAULT);
+
+ long trashInterval =
+ (long) (getConf().getFloat(
+ OMConfigKeys.OZONE_FS_TRASH_INTERVAL_KEY,
+ hadoopTrashInterval) * 10000);
+
+ // If Bucket layout is FSO and Trash is enabled
+ // In this case during delete operation move key to trash
+ if (bucket.getBucketLayout().isFileSystemOptimized() && trashInterval > 0
+ && !skipTrash && !keyName.contains(".Trash")) {
+
+ keyName = OzoneFSUtils.removeTrailingSlashIfNeeded(keyName);
+ try {
+ // Check if key exists in Ozone
+ OzoneKeyDetails key = bucket.getKey(keyName);
+ if (key == null) {
+ out().printf("Key not found %s", keyName);
+ return;
+ }
+ // Check whether directory is empty or not
+ Iterator<? extends OzoneKey> ozoneKeyIterator =
+ bucket.listKeys(keyName, keyName);
Review Comment:
bucket.listKeys() by default will return 1000 entries in each RPC. This might be a little unnecessary as we only want to check if the directory is empty or not here, Even if there is a single key with given prefix, it should conclude that directory is non-empty.
1. I couldn't find an API for listKeys with maxEntries param but instead you could use
``` java
public List<OzoneFileStatus> listStatus(String keyName, boolean recursive,
String startKey, long numEntries) throws IOException
```
and pass numEntries as 1.
3. To check if it is directory you could use ` bucket.getFileStatus(key).isDirectory();`
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/DeleteKeyHandler.java:
##########
@@ -45,6 +64,81 @@ protected void execute(OzoneClient client, OzoneAddress address)
OzoneVolume vol = client.getObjectStore().getVolume(volumeName);
OzoneBucket bucket = vol.getBucket(bucketName);
- bucket.deleteKey(keyName);
+
+ float hadoopTrashInterval = getConf().getFloat(
+ FS_TRASH_INTERVAL_KEY, FS_TRASH_INTERVAL_DEFAULT);
+
+ long trashInterval =
+ (long) (getConf().getFloat(
+ OMConfigKeys.OZONE_FS_TRASH_INTERVAL_KEY,
+ hadoopTrashInterval) * 10000);
+
+ // If Bucket layout is FSO and Trash is enabled
+ // In this case during delete operation move key to trash
+ if (bucket.getBucketLayout().isFileSystemOptimized() && trashInterval > 0
+ && !skipTrash && !keyName.contains(".Trash")) {
+
+ keyName = OzoneFSUtils.removeTrailingSlashIfNeeded(keyName);
+ try {
+ // Check if key exists in Ozone
+ OzoneKeyDetails key = bucket.getKey(keyName);
+ if (key == null) {
+ out().printf("Key not found %s", keyName);
+ return;
+ }
+ // Check whether directory is empty or not
+ Iterator<? extends OzoneKey> ozoneKeyIterator =
+ bucket.listKeys(keyName, keyName);
Review Comment:
bucket.listKeys() by default will return 1000 entries in each RPC. This might be a little unnecessary as we only want to check if the directory is empty or not here, Even if there is a single key with given prefix, it should conclude that directory is non-empty.
1. I couldn't find an API for listKeys with maxEntries param but instead you could use
``` java
public List<OzoneFileStatus> listStatus(String keyName, boolean recursive,
String startKey, long numEntries) throws IOException
```
and pass numEntries as 1.
2. To check if it is directory you could use ` bucket.getFileStatus(key).isDirectory();`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ashishkumar50 commented on a diff in pull request #4675: HDDS-8547. Support Trash for FSO bucket using ozone sh command.
Posted by "ashishkumar50 (via GitHub)" <gi...@apache.org>.
ashishkumar50 commented on code in PR #4675:
URL: https://github.com/apache/ozone/pull/4675#discussion_r1188486960
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/DeleteKeyHandler.java:
##########
@@ -45,6 +64,81 @@ protected void execute(OzoneClient client, OzoneAddress address)
OzoneVolume vol = client.getObjectStore().getVolume(volumeName);
OzoneBucket bucket = vol.getBucket(bucketName);
- bucket.deleteKey(keyName);
+
+ float hadoopTrashInterval = getConf().getFloat(
+ FS_TRASH_INTERVAL_KEY, FS_TRASH_INTERVAL_DEFAULT);
+
+ long trashInterval =
+ (long) (getConf().getFloat(
+ OMConfigKeys.OZONE_FS_TRASH_INTERVAL_KEY,
+ hadoopTrashInterval) * 10000);
+
+ // If Bucket layout is FSO and Trash is enabled
+ // In this case during delete operation move key to trash
+ if (bucket.getBucketLayout().isFileSystemOptimized() && trashInterval > 0
+ && !skipTrash && !keyName.contains(".Trash")) {
+
+ keyName = OzoneFSUtils.removeTrailingSlashIfNeeded(keyName);
+ try {
+ // Check if key exists in Ozone
+ OzoneKeyDetails key = bucket.getKey(keyName);
+ if (key == null) {
+ out().printf("Key not found %s", keyName);
+ return;
+ }
+ // Check whether directory is empty or not
+ Iterator<? extends OzoneKey> ozoneKeyIterator =
+ bucket.listKeys(keyName, keyName);
+ int count = 0;
+ while (ozoneKeyIterator.hasNext()) {
+ ozoneKeyIterator.next();
+ if (++count > 1) {
+ // Assume FSO Tree: /a/b1/c1/k1.txt
+ // And we are trying to delete key /a/b1/c1
+ // In this case count is 2 which is greater than 1
+ // /a/b1/c1/ and /a/b1/c1/k1.txt
+ out().printf("Directory is not empty");
+ return;
+ }
+ }
+ } catch (Exception e) {
+ out().printf("Key not found %s", keyName);
+ return;
+ }
+
+ final String username =
+ UserGroupInformation.getCurrentUser().getShortUserName();
+ Path trashRoot = new Path(OZONE_URI_DELIMITER, TRASH_PREFIX);
+ Path userTrash = new Path(trashRoot, username);
+ Path userTrashCurrent = new Path(userTrash, CURRENT);
+
+ String trashDirectory = (keyName.contains("/")
+ ? new Path(userTrashCurrent, keyName.substring(0,
+ keyName.lastIndexOf("/")))
+ : userTrashCurrent).toUri().getPath();
+
+ String toKeyName = new Path(userTrashCurrent, keyName).toUri().getPath();
+ OzoneKeyDetails toKeyDetails = null;
+ try {
+ // check whether key already exist in trash
+ toKeyDetails = bucket.getKey(toKeyName);
+ } catch (IOException e) {
+ // Key doesn't exist inside trash.
+ }
+
+ if (toKeyDetails != null) {
+ // if key(directory) already exist in trash, just delete the key
+ bucket.deleteKey(keyName);
+ return;
+ }
+ // Create directory inside trash
+ bucket.createDirectory(trashDirectory);
Review Comment:
I verified and I can see it is not creating directory in rename operation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sadanand48 commented on a diff in pull request #4675: HDDS-8547. Support Trash for FSO bucket using ozone sh command.
Posted by "sadanand48 (via GitHub)" <gi...@apache.org>.
sadanand48 commented on code in PR #4675:
URL: https://github.com/apache/ozone/pull/4675#discussion_r1188895412
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/DeleteKeyHandler.java:
##########
@@ -45,6 +64,81 @@ protected void execute(OzoneClient client, OzoneAddress address)
OzoneVolume vol = client.getObjectStore().getVolume(volumeName);
OzoneBucket bucket = vol.getBucket(bucketName);
- bucket.deleteKey(keyName);
+
+ float hadoopTrashInterval = getConf().getFloat(
+ FS_TRASH_INTERVAL_KEY, FS_TRASH_INTERVAL_DEFAULT);
+
+ long trashInterval =
+ (long) (getConf().getFloat(
+ OMConfigKeys.OZONE_FS_TRASH_INTERVAL_KEY,
+ hadoopTrashInterval) * 10000);
+
+ // If Bucket layout is FSO and Trash is enabled
+ // In this case during delete operation move key to trash
+ if (bucket.getBucketLayout().isFileSystemOptimized() && trashInterval > 0
+ && !skipTrash && !keyName.contains(".Trash")) {
+
+ keyName = OzoneFSUtils.removeTrailingSlashIfNeeded(keyName);
+ try {
+ // Check if key exists in Ozone
+ OzoneKeyDetails key = bucket.getKey(keyName);
+ if (key == null) {
+ out().printf("Key not found %s", keyName);
+ return;
+ }
+ // Check whether directory is empty or not
+ Iterator<? extends OzoneKey> ozoneKeyIterator =
+ bucket.listKeys(keyName, keyName);
+ int count = 0;
+ while (ozoneKeyIterator.hasNext()) {
+ ozoneKeyIterator.next();
+ if (++count > 1) {
+ // Assume FSO Tree: /a/b1/c1/k1.txt
+ // And we are trying to delete key /a/b1/c1
+ // In this case count is 2 which is greater than 1
+ // /a/b1/c1/ and /a/b1/c1/k1.txt
+ out().printf("Directory is not empty");
+ return;
+ }
+ }
+ } catch (Exception e) {
+ out().printf("Key not found %s", keyName);
+ return;
+ }
+
+ final String username =
+ UserGroupInformation.getCurrentUser().getShortUserName();
+ Path trashRoot = new Path(OZONE_URI_DELIMITER, TRASH_PREFIX);
+ Path userTrash = new Path(trashRoot, username);
+ Path userTrashCurrent = new Path(userTrash, CURRENT);
+
+ String trashDirectory = (keyName.contains("/")
+ ? new Path(userTrashCurrent, keyName.substring(0,
+ keyName.lastIndexOf("/")))
+ : userTrashCurrent).toUri().getPath();
+
+ String toKeyName = new Path(userTrashCurrent, keyName).toUri().getPath();
+ OzoneKeyDetails toKeyDetails = null;
+ try {
+ // check whether key already exist in trash
+ toKeyDetails = bucket.getKey(toKeyName);
+ } catch (IOException e) {
+ // Key doesn't exist inside trash.
+ }
+
+ if (toKeyDetails != null) {
+ // if key(directory) already exist in trash, just delete the key
+ bucket.deleteKey(keyName);
+ return;
+ }
+ // Create directory inside trash
+ bucket.createDirectory(trashDirectory);
Review Comment:
Ok, what happens if the trashDirectory already exists here. doesn't it throw directory already exists or some error. We could check for existence here.
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/DeleteKeyHandler.java:
##########
@@ -35,16 +47,110 @@
description = "deletes an existing key")
public class DeleteKeyHandler extends KeyHandler {
+ @CommandLine.Option(names = "--skipTrash",
+ description = "Specify whether to skip Trash ")
+ private boolean skipTrash = false;
+
+ private static final Path CURRENT = new Path("Current");
+
@Override
protected void execute(OzoneClient client, OzoneAddress address)
throws IOException, OzoneClientException {
String volumeName = address.getVolumeName();
String bucketName = address.getBucketName();
+ OzoneVolume vol = client.getObjectStore().getVolume(volumeName);
+ OzoneBucket bucket = vol.getBucket(bucketName);
+ String keyName = address.getKeyName();
+
+ if (bucket.getBucketLayout().isFileSystemOptimized()) {
+ // Handle FSO delete key which supports trash also
+ deleteFSOKey(client, address);
+ } else {
+ bucket.deleteKey(keyName);
+ }
+ }
+
+ private void deleteFSOKey(OzoneClient client, OzoneAddress address)
Review Comment:
minor nit : deleteFSOKey(OzoneBucket bucket, String keyName)
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/DeleteKeyHandler.java:
##########
@@ -45,6 +64,81 @@ protected void execute(OzoneClient client, OzoneAddress address)
OzoneVolume vol = client.getObjectStore().getVolume(volumeName);
OzoneBucket bucket = vol.getBucket(bucketName);
- bucket.deleteKey(keyName);
+
+ float hadoopTrashInterval = getConf().getFloat(
+ FS_TRASH_INTERVAL_KEY, FS_TRASH_INTERVAL_DEFAULT);
+
+ long trashInterval =
+ (long) (getConf().getFloat(
+ OMConfigKeys.OZONE_FS_TRASH_INTERVAL_KEY,
+ hadoopTrashInterval) * 10000);
+
+ // If Bucket layout is FSO and Trash is enabled
+ // In this case during delete operation move key to trash
+ if (bucket.getBucketLayout().isFileSystemOptimized() && trashInterval > 0
+ && !skipTrash && !keyName.contains(".Trash")) {
+
+ keyName = OzoneFSUtils.removeTrailingSlashIfNeeded(keyName);
+ try {
+ // Check if key exists in Ozone
+ OzoneKeyDetails key = bucket.getKey(keyName);
+ if (key == null) {
+ out().printf("Key not found %s", keyName);
+ return;
+ }
+ // Check whether directory is empty or not
+ Iterator<? extends OzoneKey> ozoneKeyIterator =
+ bucket.listKeys(keyName, keyName);
+ int count = 0;
+ while (ozoneKeyIterator.hasNext()) {
+ ozoneKeyIterator.next();
+ if (++count > 1) {
+ // Assume FSO Tree: /a/b1/c1/k1.txt
+ // And we are trying to delete key /a/b1/c1
+ // In this case count is 2 which is greater than 1
+ // /a/b1/c1/ and /a/b1/c1/k1.txt
+ out().printf("Directory is not empty");
+ return;
+ }
+ }
+ } catch (Exception e) {
+ out().printf("Key not found %s", keyName);
+ return;
+ }
+
+ final String username =
+ UserGroupInformation.getCurrentUser().getShortUserName();
+ Path trashRoot = new Path(OZONE_URI_DELIMITER, TRASH_PREFIX);
+ Path userTrash = new Path(trashRoot, username);
+ Path userTrashCurrent = new Path(userTrash, CURRENT);
+
+ String trashDirectory = (keyName.contains("/")
+ ? new Path(userTrashCurrent, keyName.substring(0,
+ keyName.lastIndexOf("/")))
+ : userTrashCurrent).toUri().getPath();
+
+ String toKeyName = new Path(userTrashCurrent, keyName).toUri().getPath();
+ OzoneKeyDetails toKeyDetails = null;
+ try {
+ // check whether key already exist in trash
+ toKeyDetails = bucket.getKey(toKeyName);
+ } catch (IOException e) {
+ // Key doesn't exist inside trash.
+ }
+
+ if (toKeyDetails != null) {
+ // if key(directory) already exist in trash, just delete the key
Review Comment:
Although it makes sense, I still see a problem here, Lets say we create 2 versions of same key.
create key1
delete key1 -> move to trash-> trash has key1
create key1 again -> delete again -> this will not move the second key to trash instead delete it.
How filesystem trash handles duplicate keys is by appending timestamp if it is the key with same name.
See [TrashPolicyOzone#moveToTrash](https://github.com/apache/ozone/blob/5e8b806f058a74dc854164118947c703977f94a6/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/TrashPolicyOzone.java#L204).
The above problem you are mentioning occurs probably due to parent dir creation inside trash and would happen only on directories. We can distinguish the logic based on whether object is key/directory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ashishkumar50 commented on a diff in pull request #4675: HDDS-8547. Support Trash for FSO bucket using ozone sh command.
Posted by "ashishkumar50 (via GitHub)" <gi...@apache.org>.
ashishkumar50 commented on code in PR #4675:
URL: https://github.com/apache/ozone/pull/4675#discussion_r1188492599
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/DeleteKeyHandler.java:
##########
@@ -45,6 +64,81 @@ protected void execute(OzoneClient client, OzoneAddress address)
OzoneVolume vol = client.getObjectStore().getVolume(volumeName);
OzoneBucket bucket = vol.getBucket(bucketName);
- bucket.deleteKey(keyName);
+
+ float hadoopTrashInterval = getConf().getFloat(
+ FS_TRASH_INTERVAL_KEY, FS_TRASH_INTERVAL_DEFAULT);
+
+ long trashInterval =
+ (long) (getConf().getFloat(
+ OMConfigKeys.OZONE_FS_TRASH_INTERVAL_KEY,
+ hadoopTrashInterval) * 10000);
+
+ // If Bucket layout is FSO and Trash is enabled
+ // In this case during delete operation move key to trash
+ if (bucket.getBucketLayout().isFileSystemOptimized() && trashInterval > 0
+ && !skipTrash && !keyName.contains(".Trash")) {
+
+ keyName = OzoneFSUtils.removeTrailingSlashIfNeeded(keyName);
+ try {
+ // Check if key exists in Ozone
+ OzoneKeyDetails key = bucket.getKey(keyName);
+ if (key == null) {
+ out().printf("Key not found %s", keyName);
+ return;
+ }
+ // Check whether directory is empty or not
+ Iterator<? extends OzoneKey> ozoneKeyIterator =
+ bucket.listKeys(keyName, keyName);
+ int count = 0;
+ while (ozoneKeyIterator.hasNext()) {
+ ozoneKeyIterator.next();
+ if (++count > 1) {
+ // Assume FSO Tree: /a/b1/c1/k1.txt
+ // And we are trying to delete key /a/b1/c1
+ // In this case count is 2 which is greater than 1
+ // /a/b1/c1/ and /a/b1/c1/k1.txt
+ out().printf("Directory is not empty");
+ return;
+ }
+ }
+ } catch (Exception e) {
+ out().printf("Key not found %s", keyName);
+ return;
+ }
+
+ final String username =
+ UserGroupInformation.getCurrentUser().getShortUserName();
+ Path trashRoot = new Path(OZONE_URI_DELIMITER, TRASH_PREFIX);
+ Path userTrash = new Path(trashRoot, username);
+ Path userTrashCurrent = new Path(userTrash, CURRENT);
+
+ String trashDirectory = (keyName.contains("/")
+ ? new Path(userTrashCurrent, keyName.substring(0,
+ keyName.lastIndexOf("/")))
+ : userTrashCurrent).toUri().getPath();
+
+ String toKeyName = new Path(userTrashCurrent, keyName).toUri().getPath();
+ OzoneKeyDetails toKeyDetails = null;
+ try {
+ // check whether key already exist in trash
+ toKeyDetails = bucket.getKey(toKeyName);
+ } catch (IOException e) {
+ // Key doesn't exist inside trash.
+ }
+
+ if (toKeyDetails != null) {
+ // if key(directory) already exist in trash, just delete the key
Review Comment:
Assume Key: dir1/dir2/key1
And we delete dir1/dir2/key1, then in trash there will be keys for all dir1/, dir1/dir2/, dir1/dir2/key1
Now when we delete dir1/dir2, but in trash already this already exists, so we just remove here dir1/dir2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ashishkumar50 commented on pull request #4675: HDDS-8547. Support Trash for FSO bucket using ozone sh command.
Posted by "ashishkumar50 (via GitHub)" <gi...@apache.org>.
ashishkumar50 commented on PR #4675:
URL: https://github.com/apache/ozone/pull/4675#issuecomment-1543724509
Hi @sadanand48, Handled the comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ashishkumar50 commented on a diff in pull request #4675: HDDS-8547. Support Trash for FSO bucket using ozone sh command.
Posted by "ashishkumar50 (via GitHub)" <gi...@apache.org>.
ashishkumar50 commented on code in PR #4675:
URL: https://github.com/apache/ozone/pull/4675#discussion_r1192280702
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/DeleteKeyHandler.java:
##########
@@ -35,16 +48,117 @@
description = "deletes an existing key")
public class DeleteKeyHandler extends KeyHandler {
+ @CommandLine.Option(names = "--skipTrash",
+ description = "Specify whether to skip Trash ")
+ private boolean skipTrash = false;
+
+ private static final Path CURRENT = new Path("Current");
+
@Override
protected void execute(OzoneClient client, OzoneAddress address)
throws IOException, OzoneClientException {
String volumeName = address.getVolumeName();
String bucketName = address.getBucketName();
- String keyName = address.getKeyName();
-
OzoneVolume vol = client.getObjectStore().getVolume(volumeName);
OzoneBucket bucket = vol.getBucket(bucketName);
- bucket.deleteKey(keyName);
+ String keyName = address.getKeyName();
+
+ if (bucket.getBucketLayout().isFileSystemOptimized()) {
+ // Handle FSO delete key which supports trash also
+ deleteFSOKey(bucket, keyName);
+ } else {
+ bucket.deleteKey(keyName);
+ }
+ }
+
+ private void deleteFSOKey(OzoneBucket bucket, String keyName)
+ throws IOException {
+ float hadoopTrashInterval = getConf().getFloat(
+ FS_TRASH_INTERVAL_KEY, FS_TRASH_INTERVAL_DEFAULT);
+ long trashInterval =
+ (long) (getConf().getFloat(
+ OMConfigKeys.OZONE_FS_TRASH_INTERVAL_KEY,
+ hadoopTrashInterval) * 10000);
+
+ // If Bucket layout is FSO and Trash is enabled
+ // In this case during delete operation move key to trash
+ if (trashInterval > 0 && !skipTrash &&
+ !keyName.contains(TRASH_PREFIX)) {
+ keyName = OzoneFSUtils.removeTrailingSlashIfNeeded(keyName);
+ // Check if key exists in Ozone
+ boolean bKeyExist = isKeyExist(bucket, keyName);
+ if (!bKeyExist) {
Review Comment:
Thanks, Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ashishkumar50 commented on a diff in pull request #4675: HDDS-8547. Support Trash for FSO bucket using ozone sh command.
Posted by "ashishkumar50 (via GitHub)" <gi...@apache.org>.
ashishkumar50 commented on code in PR #4675:
URL: https://github.com/apache/ozone/pull/4675#discussion_r1188487494
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/DeleteKeyHandler.java:
##########
@@ -45,6 +64,81 @@ protected void execute(OzoneClient client, OzoneAddress address)
OzoneVolume vol = client.getObjectStore().getVolume(volumeName);
OzoneBucket bucket = vol.getBucket(bucketName);
- bucket.deleteKey(keyName);
+
+ float hadoopTrashInterval = getConf().getFloat(
+ FS_TRASH_INTERVAL_KEY, FS_TRASH_INTERVAL_DEFAULT);
+
+ long trashInterval =
+ (long) (getConf().getFloat(
+ OMConfigKeys.OZONE_FS_TRASH_INTERVAL_KEY,
+ hadoopTrashInterval) * 10000);
+
+ // If Bucket layout is FSO and Trash is enabled
+ // In this case during delete operation move key to trash
+ if (bucket.getBucketLayout().isFileSystemOptimized() && trashInterval > 0
Review Comment:
Refactored in new method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ashishkumar50 commented on a diff in pull request #4675: HDDS-8547. Support Trash for FSO bucket using ozone sh command.
Posted by "ashishkumar50 (via GitHub)" <gi...@apache.org>.
ashishkumar50 commented on code in PR #4675:
URL: https://github.com/apache/ozone/pull/4675#discussion_r1188487743
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneShellHA.java:
##########
@@ -1124,6 +1129,127 @@ public void testCreateBucketWithECReplicationConfigWithoutReplicationParam() {
}
}
+
+ @Test
+ public void testKeyDeleteOrSkipTrashWhenTrashEnableFSO()
+ throws UnsupportedEncodingException {
+ // Create 100 keys
+ generateKeys("/volumefso1", "/bucket1",
+ BucketLayout.FILE_SYSTEM_OPTIMIZED.toString());
+
+ // Enable trash
+ String trashConfKey = generateSetConfString(
+ OMConfigKeys.OZONE_FS_TRASH_INTERVAL_KEY, "1");
+ String[] args =
+ new String[] {trashConfKey, "key", "delete",
+ "/volumefso1/bucket1/key4"};
+
+ // Delete one key from FSO bucket
+ execute(ozoneShell, args);
+
+ // Get key list in .Trash path
+ String prefixKey = "--prefix=.Trash";
+ args = new String[] {"key", "list", prefixKey, "o3://" +
+ omServiceId + "/volumefso1/bucket1/"};
+ out.reset();
+ execute(ozoneShell, args);
+
+ // One key should be present in .Trash
+ Assert.assertEquals(1, getNumOfKeys());
+
+ args = new String[] {"key", "list", "o3://" + omServiceId +
+ "/volumefso1/bucket1/", "-l ", "110"};
+ out.reset();
+ execute(ozoneShell, args);
+
+ // Total number of keys still 100.
+ Assert.assertEquals(100, getNumOfKeys());
+
+ // Skip Trash
+ args = new String[] {trashConfKey, "key", "delete",
+ "/volumefso1/bucket1/key5", "--skipTrash"};
+ execute(ozoneShell, args);
+
+ // .Trash should still contain 1 key
+ prefixKey = "--prefix=.Trash";
+ args = new String[] {"key", "list", prefixKey, "o3://" +
+ omServiceId + "/volumefso1/bucket1/"};
+ out.reset();
+ execute(ozoneShell, args);
+ Assert.assertEquals(1, getNumOfKeys());
+
+ args = new String[] {"key", "list", "o3://" + omServiceId +
+ "/volumefso1/bucket1/", "-l ", "110"};
+ out.reset();
+ execute(ozoneShell, args);
+ // Total number of keys now will be 99 as
+ // 1 key deleted without trash
+ Assert.assertEquals(99, getNumOfKeys());
Review Comment:
Added test for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ashishkumar50 commented on a diff in pull request #4675: HDDS-8547. Support Trash for FSO bucket using ozone sh command.
Posted by "ashishkumar50 (via GitHub)" <gi...@apache.org>.
ashishkumar50 commented on code in PR #4675:
URL: https://github.com/apache/ozone/pull/4675#discussion_r1190960156
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/DeleteKeyHandler.java:
##########
@@ -45,6 +64,81 @@ protected void execute(OzoneClient client, OzoneAddress address)
OzoneVolume vol = client.getObjectStore().getVolume(volumeName);
OzoneBucket bucket = vol.getBucket(bucketName);
- bucket.deleteKey(keyName);
+
+ float hadoopTrashInterval = getConf().getFloat(
+ FS_TRASH_INTERVAL_KEY, FS_TRASH_INTERVAL_DEFAULT);
+
+ long trashInterval =
+ (long) (getConf().getFloat(
+ OMConfigKeys.OZONE_FS_TRASH_INTERVAL_KEY,
+ hadoopTrashInterval) * 10000);
+
+ // If Bucket layout is FSO and Trash is enabled
+ // In this case during delete operation move key to trash
+ if (bucket.getBucketLayout().isFileSystemOptimized() && trashInterval > 0
+ && !skipTrash && !keyName.contains(".Trash")) {
+
+ keyName = OzoneFSUtils.removeTrailingSlashIfNeeded(keyName);
+ try {
+ // Check if key exists in Ozone
+ OzoneKeyDetails key = bucket.getKey(keyName);
+ if (key == null) {
+ out().printf("Key not found %s", keyName);
+ return;
+ }
+ // Check whether directory is empty or not
+ Iterator<? extends OzoneKey> ozoneKeyIterator =
+ bucket.listKeys(keyName, keyName);
+ int count = 0;
+ while (ozoneKeyIterator.hasNext()) {
+ ozoneKeyIterator.next();
+ if (++count > 1) {
+ // Assume FSO Tree: /a/b1/c1/k1.txt
+ // And we are trying to delete key /a/b1/c1
+ // In this case count is 2 which is greater than 1
+ // /a/b1/c1/ and /a/b1/c1/k1.txt
+ out().printf("Directory is not empty");
+ return;
+ }
+ }
+ } catch (Exception e) {
+ out().printf("Key not found %s", keyName);
+ return;
+ }
+
+ final String username =
+ UserGroupInformation.getCurrentUser().getShortUserName();
+ Path trashRoot = new Path(OZONE_URI_DELIMITER, TRASH_PREFIX);
+ Path userTrash = new Path(trashRoot, username);
+ Path userTrashCurrent = new Path(userTrash, CURRENT);
+
+ String trashDirectory = (keyName.contains("/")
+ ? new Path(userTrashCurrent, keyName.substring(0,
+ keyName.lastIndexOf("/")))
+ : userTrashCurrent).toUri().getPath();
+
+ String toKeyName = new Path(userTrashCurrent, keyName).toUri().getPath();
+ OzoneKeyDetails toKeyDetails = null;
+ try {
+ // check whether key already exist in trash
+ toKeyDetails = bucket.getKey(toKeyName);
+ } catch (IOException e) {
+ // Key doesn't exist inside trash.
+ }
+
+ if (toKeyDetails != null) {
+ // if key(directory) already exist in trash, just delete the key
+ bucket.deleteKey(keyName);
+ return;
+ }
+ // Create directory inside trash
+ bucket.createDirectory(trashDirectory);
Review Comment:
If the directory already exists and we create directory again it doesn't throw error to client because currently it discards DIRECTORY_ALREADY_EXISTS error.
Anyway I have changed now to check if directory exist or not before creating.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ashishkumar50 commented on a diff in pull request #4675: HDDS-8547. Support Trash for FSO bucket using ozone sh command.
Posted by "ashishkumar50 (via GitHub)" <gi...@apache.org>.
ashishkumar50 commented on code in PR #4675:
URL: https://github.com/apache/ozone/pull/4675#discussion_r1192280059
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/DeleteKeyHandler.java:
##########
@@ -35,16 +48,117 @@
description = "deletes an existing key")
public class DeleteKeyHandler extends KeyHandler {
+ @CommandLine.Option(names = "--skipTrash",
+ description = "Specify whether to skip Trash ")
+ private boolean skipTrash = false;
+
+ private static final Path CURRENT = new Path("Current");
+
@Override
protected void execute(OzoneClient client, OzoneAddress address)
throws IOException, OzoneClientException {
String volumeName = address.getVolumeName();
String bucketName = address.getBucketName();
- String keyName = address.getKeyName();
-
OzoneVolume vol = client.getObjectStore().getVolume(volumeName);
OzoneBucket bucket = vol.getBucket(bucketName);
- bucket.deleteKey(keyName);
+ String keyName = address.getKeyName();
+
+ if (bucket.getBucketLayout().isFileSystemOptimized()) {
+ // Handle FSO delete key which supports trash also
+ deleteFSOKey(bucket, keyName);
+ } else {
+ bucket.deleteKey(keyName);
+ }
+ }
+
+ private void deleteFSOKey(OzoneBucket bucket, String keyName)
+ throws IOException {
+ float hadoopTrashInterval = getConf().getFloat(
+ FS_TRASH_INTERVAL_KEY, FS_TRASH_INTERVAL_DEFAULT);
+ long trashInterval =
+ (long) (getConf().getFloat(
+ OMConfigKeys.OZONE_FS_TRASH_INTERVAL_KEY,
+ hadoopTrashInterval) * 10000);
+
+ // If Bucket layout is FSO and Trash is enabled
+ // In this case during delete operation move key to trash
+ if (trashInterval > 0 && !skipTrash &&
Review Comment:
trashInterval is used to determine whether trash is enabled or not, So I think first we should check whether trash enabled or not then we can check trash related configs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ashishkumar50 commented on a diff in pull request #4675: HDDS-8547. Support Trash for FSO bucket using ozone sh command.
Posted by "ashishkumar50 (via GitHub)" <gi...@apache.org>.
ashishkumar50 commented on code in PR #4675:
URL: https://github.com/apache/ozone/pull/4675#discussion_r1190953428
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/DeleteKeyHandler.java:
##########
@@ -45,6 +64,81 @@ protected void execute(OzoneClient client, OzoneAddress address)
OzoneVolume vol = client.getObjectStore().getVolume(volumeName);
OzoneBucket bucket = vol.getBucket(bucketName);
- bucket.deleteKey(keyName);
+
+ float hadoopTrashInterval = getConf().getFloat(
+ FS_TRASH_INTERVAL_KEY, FS_TRASH_INTERVAL_DEFAULT);
+
+ long trashInterval =
+ (long) (getConf().getFloat(
+ OMConfigKeys.OZONE_FS_TRASH_INTERVAL_KEY,
+ hadoopTrashInterval) * 10000);
+
+ // If Bucket layout is FSO and Trash is enabled
+ // In this case during delete operation move key to trash
+ if (bucket.getBucketLayout().isFileSystemOptimized() && trashInterval > 0
+ && !skipTrash && !keyName.contains(".Trash")) {
+
+ keyName = OzoneFSUtils.removeTrailingSlashIfNeeded(keyName);
+ try {
+ // Check if key exists in Ozone
+ OzoneKeyDetails key = bucket.getKey(keyName);
+ if (key == null) {
+ out().printf("Key not found %s", keyName);
+ return;
+ }
+ // Check whether directory is empty or not
+ Iterator<? extends OzoneKey> ozoneKeyIterator =
+ bucket.listKeys(keyName, keyName);
+ int count = 0;
+ while (ozoneKeyIterator.hasNext()) {
+ ozoneKeyIterator.next();
+ if (++count > 1) {
+ // Assume FSO Tree: /a/b1/c1/k1.txt
+ // And we are trying to delete key /a/b1/c1
+ // In this case count is 2 which is greater than 1
+ // /a/b1/c1/ and /a/b1/c1/k1.txt
+ out().printf("Directory is not empty");
+ return;
+ }
+ }
+ } catch (Exception e) {
+ out().printf("Key not found %s", keyName);
+ return;
+ }
+
+ final String username =
+ UserGroupInformation.getCurrentUser().getShortUserName();
+ Path trashRoot = new Path(OZONE_URI_DELIMITER, TRASH_PREFIX);
+ Path userTrash = new Path(trashRoot, username);
+ Path userTrashCurrent = new Path(userTrash, CURRENT);
+
+ String trashDirectory = (keyName.contains("/")
+ ? new Path(userTrashCurrent, keyName.substring(0,
+ keyName.lastIndexOf("/")))
+ : userTrashCurrent).toUri().getPath();
+
+ String toKeyName = new Path(userTrashCurrent, keyName).toUri().getPath();
+ OzoneKeyDetails toKeyDetails = null;
+ try {
+ // check whether key already exist in trash
+ toKeyDetails = bucket.getKey(toKeyName);
+ } catch (IOException e) {
+ // Key doesn't exist inside trash.
+ }
+
+ if (toKeyDetails != null) {
+ // if key(directory) already exist in trash, just delete the key
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ashishkumar50 commented on a diff in pull request #4675: HDDS-8547. Support Trash for FSO bucket using ozone sh command.
Posted by "ashishkumar50 (via GitHub)" <gi...@apache.org>.
ashishkumar50 commented on code in PR #4675:
URL: https://github.com/apache/ozone/pull/4675#discussion_r1192280368
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/DeleteKeyHandler.java:
##########
@@ -35,16 +48,117 @@
description = "deletes an existing key")
public class DeleteKeyHandler extends KeyHandler {
+ @CommandLine.Option(names = "--skipTrash",
+ description = "Specify whether to skip Trash ")
+ private boolean skipTrash = false;
+
+ private static final Path CURRENT = new Path("Current");
+
@Override
protected void execute(OzoneClient client, OzoneAddress address)
throws IOException, OzoneClientException {
String volumeName = address.getVolumeName();
String bucketName = address.getBucketName();
- String keyName = address.getKeyName();
-
OzoneVolume vol = client.getObjectStore().getVolume(volumeName);
OzoneBucket bucket = vol.getBucket(bucketName);
- bucket.deleteKey(keyName);
+ String keyName = address.getKeyName();
+
+ if (bucket.getBucketLayout().isFileSystemOptimized()) {
+ // Handle FSO delete key which supports trash also
+ deleteFSOKey(bucket, keyName);
+ } else {
+ bucket.deleteKey(keyName);
+ }
+ }
+
+ private void deleteFSOKey(OzoneBucket bucket, String keyName)
+ throws IOException {
+ float hadoopTrashInterval = getConf().getFloat(
+ FS_TRASH_INTERVAL_KEY, FS_TRASH_INTERVAL_DEFAULT);
+ long trashInterval =
+ (long) (getConf().getFloat(
+ OMConfigKeys.OZONE_FS_TRASH_INTERVAL_KEY,
+ hadoopTrashInterval) * 10000);
+
+ // If Bucket layout is FSO and Trash is enabled
+ // In this case during delete operation move key to trash
+ if (trashInterval > 0 && !skipTrash &&
+ !keyName.contains(TRASH_PREFIX)) {
+ keyName = OzoneFSUtils.removeTrailingSlashIfNeeded(keyName);
+ // Check if key exists in Ozone
+ boolean bKeyExist = isKeyExist(bucket, keyName);
+ if (!bKeyExist) {
+ out().printf("Key not found %s %n", keyName);
+ return;
+ }
+
+ if (bucket.getFileStatus(keyName).isDirectory()) {
+ List<OzoneFileStatus> ozoneFileStatusList =
+ bucket.listStatus(keyName, false, "", 1);
+ if (ozoneFileStatusList != null && !ozoneFileStatusList.isEmpty()) {
+ out().printf("Directory is not empty %n");
+ return;
+ }
+ }
+
+ final String username =
+ UserGroupInformation.getCurrentUser().getShortUserName();
+ Path trashRoot = new Path(OZONE_URI_DELIMITER, TRASH_PREFIX);
+ Path userTrash = new Path(trashRoot, username);
+ Path userTrashCurrent = new Path(userTrash, CURRENT);
+
+ String trashDirectory = (keyName.contains("/")
+ ? new Path(userTrashCurrent, keyName.substring(0,
+ keyName.lastIndexOf("/")))
+ : userTrashCurrent).toUri().getPath();
+
+ String toKeyName = new Path(userTrashCurrent, keyName).toUri().getPath();
+ bKeyExist = isKeyExist(bucket, toKeyName);
+
+ if (bKeyExist) {
+ if (bucket.getFileStatus(toKeyName).isDirectory()) {
+ // if directory already exist in trash, just delete the directory
+ bucket.deleteKey(keyName);
+ return;
+ }
+ // Key already exists in trash, Append timestamp with keyName
+ // And move into trash
+ // Same behaviour as filesystem trash
+ toKeyName += Time.now();
+ }
+
+ // Check whether trash directory already exist inside bucket
+ bKeyExist = isKeyExist(bucket, trashDirectory);
+ if (!bKeyExist) {
+ // Trash directory doesn't exist
+ // Create directory inside trash
+ bucket.createDirectory(trashDirectory);
+ }
+
+ // Rename key to move inside trash folder
+ bucket.renameKey(keyName, toKeyName);
+ out().printf("Key moved inside Trash: %s %n", toKeyName);
+ } else if (trashInterval > 0 && !skipTrash &&
+ keyName.contains(TRASH_PREFIX)) {
+ // Delete from trash not possible when user didn't do skipTrash
+ out().printf("Use --skipTrash to delete key from Trash %n");
+ } else {
+ bucket.deleteKey(keyName);
+ }
+ }
+
+ private boolean isKeyExist(OzoneBucket bucket, String keyName) {
+ OzoneKeyDetails keyDetails;
+ try {
+ // check whether key exist
+ keyDetails = bucket.getKey(keyName);
+ } catch (IOException e) {
+ return false;
+ }
+ if (keyDetails == null) {
+ return false;
+ }
+ return true;
Review Comment:
Thanks, updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sadanand48 merged pull request #4675: HDDS-8547. Support Trash for FSO bucket using ozone sh command.
Posted by "sadanand48 (via GitHub)" <gi...@apache.org>.
sadanand48 merged PR #4675:
URL: https://github.com/apache/ozone/pull/4675
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ashishkumar50 commented on a diff in pull request #4675: HDDS-8547. Support Trash for FSO bucket using ozone sh command.
Posted by "ashishkumar50 (via GitHub)" <gi...@apache.org>.
ashishkumar50 commented on code in PR #4675:
URL: https://github.com/apache/ozone/pull/4675#discussion_r1188485986
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/DeleteKeyHandler.java:
##########
@@ -45,6 +64,81 @@ protected void execute(OzoneClient client, OzoneAddress address)
OzoneVolume vol = client.getObjectStore().getVolume(volumeName);
OzoneBucket bucket = vol.getBucket(bucketName);
- bucket.deleteKey(keyName);
+
+ float hadoopTrashInterval = getConf().getFloat(
+ FS_TRASH_INTERVAL_KEY, FS_TRASH_INTERVAL_DEFAULT);
+
+ long trashInterval =
+ (long) (getConf().getFloat(
+ OMConfigKeys.OZONE_FS_TRASH_INTERVAL_KEY,
+ hadoopTrashInterval) * 10000);
+
+ // If Bucket layout is FSO and Trash is enabled
+ // In this case during delete operation move key to trash
+ if (bucket.getBucketLayout().isFileSystemOptimized() && trashInterval > 0
+ && !skipTrash && !keyName.contains(".Trash")) {
+
+ keyName = OzoneFSUtils.removeTrailingSlashIfNeeded(keyName);
+ try {
+ // Check if key exists in Ozone
+ OzoneKeyDetails key = bucket.getKey(keyName);
+ if (key == null) {
+ out().printf("Key not found %s", keyName);
+ return;
+ }
+ // Check whether directory is empty or not
+ Iterator<? extends OzoneKey> ozoneKeyIterator =
+ bucket.listKeys(keyName, keyName);
Review Comment:
Handled the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ashishkumar50 commented on a diff in pull request #4675: HDDS-8547. Support Trash for FSO bucket using ozone sh command.
Posted by "ashishkumar50 (via GitHub)" <gi...@apache.org>.
ashishkumar50 commented on code in PR #4675:
URL: https://github.com/apache/ozone/pull/4675#discussion_r1188487163
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/DeleteKeyHandler.java:
##########
@@ -45,6 +64,81 @@ protected void execute(OzoneClient client, OzoneAddress address)
OzoneVolume vol = client.getObjectStore().getVolume(volumeName);
OzoneBucket bucket = vol.getBucket(bucketName);
- bucket.deleteKey(keyName);
+
+ float hadoopTrashInterval = getConf().getFloat(
+ FS_TRASH_INTERVAL_KEY, FS_TRASH_INTERVAL_DEFAULT);
+
+ long trashInterval =
+ (long) (getConf().getFloat(
+ OMConfigKeys.OZONE_FS_TRASH_INTERVAL_KEY,
+ hadoopTrashInterval) * 10000);
+
+ // If Bucket layout is FSO and Trash is enabled
+ // In this case during delete operation move key to trash
+ if (bucket.getBucketLayout().isFileSystemOptimized() && trashInterval > 0
+ && !skipTrash && !keyName.contains(".Trash")) {
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on a diff in pull request #4675: HDDS-8547. Support Trash for FSO bucket using ozone sh command.
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on code in PR #4675:
URL: https://github.com/apache/ozone/pull/4675#discussion_r1192190872
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/DeleteKeyHandler.java:
##########
@@ -35,16 +48,117 @@
description = "deletes an existing key")
public class DeleteKeyHandler extends KeyHandler {
+ @CommandLine.Option(names = "--skipTrash",
+ description = "Specify whether to skip Trash ")
+ private boolean skipTrash = false;
+
+ private static final Path CURRENT = new Path("Current");
+
@Override
protected void execute(OzoneClient client, OzoneAddress address)
throws IOException, OzoneClientException {
String volumeName = address.getVolumeName();
String bucketName = address.getBucketName();
- String keyName = address.getKeyName();
-
OzoneVolume vol = client.getObjectStore().getVolume(volumeName);
OzoneBucket bucket = vol.getBucket(bucketName);
- bucket.deleteKey(keyName);
+ String keyName = address.getKeyName();
+
+ if (bucket.getBucketLayout().isFileSystemOptimized()) {
+ // Handle FSO delete key which supports trash also
+ deleteFSOKey(bucket, keyName);
+ } else {
+ bucket.deleteKey(keyName);
+ }
+ }
+
+ private void deleteFSOKey(OzoneBucket bucket, String keyName)
+ throws IOException {
+ float hadoopTrashInterval = getConf().getFloat(
+ FS_TRASH_INTERVAL_KEY, FS_TRASH_INTERVAL_DEFAULT);
+ long trashInterval =
+ (long) (getConf().getFloat(
+ OMConfigKeys.OZONE_FS_TRASH_INTERVAL_KEY,
+ hadoopTrashInterval) * 10000);
+
+ // If Bucket layout is FSO and Trash is enabled
+ // In this case during delete operation move key to trash
+ if (trashInterval > 0 && !skipTrash &&
+ !keyName.contains(TRASH_PREFIX)) {
+ keyName = OzoneFSUtils.removeTrailingSlashIfNeeded(keyName);
+ // Check if key exists in Ozone
+ boolean bKeyExist = isKeyExist(bucket, keyName);
+ if (!bKeyExist) {
Review Comment:
can we directly check the if condition on "`isKeyExist`" method if "bKeyExist" not referenced ?
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/DeleteKeyHandler.java:
##########
@@ -35,16 +48,117 @@
description = "deletes an existing key")
public class DeleteKeyHandler extends KeyHandler {
+ @CommandLine.Option(names = "--skipTrash",
+ description = "Specify whether to skip Trash ")
+ private boolean skipTrash = false;
+
+ private static final Path CURRENT = new Path("Current");
+
@Override
protected void execute(OzoneClient client, OzoneAddress address)
throws IOException, OzoneClientException {
String volumeName = address.getVolumeName();
String bucketName = address.getBucketName();
- String keyName = address.getKeyName();
-
OzoneVolume vol = client.getObjectStore().getVolume(volumeName);
OzoneBucket bucket = vol.getBucket(bucketName);
- bucket.deleteKey(keyName);
+ String keyName = address.getKeyName();
+
+ if (bucket.getBucketLayout().isFileSystemOptimized()) {
+ // Handle FSO delete key which supports trash also
+ deleteFSOKey(bucket, keyName);
+ } else {
+ bucket.deleteKey(keyName);
+ }
+ }
+
+ private void deleteFSOKey(OzoneBucket bucket, String keyName)
+ throws IOException {
+ float hadoopTrashInterval = getConf().getFloat(
+ FS_TRASH_INTERVAL_KEY, FS_TRASH_INTERVAL_DEFAULT);
+ long trashInterval =
+ (long) (getConf().getFloat(
+ OMConfigKeys.OZONE_FS_TRASH_INTERVAL_KEY,
+ hadoopTrashInterval) * 10000);
+
+ // If Bucket layout is FSO and Trash is enabled
+ // In this case during delete operation move key to trash
+ if (trashInterval > 0 && !skipTrash &&
Review Comment:
Since we are adding the skipTrash flag here, just a minor suggestion to reduce one operation in if condition for trashInterval and check like below:
if (!skipTrash && trashInterval > 0 && !keyName.contains(TRASH_PREFIX)) as skipTrash itself may be false.
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/DeleteKeyHandler.java:
##########
@@ -35,16 +48,117 @@
description = "deletes an existing key")
public class DeleteKeyHandler extends KeyHandler {
+ @CommandLine.Option(names = "--skipTrash",
+ description = "Specify whether to skip Trash ")
+ private boolean skipTrash = false;
+
+ private static final Path CURRENT = new Path("Current");
+
@Override
protected void execute(OzoneClient client, OzoneAddress address)
throws IOException, OzoneClientException {
String volumeName = address.getVolumeName();
String bucketName = address.getBucketName();
- String keyName = address.getKeyName();
-
OzoneVolume vol = client.getObjectStore().getVolume(volumeName);
OzoneBucket bucket = vol.getBucket(bucketName);
- bucket.deleteKey(keyName);
+ String keyName = address.getKeyName();
+
+ if (bucket.getBucketLayout().isFileSystemOptimized()) {
+ // Handle FSO delete key which supports trash also
+ deleteFSOKey(bucket, keyName);
+ } else {
+ bucket.deleteKey(keyName);
+ }
+ }
+
+ private void deleteFSOKey(OzoneBucket bucket, String keyName)
+ throws IOException {
+ float hadoopTrashInterval = getConf().getFloat(
+ FS_TRASH_INTERVAL_KEY, FS_TRASH_INTERVAL_DEFAULT);
+ long trashInterval =
+ (long) (getConf().getFloat(
+ OMConfigKeys.OZONE_FS_TRASH_INTERVAL_KEY,
+ hadoopTrashInterval) * 10000);
+
+ // If Bucket layout is FSO and Trash is enabled
+ // In this case during delete operation move key to trash
+ if (trashInterval > 0 && !skipTrash &&
+ !keyName.contains(TRASH_PREFIX)) {
+ keyName = OzoneFSUtils.removeTrailingSlashIfNeeded(keyName);
+ // Check if key exists in Ozone
+ boolean bKeyExist = isKeyExist(bucket, keyName);
+ if (!bKeyExist) {
+ out().printf("Key not found %s %n", keyName);
+ return;
+ }
+
+ if (bucket.getFileStatus(keyName).isDirectory()) {
+ List<OzoneFileStatus> ozoneFileStatusList =
+ bucket.listStatus(keyName, false, "", 1);
+ if (ozoneFileStatusList != null && !ozoneFileStatusList.isEmpty()) {
+ out().printf("Directory is not empty %n");
+ return;
+ }
+ }
+
+ final String username =
+ UserGroupInformation.getCurrentUser().getShortUserName();
+ Path trashRoot = new Path(OZONE_URI_DELIMITER, TRASH_PREFIX);
+ Path userTrash = new Path(trashRoot, username);
+ Path userTrashCurrent = new Path(userTrash, CURRENT);
+
+ String trashDirectory = (keyName.contains("/")
+ ? new Path(userTrashCurrent, keyName.substring(0,
+ keyName.lastIndexOf("/")))
+ : userTrashCurrent).toUri().getPath();
+
+ String toKeyName = new Path(userTrashCurrent, keyName).toUri().getPath();
+ bKeyExist = isKeyExist(bucket, toKeyName);
+
+ if (bKeyExist) {
+ if (bucket.getFileStatus(toKeyName).isDirectory()) {
+ // if directory already exist in trash, just delete the directory
+ bucket.deleteKey(keyName);
+ return;
+ }
+ // Key already exists in trash, Append timestamp with keyName
+ // And move into trash
+ // Same behaviour as filesystem trash
+ toKeyName += Time.now();
+ }
+
+ // Check whether trash directory already exist inside bucket
+ bKeyExist = isKeyExist(bucket, trashDirectory);
+ if (!bKeyExist) {
+ // Trash directory doesn't exist
+ // Create directory inside trash
+ bucket.createDirectory(trashDirectory);
+ }
+
+ // Rename key to move inside trash folder
+ bucket.renameKey(keyName, toKeyName);
+ out().printf("Key moved inside Trash: %s %n", toKeyName);
+ } else if (trashInterval > 0 && !skipTrash &&
+ keyName.contains(TRASH_PREFIX)) {
+ // Delete from trash not possible when user didn't do skipTrash
+ out().printf("Use --skipTrash to delete key from Trash %n");
+ } else {
+ bucket.deleteKey(keyName);
+ }
+ }
+
+ private boolean isKeyExist(OzoneBucket bucket, String keyName) {
+ OzoneKeyDetails keyDetails;
+ try {
+ // check whether key exist
+ keyDetails = bucket.getKey(keyName);
+ } catch (IOException e) {
+ return false;
+ }
+ if (keyDetails == null) {
+ return false;
+ }
+ return true;
Review Comment:
can we change with one line like below:
return (keyDetails != null)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sadanand48 commented on a diff in pull request #4675: HDDS-8547. Support Trash for FSO bucket using ozone sh command.
Posted by "sadanand48 (via GitHub)" <gi...@apache.org>.
sadanand48 commented on code in PR #4675:
URL: https://github.com/apache/ozone/pull/4675#discussion_r1187328558
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/DeleteKeyHandler.java:
##########
@@ -45,6 +64,81 @@ protected void execute(OzoneClient client, OzoneAddress address)
OzoneVolume vol = client.getObjectStore().getVolume(volumeName);
OzoneBucket bucket = vol.getBucket(bucketName);
- bucket.deleteKey(keyName);
+
+ float hadoopTrashInterval = getConf().getFloat(
+ FS_TRASH_INTERVAL_KEY, FS_TRASH_INTERVAL_DEFAULT);
+
+ long trashInterval =
+ (long) (getConf().getFloat(
+ OMConfigKeys.OZONE_FS_TRASH_INTERVAL_KEY,
+ hadoopTrashInterval) * 10000);
+
+ // If Bucket layout is FSO and Trash is enabled
+ // In this case during delete operation move key to trash
+ if (bucket.getBucketLayout().isFileSystemOptimized() && trashInterval > 0
+ && !skipTrash && !keyName.contains(".Trash")) {
+
+ keyName = OzoneFSUtils.removeTrailingSlashIfNeeded(keyName);
+ try {
+ // Check if key exists in Ozone
+ OzoneKeyDetails key = bucket.getKey(keyName);
+ if (key == null) {
+ out().printf("Key not found %s", keyName);
+ return;
+ }
+ // Check whether directory is empty or not
+ Iterator<? extends OzoneKey> ozoneKeyIterator =
+ bucket.listKeys(keyName, keyName);
+ int count = 0;
+ while (ozoneKeyIterator.hasNext()) {
+ ozoneKeyIterator.next();
+ if (++count > 1) {
+ // Assume FSO Tree: /a/b1/c1/k1.txt
+ // And we are trying to delete key /a/b1/c1
+ // In this case count is 2 which is greater than 1
+ // /a/b1/c1/ and /a/b1/c1/k1.txt
+ out().printf("Directory is not empty");
+ return;
+ }
+ }
+ } catch (Exception e) {
+ out().printf("Key not found %s", keyName);
+ return;
+ }
+
+ final String username =
+ UserGroupInformation.getCurrentUser().getShortUserName();
+ Path trashRoot = new Path(OZONE_URI_DELIMITER, TRASH_PREFIX);
+ Path userTrash = new Path(trashRoot, username);
+ Path userTrashCurrent = new Path(userTrash, CURRENT);
+
+ String trashDirectory = (keyName.contains("/")
+ ? new Path(userTrashCurrent, keyName.substring(0,
+ keyName.lastIndexOf("/")))
+ : userTrashCurrent).toUri().getPath();
+
+ String toKeyName = new Path(userTrashCurrent, keyName).toUri().getPath();
+ OzoneKeyDetails toKeyDetails = null;
+ try {
+ // check whether key already exist in trash
+ toKeyDetails = bucket.getKey(toKeyName);
+ } catch (IOException e) {
+ // Key doesn't exist inside trash.
+ }
+
+ if (toKeyDetails != null) {
+ // if key(directory) already exist in trash, just delete the key
Review Comment:
Not sure about this, we don't follow the same behaviour with filesystem trash, after deleting the key once , it should not be possible to the user to delete again, we have skipTrash for the same reason. If user wants to delete it from trash then the path must be given accordingly.
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/DeleteKeyHandler.java:
##########
@@ -45,6 +64,81 @@ protected void execute(OzoneClient client, OzoneAddress address)
OzoneVolume vol = client.getObjectStore().getVolume(volumeName);
OzoneBucket bucket = vol.getBucket(bucketName);
- bucket.deleteKey(keyName);
+
+ float hadoopTrashInterval = getConf().getFloat(
+ FS_TRASH_INTERVAL_KEY, FS_TRASH_INTERVAL_DEFAULT);
+
+ long trashInterval =
+ (long) (getConf().getFloat(
+ OMConfigKeys.OZONE_FS_TRASH_INTERVAL_KEY,
+ hadoopTrashInterval) * 10000);
+
+ // If Bucket layout is FSO and Trash is enabled
+ // In this case during delete operation move key to trash
+ if (bucket.getBucketLayout().isFileSystemOptimized() && trashInterval > 0
+ && !skipTrash && !keyName.contains(".Trash")) {
Review Comment:
TRASH_PREFIX instead of ".Trash"
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/DeleteKeyHandler.java:
##########
@@ -45,6 +64,81 @@ protected void execute(OzoneClient client, OzoneAddress address)
OzoneVolume vol = client.getObjectStore().getVolume(volumeName);
OzoneBucket bucket = vol.getBucket(bucketName);
- bucket.deleteKey(keyName);
+
+ float hadoopTrashInterval = getConf().getFloat(
+ FS_TRASH_INTERVAL_KEY, FS_TRASH_INTERVAL_DEFAULT);
+
+ long trashInterval =
+ (long) (getConf().getFloat(
+ OMConfigKeys.OZONE_FS_TRASH_INTERVAL_KEY,
+ hadoopTrashInterval) * 10000);
+
+ // If Bucket layout is FSO and Trash is enabled
+ // In this case during delete operation move key to trash
+ if (bucket.getBucketLayout().isFileSystemOptimized() && trashInterval > 0
Review Comment:
Can we add all this logic to a separate method to handle FSO key deletes? something like deleteFSOKey()
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/DeleteKeyHandler.java:
##########
@@ -45,6 +64,81 @@ protected void execute(OzoneClient client, OzoneAddress address)
OzoneVolume vol = client.getObjectStore().getVolume(volumeName);
OzoneBucket bucket = vol.getBucket(bucketName);
- bucket.deleteKey(keyName);
+
+ float hadoopTrashInterval = getConf().getFloat(
+ FS_TRASH_INTERVAL_KEY, FS_TRASH_INTERVAL_DEFAULT);
+
+ long trashInterval =
+ (long) (getConf().getFloat(
+ OMConfigKeys.OZONE_FS_TRASH_INTERVAL_KEY,
+ hadoopTrashInterval) * 10000);
+
+ // If Bucket layout is FSO and Trash is enabled
+ // In this case during delete operation move key to trash
+ if (bucket.getBucketLayout().isFileSystemOptimized() && trashInterval > 0
+ && !skipTrash && !keyName.contains(".Trash")) {
+
+ keyName = OzoneFSUtils.removeTrailingSlashIfNeeded(keyName);
+ try {
+ // Check if key exists in Ozone
+ OzoneKeyDetails key = bucket.getKey(keyName);
+ if (key == null) {
+ out().printf("Key not found %s", keyName);
+ return;
+ }
+ // Check whether directory is empty or not
+ Iterator<? extends OzoneKey> ozoneKeyIterator =
+ bucket.listKeys(keyName, keyName);
Review Comment:
bucket.listKeys() by default will return 1000 entries in each RPC. This might be a little unnecessary as we only want to check if the directory is empty or not here, Even there is a single key with given prefix, it should conclude that directory is non-empty.
1. I couldn't find an API for listKeys with maxEntries param but instead you could use
``` java
public List<OzoneFileStatus> listStatus(String keyName, boolean recursive,
String startKey, long numEntries) throws IOException
```
and pass numEntries as 1.
3. To check if it is directory you could use ` bucket.getFileStatus(key).isDirectory();`
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/DeleteKeyHandler.java:
##########
@@ -45,6 +64,81 @@ protected void execute(OzoneClient client, OzoneAddress address)
OzoneVolume vol = client.getObjectStore().getVolume(volumeName);
OzoneBucket bucket = vol.getBucket(bucketName);
- bucket.deleteKey(keyName);
+
+ float hadoopTrashInterval = getConf().getFloat(
+ FS_TRASH_INTERVAL_KEY, FS_TRASH_INTERVAL_DEFAULT);
+
+ long trashInterval =
+ (long) (getConf().getFloat(
+ OMConfigKeys.OZONE_FS_TRASH_INTERVAL_KEY,
+ hadoopTrashInterval) * 10000);
+
+ // If Bucket layout is FSO and Trash is enabled
+ // In this case during delete operation move key to trash
+ if (bucket.getBucketLayout().isFileSystemOptimized() && trashInterval > 0
+ && !skipTrash && !keyName.contains(".Trash")) {
+
+ keyName = OzoneFSUtils.removeTrailingSlashIfNeeded(keyName);
+ try {
+ // Check if key exists in Ozone
+ OzoneKeyDetails key = bucket.getKey(keyName);
+ if (key == null) {
+ out().printf("Key not found %s", keyName);
+ return;
+ }
+ // Check whether directory is empty or not
+ Iterator<? extends OzoneKey> ozoneKeyIterator =
+ bucket.listKeys(keyName, keyName);
+ int count = 0;
+ while (ozoneKeyIterator.hasNext()) {
+ ozoneKeyIterator.next();
+ if (++count > 1) {
+ // Assume FSO Tree: /a/b1/c1/k1.txt
+ // And we are trying to delete key /a/b1/c1
+ // In this case count is 2 which is greater than 1
+ // /a/b1/c1/ and /a/b1/c1/k1.txt
+ out().printf("Directory is not empty");
+ return;
+ }
+ }
+ } catch (Exception e) {
+ out().printf("Key not found %s", keyName);
+ return;
+ }
+
+ final String username =
+ UserGroupInformation.getCurrentUser().getShortUserName();
+ Path trashRoot = new Path(OZONE_URI_DELIMITER, TRASH_PREFIX);
+ Path userTrash = new Path(trashRoot, username);
+ Path userTrashCurrent = new Path(userTrash, CURRENT);
+
+ String trashDirectory = (keyName.contains("/")
+ ? new Path(userTrashCurrent, keyName.substring(0,
+ keyName.lastIndexOf("/")))
+ : userTrashCurrent).toUri().getPath();
+
+ String toKeyName = new Path(userTrashCurrent, keyName).toUri().getPath();
+ OzoneKeyDetails toKeyDetails = null;
+ try {
+ // check whether key already exist in trash
+ toKeyDetails = bucket.getKey(toKeyName);
+ } catch (IOException e) {
+ // Key doesn't exist inside trash.
+ }
+
+ if (toKeyDetails != null) {
+ // if key(directory) already exist in trash, just delete the key
+ bucket.deleteKey(keyName);
+ return;
+ }
+ // Create directory inside trash
+ bucket.createDirectory(trashDirectory);
Review Comment:
Do we need to manually create this , Doesn't rename take care of creating missing parent directories for FSO keys. I think it does.
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneShellHA.java:
##########
@@ -1124,6 +1129,127 @@ public void testCreateBucketWithECReplicationConfigWithoutReplicationParam() {
}
}
+
+ @Test
+ public void testKeyDeleteOrSkipTrashWhenTrashEnableFSO()
+ throws UnsupportedEncodingException {
+ // Create 100 keys
+ generateKeys("/volumefso1", "/bucket1",
+ BucketLayout.FILE_SYSTEM_OPTIMIZED.toString());
+
+ // Enable trash
+ String trashConfKey = generateSetConfString(
+ OMConfigKeys.OZONE_FS_TRASH_INTERVAL_KEY, "1");
+ String[] args =
+ new String[] {trashConfKey, "key", "delete",
+ "/volumefso1/bucket1/key4"};
+
+ // Delete one key from FSO bucket
+ execute(ozoneShell, args);
+
+ // Get key list in .Trash path
+ String prefixKey = "--prefix=.Trash";
+ args = new String[] {"key", "list", prefixKey, "o3://" +
+ omServiceId + "/volumefso1/bucket1/"};
+ out.reset();
+ execute(ozoneShell, args);
+
+ // One key should be present in .Trash
+ Assert.assertEquals(1, getNumOfKeys());
+
+ args = new String[] {"key", "list", "o3://" + omServiceId +
+ "/volumefso1/bucket1/", "-l ", "110"};
+ out.reset();
+ execute(ozoneShell, args);
+
+ // Total number of keys still 100.
+ Assert.assertEquals(100, getNumOfKeys());
+
+ // Skip Trash
+ args = new String[] {trashConfKey, "key", "delete",
+ "/volumefso1/bucket1/key5", "--skipTrash"};
+ execute(ozoneShell, args);
+
+ // .Trash should still contain 1 key
+ prefixKey = "--prefix=.Trash";
+ args = new String[] {"key", "list", prefixKey, "o3://" +
+ omServiceId + "/volumefso1/bucket1/"};
+ out.reset();
+ execute(ozoneShell, args);
+ Assert.assertEquals(1, getNumOfKeys());
+
+ args = new String[] {"key", "list", "o3://" + omServiceId +
+ "/volumefso1/bucket1/", "-l ", "110"};
+ out.reset();
+ execute(ozoneShell, args);
+ // Total number of keys now will be 99 as
+ // 1 key deleted without trash
+ Assert.assertEquals(99, getNumOfKeys());
Review Comment:
maybe add a small testcase to check delete key from trash should be a no op. then delete key from trash with skipTrash should actually delete it
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/DeleteKeyHandler.java:
##########
@@ -45,6 +64,81 @@ protected void execute(OzoneClient client, OzoneAddress address)
OzoneVolume vol = client.getObjectStore().getVolume(volumeName);
OzoneBucket bucket = vol.getBucket(bucketName);
- bucket.deleteKey(keyName);
+
+ float hadoopTrashInterval = getConf().getFloat(
+ FS_TRASH_INTERVAL_KEY, FS_TRASH_INTERVAL_DEFAULT);
+
+ long trashInterval =
+ (long) (getConf().getFloat(
+ OMConfigKeys.OZONE_FS_TRASH_INTERVAL_KEY,
+ hadoopTrashInterval) * 10000);
+
+ // If Bucket layout is FSO and Trash is enabled
+ // In this case during delete operation move key to trash
+ if (bucket.getBucketLayout().isFileSystemOptimized() && trashInterval > 0
Review Comment:
Can we add all this logic to a separate method to handle FSO key deletes? something like :
```
if (bucket.isFSO()) {
deleteFSOKey();
} else {
bucket.deleteKey();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org