You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2021/07/26 03:33:20 UTC

[GitHub] [ozone] neils-dev opened a new pull request #2458: Minor changes to raise exception with hint should ozone fs shell recu…

neils-dev opened a new pull request #2458:
URL: https://github.com/apache/ozone/pull/2458


   …rive delete of bucket invoked with Trash enabled.
   
   ## What changes were proposed in this pull request?
   
   When the trash service is enabled and a _bucket_ recursive delete, `rm -r`, ozone fs command is invoked a fatal error occurs ( _**Fatal internal error**_ for _ofs_ and _**Null Pointer Exception**_ for _o3fs_). 
   
   This patch fixes the fatal error condition for handling the ozone fs shell `rm -r` command for buckets through the ozone trash policy specific _moveToTrash_ method.  When the recursive rm is invoked with trash enabled, the trash policy _moveToTrash_ method is called to provide placing the item (through rename) to the bucket specific trash.  Because the the bucket is recursively deleted along with all keys in this case, the Trash directory (found under the bucket) is also affected causing the fatal error.  This patch detects the error condition and handles the condition by throwing an informative error and directs the user to apply _**-skipTrash**_ option to override and perform the delete.
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-5175
   
   ## How was this patch tested?
   Patch was tested invoking ozone fs shell commands with trash service enabled on the docker ozonesecure cluster.
   
   ```
   $ cd hadoop-ozone/dist/target/ozone-1.2.0-SNAPSHOT/compose/ozonesecure
   $ docker-compose up -d --scale datanode=3
   $ docker-compose exec scm bash
   ```
   
   ```
   bash-4.2$ kinit -kt /etc/security/keytabs/testuser.keytab testuser/scm@EXAMPLE.COM
   bash-4.2$ ozone sh volume create vol1
   bash-4.2$  ozone fs -mkdir -p /vol1/bucket1
   bash-4.2$ ozone fs -touch /vol1/bucket1/key1
   
   ```
   To create error:
   ofs:
   `bash-4.2$ ozone fs -rm -R /vol1/bucket1`
   o3fs:
   `ozone fs -rm -R o3fs://bucket1.vol1`
   
   **_Apply patch through core-site configuration: using ozone specific trash policy:_**
   
   set **`_<property> <name>fs.trash.classname</name><value>org.apache.hadoop.ozone.om.TrashPolicyOzone</value> </property>`_**
   
   set `fs.trash.classname=org.apache.hadoop.ozone.om.TrashPolicyOzone
   `
   `bash-4.2$ vi /etc/hadoop/core-site.xml`
   insert `<property> <name>fs.trash.classname</name><value>org.apache.hadoop.ozone.om.TrashPolicyOzone</value> </property>`
   
   _ofs fix:_
   ![recursive_rm_bucket_ofs](https://user-images.githubusercontent.com/81126310/126928241-ed5f0898-043f-42eb-b02a-cb6fcaf32783.png)
   
   _o3fs fix:_
   ![recursive_rm_bucket_o3fs](https://user-images.githubusercontent.com/81126310/126928256-9fa00a1c-46b0-4ac2-b2f4-50f4293fd81b.png)
   
   
   override and delete with `-skipTrash` option:
   ![recursive_rm_bucket_skiptrash_ofs](https://user-images.githubusercontent.com/81126310/126928306-82b6f0b7-e992-4d5e-ae1b-ad8f595e3d69.png)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] sadanand48 commented on pull request #2458: HDDS-5175. ozone fs trash service does not handle bucket -rm -R

Posted by GitBox <gi...@apache.org>.
sadanand48 commented on pull request #2458:
URL: https://github.com/apache/ozone/pull/2458#issuecomment-887375210


   Thanks @avijayanhwx for the suggestion . We can  maybe mention this in the ofs/o3fs docs (https://github.com/apache/ozone/blame/master/hadoop-hdds/docs/content/interface/Ofs.md#L199) 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] adoroszlai commented on pull request #2458: HDDS-5175. ozone fs trash service does not handle bucket -rm -R

Posted by GitBox <gi...@apache.org>.
adoroszlai commented on pull request #2458:
URL: https://github.com/apache/ozone/pull/2458#issuecomment-901032151


   Thanks @neils-dev for the patch, @avijayanhwx and @sadanand48 for the review.
   
   @sadanand48 FYI, the same problem affects Ozone with `TrashPolicyDefault` (if `fs.trash.classname` is not set).  Is this a valid use case?  Can we do something to avoid using this trash policy, e.g. update Ozone's default setting to the custom implementation?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] adoroszlai commented on pull request #2458: HDDS-5175. ozone fs trash service does not handle bucket -rm -R

Posted by GitBox <gi...@apache.org>.
adoroszlai commented on pull request #2458:
URL: https://github.com/apache/ozone/pull/2458#issuecomment-901032151


   Thanks @neils-dev for the patch, @avijayanhwx and @sadanand48 for the review.
   
   @sadanand48 FYI, the same problem affects Ozone with `TrashPolicyDefault` (if `fs.trash.classname` is not set).  Is this a valid use case?  Can we do something to avoid using this trash policy, e.g. update Ozone's default setting to the custom implementation?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] adoroszlai merged pull request #2458: HDDS-5175. ozone fs trash service does not handle bucket -rm -R

Posted by GitBox <gi...@apache.org>.
adoroszlai merged pull request #2458:
URL: https://github.com/apache/ozone/pull/2458


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] neils-dev commented on pull request #2458: HDDS-5175. ozone fs trash service does not handle bucket -rm -R

Posted by GitBox <gi...@apache.org>.
neils-dev commented on pull request #2458:
URL: https://github.com/apache/ozone/pull/2458#issuecomment-887242940


   > Thanks for working on this @neils-dev. LGTM. Is there a doc item for Ozone trash or Bucket delete where we can add a statement that bucket delete is not allowed without skipTrash?
   
   Thanks for your review and comments @avijayanhwx.  There indeed is a doc and associated jira for the Ozone trash feature, I believe it is HDDS-3367.  I'd be happy to add a statement to the jira.  Also, @sadanand48 any thoughts on this? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] siddhantsangwan commented on pull request #2458: HDDS-5175. ozone fs trash service does not handle bucket -rm -R

Posted by GitBox <gi...@apache.org>.
siddhantsangwan commented on pull request #2458:
URL: https://github.com/apache/ozone/pull/2458#issuecomment-951641481


    > FYI, the same problem affects Ozone with `TrashPolicyDefault` (if `fs.trash.classname` is not set). Is this a valid use case? Can we do something to avoid using this trash policy, e.g. update Ozone's default setting to the custom implementation?
   
   @adoroszlai If you are aware of any progress on this problem in the case of `TrashPolicyDefault`, please let me know. I've created [HDDS-5901](https://issues.apache.org/jira/browse/HDDS-5901) to track this.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] sadanand48 commented on pull request #2458: HDDS-5175. ozone fs trash service does not handle bucket -rm -R

Posted by GitBox <gi...@apache.org>.
sadanand48 commented on pull request #2458:
URL: https://github.com/apache/ozone/pull/2458#issuecomment-893580886


   Created PR https://github.com/apache/ozone/pull/2505 for the documentation part.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] adoroszlai merged pull request #2458: HDDS-5175. ozone fs trash service does not handle bucket -rm -R

Posted by GitBox <gi...@apache.org>.
adoroszlai merged pull request #2458:
URL: https://github.com/apache/ozone/pull/2458


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org