You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "bb (Jira)" <ji...@apache.org> on 2020/02/08 19:43:00 UTC
[jira] [Updated] (ARROW-7805) Apache Arrow HDFS Remove (rm)
operation defaults to SkipTrash
[ https://issues.apache.org/jira/browse/ARROW-7805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
bb updated ARROW-7805:
----------------------
Description:
The Pyarrow lib (using LIBHDFS) appears to default to a SkipTrash option (which is _not_ the Hadoop default behavior). This turned out to be a pretty major issue for a project. The HadoopFileSystem `delete` method currently has a default behavior of `recursive=False` I would believe a similar `skipTrash=False` default would be appropriate here.
{code:java}
# test using hadoop fs shell commands
# setup test & confirm that file exists
$ testfile="/user/myusername/testfile1"
$ hadoop fs -touchz $testfile && hadoop fs -ls $testfile
-rw-r----- 3 myusername mygroup 0 2020-02-08 13:25 /user/myusername/testfile1
# remove the file and confirm that it is moved to the Trash
$ hadoop fs -rm $testfile
20/02/08 13:26:04 INFO fs.TrashPolicyDefault: Moved: 'hdfs://nameservice1/user/myusername/testfile1' to trash at: hdfs://nameservice1/user/.Trash/myusername/Current/user/myusername/testfile1
# verify that it is in the Trash
$ hadoop fs -ls /user/.Trash/myusername/Current/user/myusername/testfile1
-rw-r----- 3 myusername mygroup 0 2020-02-08 13:25 /user/.Trash/myusername/Current/user/myusername/testfile1
{code}
was:
The Pyarrow lib (using LIBHDFS) appears to default to a SkipTrash option (which is _not_ the Hadoop default behavior). This turned out to be a pretty major issue for a project. The HadoopFileSystem `delete` method currently has a default behavior of `recursive=False` I would believe a similar `skipTrash=False` default would be appropriate here.
{code:java}
# test to prove that default hdfs behavior moves a file to the Trash{code}
> Apache Arrow HDFS Remove (rm) operation defaults to SkipTrash
> -------------------------------------------------------------
>
> Key: ARROW-7805
> URL: https://issues.apache.org/jira/browse/ARROW-7805
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
> Affects Versions: 0.13.0
> Reporter: bb
> Priority: Major
>
> The Pyarrow lib (using LIBHDFS) appears to default to a SkipTrash option (which is _not_ the Hadoop default behavior). This turned out to be a pretty major issue for a project. The HadoopFileSystem `delete` method currently has a default behavior of `recursive=False` I would believe a similar `skipTrash=False` default would be appropriate here.
>
> {code:java}
> # test using hadoop fs shell commands
> # setup test & confirm that file exists
> $ testfile="/user/myusername/testfile1"
> $ hadoop fs -touchz $testfile && hadoop fs -ls $testfile
> -rw-r----- 3 myusername mygroup 0 2020-02-08 13:25 /user/myusername/testfile1
> # remove the file and confirm that it is moved to the Trash
> $ hadoop fs -rm $testfile
> 20/02/08 13:26:04 INFO fs.TrashPolicyDefault: Moved: 'hdfs://nameservice1/user/myusername/testfile1' to trash at: hdfs://nameservice1/user/.Trash/myusername/Current/user/myusername/testfile1
> # verify that it is in the Trash
> $ hadoop fs -ls /user/.Trash/myusername/Current/user/myusername/testfile1
> -rw-r----- 3 myusername mygroup 0 2020-02-08 13:25 /user/.Trash/myusername/Current/user/myusername/testfile1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)