[jira] [Commented] (ARROW-7805) [C++][Python] HDFS Remove (rm) operation defaults to SkipTrash

["Wes McKinney (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-7805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17033697#comment-17033697 ] 

Wes McKinney commented on ARROW-7805:
-------------------------------------

Can you look at our usage of the libhdfs C APIs and see if we're doing something wrong?

> [C++][Python] HDFS Remove (rm) operation defaults to SkipTrash
> --------------------------------------------------------------
>
>                 Key: ARROW-7805
>                 URL: https://issues.apache.org/jira/browse/ARROW-7805
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: Python
>    Affects Versions: 0.13.0
>            Reporter: bb
>            Priority: Major
>
> The Pyarrow lib (using LIBHDFS) appears to default to a SkipTrash option (which is _not_ the Hadoop default behavior).  This turned out to be a pretty major issue for a recent project.  The HadoopFileSystem `delete` method currently has a default behavior of `recursive=False` I would believe a similar `skipTrash=False` default would be appropriate here.
> Or, if this is not possible it may be appropriate and best to print a `WARNING` or something to the console to warn users that this is a point-of-no-return.
> The 2 below tests show the difference in default behaviors between `hadoop fs` shell commands and the default `pyarrow` behavior.
>  
> {code:java}
> # test using hadoop fs shell commands 
> # setup test & confirm that file exists 
> $ testfile="/user/myusername/testfile1" 
> $ hadoop fs -touchz $testfile && hadoop fs -ls $testfile 
> -rw-r----- 3 myusername mygroup 0 2020-02-08 13:25 /user/myusername/testfile1 
> # remove the file and confirm that it is moved to the Trash 
> $ hadoop fs -rm $testfile 
> 20/02/08 13:26:04 INFO fs.TrashPolicyDefault: Moved: 'hdfs://nameservice1/user/myusername/testfile1' to trash at: hdfs://nameservice1/user/.Trash/myusername/Current/user/myusername/testfile1 
> # verify that it is in the Trash 
> $ hadoop fs -ls /user/.Trash/myusername/Current/user/myusername/testfile1 
> -rw-r----- 3 myusername mygroup 0 2020-02-08 13:25 /user/.Trash/myusername/Current/user/myusername/testfile1
> {code}
> {code:java}
>  # test using pyarrow
> import os
> import subprocess
> from app import conf
> LIBHDFS_PATH = conf["libhdfs_path"]
> os.environ["ARROW_LIBHDFS_DIR"] = LIBHDFS_PATH
> import pyarrow
> TEST_FILE = 'testfile2'
> TEST_FILE_PATH = f'/user/myusername/{TEST_FILE}'
> TRASH_FILE_PATH = f'/user/.Trash/myusername/Current/user/myusername/{TEST_FILE}'
> fs = pyarrow.hdfs.connect(driver="libhdfs")
> def setup_test():
>     """Create the testfile"""
>     print('create test file...')
>     subprocess.run(f'hadoop fs -touchz {TEST_FILE_PATH}'.split())
> def run_test():
>     """run the removal, try to remove the file and verify if it's moved to the Trash"""
>     setup_test()
>     try:
>         print(f'check if test file {TEST_FILE} exists: {bool(fs.ls(TEST_FILE))}')
>         print(f'attempt to remove: {TEST_FILE}')
>         fs.rm(TEST_FILE)
>         print(f'file {TEST_FILE} removed successfully')
>     except:
>         print('encountered an error in run_test')
> def check_file_in_hdfs_trash():
>     try:
>         fs.ls(TRASH_FILE_PATH)
>     except:
>         print(f'test file {TEST_FILE} not found in {TRASH_FILE_PATH}!!')
> run_test()
> check_file_in_hdfs_trash()
> # output...
> create test file...
> check if test file testfile2 exists: True
> attempt to remove: testfile2
> file testfile2 removed successfully
> test file testfile2 not found in /user/.Trash/myusername/Current/user/myusername/testfile2!!
> {code}
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)