[jira] Commented: (HADOOP-6890) Improve listFiles API introduced by HADOOP-6870

["Hairong Kuang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893845#action_12893845 ] 

Hairong Kuang commented on HADOOP-6890:
---------------------------------------

> I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
We could put this new API in FileContext.utils. FileSystem will be deprecated soon so no need to worry about it.

> Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. 
> Should we ignore if a directory was in stack and was not found later when iterating through it?
GetFiles is used by MapReduce job client. In the scenario that you described, it is good to throw an exception so fail the job earlier. Otherwise, the job won't fail until map tasks are launched.

I will incorporate the rest of the comments. 

> Improve listFiles API introduced by HADOOP-6870
> -----------------------------------------------
>
>                 Key: HADOOP-6890
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>
> This jira is mainly for addressing Suresh's review comments for HADOOP-6870:
>    1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
>    2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
>    3. FileSystem.java
>           * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
>           * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
>    4. TestListFiles.java
>           * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.