[jira] [Commented] (HADOOP-10340) FileInputFormat.listStatus() including directories in its results

["Jason Lowe (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-10340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899589#comment-13899589 ] 

Jason Lowe commented on HADOOP-10340:
-------------------------------------

Looking at the 1.x code, it appears it will also add directories to the results but somewhat inconsistently.  It will only add them if they are not immediately under the initial input path.  From the FileInputFormat.listStatus() code:

{code}
      FileStatus[] matches = fs.globStatus(p, inputFilter);
      if (matches == null) {
        errors.add(new IOException("Input path does not exist: " + p));
      } else if (matches.length == 0) {
        errors.add(new IOException("Input Pattern " + p + " matches 0 files"));
      } else {
        for (FileStatus globStat: matches) {
          if (globStat.isDir()) {
            for(FileStatus stat: fs.listStatus(globStat.getPath(),
                inputFilter)) {
              result.add(stat);
            }          
          } else {
            result.add(globStat);
          }
{code}

Note how it blindly just adds all the results of the second-level directory listing to the results rather than recursing the directory handling logic.  That inconsistent directory handling in 1.x seems like a bug to me.  However note that it does not skip any directories -- it either adds the contents of the directory or the directory itself.  I don't think it's OK to skip the directory entirely when gathering the input or we could easily, silently drop input data for the job.

> FileInputFormat.listStatus() including directories in its results
> -----------------------------------------------------------------
>
>                 Key: HADOOP-10340
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10340
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jason Dere
>
> Trying to track down HIVE-6401, where we see some "is not a file" errors because getSplits() is giving us directories.  I believe the culprit is FileInputFormat.listStatus():
> {code}
>                 if (recursive && stat.isDirectory()) {
>                   addInputPathRecursively(result, fs, stat.getPath(),
>                       inputFilter);
>                 } else {
>                   result.add(stat);
>                 }
> {code}
> Which seems to be allowing directories to be added to the results if recursive is false.  Is this meant to return directories? If not, I think it should look like this:
> {code}
>                 if (stat.isDirectory()) {
>                  if (recursive) {
>                   addInputPathRecursively(result, fs, stat.getPath(),
>                       inputFilter);
>                  }
>                 } else {
>                   result.add(stat);
>                 }
> {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)