You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Sergey Shelukhin (JIRA)" <ji...@apache.org> on 2018/04/20 21:42:00 UTC
[jira] [Updated] (HADOOP-15403) FileInputFormat recursive=false
fails instead of ignoring the directories.
[ https://issues.apache.org/jira/browse/HADOOP-15403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sergey Shelukhin updated HADOOP-15403:
--------------------------------------
Attachment: HADOOP-15403.patch
> FileInputFormat recursive=false fails instead of ignoring the directories.
> --------------------------------------------------------------------------
>
> Key: HADOOP-15403
> URL: https://issues.apache.org/jira/browse/HADOOP-15403
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Sergey Shelukhin
> Assignee: Sergey Shelukhin
> Priority: Major
> Attachments: HADOOP-15403.patch
>
>
> We are trying to create a split in Hive that will only read files in a directory and not subdirectories.
> That fails with the below error.
> Given how this error comes about (two pieces of code interact, one explicitly adding directories to results without failing, and one failing on any directories in results), this seems like a bug.
> {noformat}
> Caused by: java.io.IOException: Not a file: file:/,...warehouse/simple_to_mm_text/delta_0000001_0000001_0000
> at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:329) ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:553) ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:754) ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:203) ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> {noformat}
> This code, when recursion is disabled, adds directories to results
> {noformat}
> if (recursive && stat.isDirectory()) {
> result.dirsNeedingRecursiveCalls.add(stat);
> } else {
> result.locatedFileStatuses.add(stat);
> }
> {noformat}
> However the getSplits code after that computes the size like this
> {noformat}
> long totalSize = 0; // compute total size
> for (FileStatus file: files) { // check we have valid files
> if (file.isDirectory()) {
> throw new IOException("Not a file: "+ file.getPath());
> }
> totalSize +=
> {noformat}
> which would always fail combined with the above code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org