You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Xuefu Zhang (JIRA)" <ji...@apache.org> on 2017/06/01 20:07:04 UTC
[jira] [Assigned] (HIVE-16806) Utilities isEmptyPath Loads All
Files
[ https://issues.apache.org/jira/browse/HIVE-16806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xuefu Zhang reassigned HIVE-16806:
----------------------------------
Assignee: BELUGA BEHR
> Utilities isEmptyPath Loads All Files
> -------------------------------------
>
> Key: HIVE-16806
> URL: https://issues.apache.org/jira/browse/HIVE-16806
> Project: Hive
> Issue Type: Improvement
> Affects Versions: 2.1.1, 3.0.0
> Reporter: BELUGA BEHR
> Assignee: BELUGA BEHR
> Priority: Minor
> Attachments: HIVE-16806.1.patch
>
>
> {code:title=org.apache.hadoop.hive.ql.exec.Utilities.isEmptyPath(Configuration, Path)}
> public static boolean isEmptyPath(Configuration job, Path dirPath) throws IOException {
> FileSystem inpFs = dirPath.getFileSystem(job);
> try {
> FileStatus[] fStats = inpFs.listStatus(dirPath, FileUtils.HIDDEN_FILES_PATH_FILTER);
> if (fStats.length > 0) {
> return false;
> }
> } catch(FileNotFoundException fnf) {
> return true;
> }
> return true;
> }
> {code}
> You can see here that the code is loading every instance of {{FileStatus}} even though all we care about here is if there are any. I propose adding a new filter which stops collecting files into this array once it has found at least one.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)