[jira] [Commented] (HIVE-16806) Utilities isEmptyPath Loads All Files

["Hive QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-16806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16033819#comment-16033819 ] 

Hive QA commented on HIVE-16806:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12870845/HIVE-16806.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10813 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1] (batchId=237)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr] (batchId=145)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=232)
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore (batchId=229)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5501/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5501/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5501/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12870845 - PreCommit-HIVE-Build

> Utilities isEmptyPath Loads All Files
> -------------------------------------
>
>                 Key: HIVE-16806
>                 URL: https://issues.apache.org/jira/browse/HIVE-16806
>             Project: Hive
>          Issue Type: Improvement
>    Affects Versions: 2.1.1, 3.0.0
>            Reporter: BELUGA BEHR
>            Assignee: BELUGA BEHR
>            Priority: Minor
>         Attachments: HIVE-16806.1.patch
>
>
> {code:title=org.apache.hadoop.hive.ql.exec.Utilities.isEmptyPath(Configuration, Path)}
>   public static boolean isEmptyPath(Configuration job, Path dirPath) throws IOException {
>     FileSystem inpFs = dirPath.getFileSystem(job);
>     try {
>       FileStatus[] fStats = inpFs.listStatus(dirPath, FileUtils.HIDDEN_FILES_PATH_FILTER);
>       if (fStats.length > 0) {
>         return false;
>       }
>     } catch(FileNotFoundException fnf) {
>       return true;
>     }
>     return true;
>   }
> {code}
> You can see here that the code is loading every instance of {{FileStatus}} even though all we care about here is if there are any.  I propose adding a new filter which stops collecting files into this array once it has found at least one. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)