[jira] [Created] (IMPALA-14189) Cleanup subdirectories in truncate/insert overwrite if recursing listing is enabled

["Csaba Ringhofer (Jira)" <ji...@apache.org>]

Csaba Ringhofer created IMPALA-14189:
----------------------------------------

             Summary: Cleanup subdirectories in truncate/insert overwrite if recursing listing is enabled
                 Key: IMPALA-14189
                 URL: https://issues.apache.org/jira/browse/IMPALA-14189
             Project: IMPALA
          Issue Type: Improvement
            Reporter: Csaba Ringhofer


Currently Impala doesn't delete files in sub directories while Hive does, though both Hive and Impala do recursive listing by default in external tables (can be disabled with 
impala.disable.recursive.listing).
Example:
{code}
show files in texternal; -- return a single file in a subdirectory (nested_dir)
-> hdfs://localhost:20500/test-warehouse/texternal/nested_dir/a.txt 
truncate texternal;
show files in texternal; --returns the same result
-> hdfs://localhost:20500/test-warehouse/texternal/nested_dir/a.txt 
insert overwrite texternal select * from texternal;
show files in texternal; -- the file in the subdir is still kept after insert overwrite
 hdfs://localhost:20500/test-warehouse/texternal/f549975b8cf16b86-19a0de0d00000000_1586861351_data.0.txt 
hdfs://localhost:20500/test-warehouse/texternal/nested_dir/a.txt
{code}

Hive deletes sub directories both during truncate and insert overwrite (probably skips hidden folders, didn't check)

I think that the correct solution would be to always delete the files that are considered part of the table.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)