You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2020/08/10 11:24:00 UTC
[jira] [Updated] (HIVE-24021) Read insert-only tables truncated by
Impala correctly
[ https://issues.apache.org/jira/browse/HIVE-24021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated HIVE-24021:
----------------------------------
Labels: pull-request-available (was: )
> Read insert-only tables truncated by Impala correctly
> -----------------------------------------------------
>
> Key: HIVE-24021
> URL: https://issues.apache.org/jira/browse/HIVE-24021
> Project: Hive
> Issue Type: Bug
> Reporter: Karen Coppage
> Assignee: Karen Coppage
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Impala truncates insert-only tables by writing a base directory containing an empty file named "_empty". (Like Hive should, see HIVE-20137) Generally in Hive a file name beginning with an underscore connotes a temporary file that isn't supposed to be read by operations that didn't create it.
> Before HIVE-23495, getAcidState listed each directory in the table (HdfsUtils#listLocatedStatus) – and filtered out directories with names beginning with an underscore or period as they are presumably temporary. This allowed files called "_empty" to be read, since hive checked the directory name and not the file name.
> After HIVE-23495, we recursively list each file in the table (AcidUtils#getHdfsDirSnapshots) with a filter that doesn't accept files with names beginning with an underscore or period as they are presumably temporary. As a result Hive reads the table data as if the truncate operation had not happened.
> Since performance in getAcidState is important, probably the best solution is make an exception in the filter and accept files with the name "_empty".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)