You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "WangHualei (Jira)" <ji...@apache.org> on 2021/04/19 08:14:00 UTC
[jira] [Resolved] (HIVE-25025) Distcp In MoveTask may cause stats
info lost
[ https://issues.apache.org/jira/browse/HIVE-25025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
WangHualei resolved HIVE-25025.
-------------------------------
Resolution: Fixed
distcp method different from FileUtil.copy, because distcp method ignore parent directory, then may overwrite stats files,cause stats info lost.?
modify dst value, add parent directory to dst path, make distcp method run result as same as?FileUtil.copy.
> Distcp In MoveTask may cause stats info lost
> --------------------------------------------
>
> Key: HIVE-25025
> URL: https://issues.apache.org/jira/browse/HIVE-25025
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Reporter: WangHualei
> Assignee: WangHualei
> Priority: Major
> Labels: patch, pull-request-available
> Attachments: HIVE-25025.patch
>
> Original Estimate: 72h
> Time Spent: 10m
> Remaining Estimate: 71h 50m
>
> after set _Run_ _as_ _end_ _user_ _instead_ _of_ _Hive_ _user_ , when execute insert overwrite , In MoveTask ,if source byte > HIVE_EXEC_COPYFILE_MAXSIZE and source file count> HIVE_EXEC_COPYFILE_MAXNUMFILES , HIve will use distcp method, it may cause tmp stats file lost.
> example:
> set hive.exec.copyfile.maxsize=0;
> set hive.exec.copyfile.maxnumfiles=0;
> insert overwrite table abc_new select * from abc;
> select count(1) from abc_new ;
> select * from abc_new ;
> then the count(1) result will be 0, but select * will display real data, because stats info lost.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)