You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2021/04/19 07:46:00 UTC
[jira] [Work logged] (HIVE-25025) Distcp In MoveTask may cause
stats info lost
[ https://issues.apache.org/jira/browse/HIVE-25025?focusedWorklogId=584939&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584939 ]
ASF GitHub Bot logged work on HIVE-25025:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 19/Apr/21 07:45
Start Date: 19/Apr/21 07:45
Worklog Time Spent: 10m
Work Description: wanghualei opened a new pull request #2187:
URL: https://github.com/apache/hive/pull/2187
after set Run as end user instead of Hive user , when execute insert overwrite , In MoveTask ,if source byte > HIVE_EXEC_COPYFILE_MAXSIZE and source file count> HIVE_EXEC_COPYFILE_MAXNUMFILES , HIve will use distcp method, it may cause tmp stats file lost.
example:
set hive.exec.copyfile.maxsize=0;
set hive.exec.copyfile.maxnumfiles=0;
insert overwrite table abc_new select * from abc;
select count(1) from abc_new ;
select * from abc_new ;
then the count(1) result will be 0, but select * will display real data, because stats info lost.
fix:
distcp method different from FileUtil.copy, because distcp method ignore parent directory, then may overwrite stats files,cause stats info lost.?
modify dst value, add parent directory to dst path, make distcp method run result as same as?FileUtil.copy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 584939)
Remaining Estimate: 71h 50m (was: 72h)
Time Spent: 10m
> Distcp In MoveTask may cause stats info lost
> --------------------------------------------
>
> Key: HIVE-25025
> URL: https://issues.apache.org/jira/browse/HIVE-25025
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Reporter: WangHualei
> Assignee: WangHualei
> Priority: Major
> Original Estimate: 72h
> Time Spent: 10m
> Remaining Estimate: 71h 50m
>
> after set _Run_ _as_ _end_ _user_ _instead_ _of_ _Hive_ _user_ , when execute insert overwrite , In MoveTask ,if source byte > HIVE_EXEC_COPYFILE_MAXSIZE and source file count> HIVE_EXEC_COPYFILE_MAXNUMFILES , HIve will use distcp method, it may cause tmp stats file lost.
> example:
> set hive.exec.copyfile.maxsize=0;
> set hive.exec.copyfile.maxnumfiles=0;
> insert overwrite table abc_new select * from abc;
> select count(1) from abc_new ;
> select * from abc_new ;
> then the count(1) result will be 0, but select * will display real data, because stats info lost.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)