You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Zihao Ye (JIRA)" <ji...@apache.org> on 2018/11/14 11:59:00 UTC
[jira] [Updated] (HIVE-20912) Output data might be duplicated while
speculation is enabled
[ https://issues.apache.org/jira/browse/HIVE-20912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zihao Ye updated HIVE-20912:
----------------------------
Priority: Critical (was: Major)
> Output data might be duplicated while speculation is enabled
> ------------------------------------------------------------
>
> Key: HIVE-20912
> URL: https://issues.apache.org/jira/browse/HIVE-20912
> Project: Hive
> Issue Type: Bug
> Components: Hive, Operators
> Affects Versions: 1.2.1
> Environment: Hive 1.2.1
> Hadoop 2.7.3
> Tez 0.7.0
> Reporter: Zihao Ye
> Priority: Critical
> Attachments: image-2018-11-14-17-48-59-826.png, image-2018-11-14-17-53-13-191.png, image-2018-11-14-17-53-50-171.png, image-2018-11-14-19-28-18-924.png
>
>
> The file merge stage had two tasks, which should create two files, but there was three files created.
> !image-2018-11-14-19-28-18-924.png!
> By tracing the log, we found that there were two task attempts(one of them was a speculation) finished in one second by such a coincidence. Although the later one received a kill signal from AM, the rename operation was already done at that time, which cause the data duplication.
> The rename operation was done at _AbstractFileMergeOperator.closeOp()_, the __ final path name was determined by the task attempt id rather than the task id. In this case, the final path ended with '000000_0' and '000000_1' rather than '000000'. IMHO, by making the final path name ended with task id without task attempt id, one task can only generate at most one file, which could solve this issue. But I don't know the side effects for changing the final path name.
> This issue also affects other operators related to file renaming like JoinOperator and FileSinkOperator.
> !image-2018-11-14-17-53-13-191.png!
> !image-2018-11-14-17-53-50-171.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)