You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Nikhil Gupta (Jira)" <ji...@apache.org> on 2022/02/05 07:09:00 UTC
[jira] [Commented] (HIVE-25452) Hive job fails while closing reducer output - Unable to rename
[ https://issues.apache.org/jira/browse/HIVE-25452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17487426#comment-17487426 ]
Nikhil Gupta commented on HIVE-25452:
-------------------------------------
The issue is ABFS storage specific.
ABFS doesn't support the same lease mechanism that HDFS supports. If one tries to have the staging directory as ABFS then due to the background thread clearing the scratch directory causes the rename operation to fail with DestParentDirectoryNotFound Exception.
To avoid this issue set config `hive.server2.clear.dangling.scratchdir=false` at server level
> Hive job fails while closing reducer output - Unable to rename
> --------------------------------------------------------------
>
> Key: HIVE-25452
> URL: https://issues.apache.org/jira/browse/HIVE-25452
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 3.1.0
> Reporter: Nikhil Gupta
> Assignee: Nikhil Gupta
> Priority: Major
>
> I have observed similar issue for Hive on LLAP:
> {noformat}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename output from:
> abfs://<redacted>/tmp/hive/root/277a283f-9d32-4525-b35f-03551c6a364e/hive_2021-08-15_20-22-24_848_2416200416962826887-27/-mr-10001/.hive-staging_hive_2021-08-15_20-22-24_848_2416200416962826887-27/_task_tmp.-ext-10002/_tmp.000000_3 to:
> abfs://<redacted>/tmp/hive/root/277a283f-9d32-4525-b35f-03551c6a364e/hive_2021-08-15_20-22-24_848_2416200416962826887-27/-mr-10001/.hive-staging_hive_2021-08-15_20-22-24_848_2416200416962826887-27/_tmp.-ext-10002/000000_3
> at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commitOneOutPath(FileSinkOperator.java:272)
> at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:236)
> at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$400(FileSinkOperator.java:168)
> at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1331)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:732)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:756)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:756)
> at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:383) ... 16 more
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)