[jira] [Commented] (HIVE-18271) Druid Insert into fails with exception when committing files

["Jason Dere (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-18271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16289799#comment-16289799 ] 

Jason Dere commented on HIVE-18271:
-----------------------------------

Agree with the patch .. the logic should ensure the new tmpPath is a unique FS path, and since Utilities.mvFileToFinalPath() has renamed the original tmpPath, then anything that tried to delete the original path later would not work because it has been renamed to tmpPath.moved. So there might need to be cleanup of tmpPath if we are renaming it here.

One thing - can you move the call to fs.delete(tmpPath, true) down one line, to just after the if/else statement? That way tmpPath gets cleaned up for both BlobStore/non-BlobStore FS cases. And maybe add a comment to the fs.delete() call - reference this use case or Jira.

> Druid Insert into fails with exception when committing files
> ------------------------------------------------------------
>
>                 Key: HIVE-18271
>                 URL: https://issues.apache.org/jira/browse/HIVE-18271
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Nishant Bangarwa
>            Assignee: Nishant Bangarwa
>             Fix For: 3.0.0
>
>         Attachments: HIVE-18271.patch
>
>
> Exception - 
> {code}
> 000003.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty to: hdfs://ctr-e136-1513029738776-2163-01-000003.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty.moved)'
> org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move: hdfs://ctr-e136-1513029738776-2163-01-000003.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty to: hdfs://ctr-e136-1513029738776-2163-01-000003.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty.moved
> 	at org.apache.hadoop.hive.ql.exec.Utilities.rename(Utilities.java:1129)
> 	at org.apache.hadoop.hive.ql.exec.Utilities.mvFileToFinalPath(Utilities.java:1460)
> 	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.jobCloseOp(FileSinkOperator.java:1135)
> 	at org.apache.hadoop.hive.ql.exec.Operator.jobClose(Operator.java:765)
> 	at org.apache.hadoop.hive.ql.exec.Operator.jobClose(Operator.java:770)
> 	at org.apache.hadoop.hive.ql.exec.tez.TezTask.close(TezTask.java:588)
> 	at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:286)
> 	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
> 	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> 	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1987)
> 	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1667)
> 	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1414)
> 	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1211)
> 	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1204)
> 	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:242)
> 	at org.apache.hive.service.cli.operation.SQLOperation.access$800(SQLOperation.java:91)
> 	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:336)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:422)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> 	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:350)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 	at java.lang.Thread.run(Thread.java:748)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)