[jira] [Comment Edited] (HIVE-13743) Data move codepath is broken with hive (2.1.0-SNAPSHOT)

["Ashutosh Chauhan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-13743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15281908#comment-15281908 ] 

Ashutosh Chauhan edited comment on HIVE-13743 at 5/12/16 6:50 PM:
------------------------------------------------------------------

This is because of change in behavior of HDFS from 2.6 to 2.8 wherein api hdfsAdmin.getEncryptionZoneForPath(path) used to return null for non-existent path in 2.6, now throws FNFE.
[~rajesh.balamohan] Can you test this out in 2.8 cluster? Can't write unit test for this since Hive currently uses 2.6 hadoop


was (Author: ashutoshc):
This is because of change in behavior of HDFS from 2.6 to 2.8 wherein api hdfsAdmin.getEncryptionZoneForPath(path) used to return null for non-existent path in 2.6, now throws FNFE.
[~rajesh.balamohan] Can you test this out in 2.8 cluster? Can't return unit test for this since Hive currently uses 2.6 hadoop/

> Data move codepath is broken with hive (2.1.0-SNAPSHOT)
> -------------------------------------------------------
>
>                 Key: HIVE-13743
>                 URL: https://issues.apache.org/jira/browse/HIVE-13743
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Rajesh Balamohan
>            Assignee: Ashutosh Chauhan
>         Attachments: HIVE-13743.patch
>
>
> Data move codepath is broken with hive 2.1.0-SNAPSHOT with hadoop 2.8.0-snapshot.
> {noformat}
> Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): Path not found: /apps/hive/warehouse/tpcds_bin_partitioned_orc_10000.db/date_dim1
>         at org.apache.hadoop.hdfs.server.namenode.FSDirEncryptionZoneOp.getEZForPath(FSDirEncryptionZoneOp.java:178)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getEZForPath(FSNamesystem.java:7336)
>         at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEZForPath(NameNodeRpcServer.java:1973)
>         at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEZForPath(ClientNamenodeProtocolServerSideTranslatorPB.java:1376)
>         at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>         at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:645)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2339)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2335)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2333)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1448)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1385)
>         at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
>         at com.sun.proxy.$Proxy30.getEZForPath(Unknown Source)/apps/hive/warehouse/tpcds_bin_partitioned_orc_200.db/
> ...
> ...
> ...
> 2016-05-11T09:40:43,760 ERROR [main]: ql.Driver (:()) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. Unable to move source hdfs://xyz:8020/apps/hive/warehouse/tpcds_bin_partitioned_orc_10000.db/.hive-staging_hive_2016-05-11_09-40-42_489_5056654133706433454-1/-ext-10002 to destination hdfs://xyz:8020/apps/hive/warehouse/tpcds_bin_partitioned_orc_10000.db/date_dim1
> {noformat}
> https://github.com/apache/hive/blob/26b5c7b56a4f28ce3eabc0207566cce46b29b558/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2836
> hdfsEncryptionShim.isPathEncrypted(destf) in Hive could end up throwing FileNotFoundException as the destf is not present yet.  This causes moveFile to fail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)