[GitHub] [spark] Deegue opened a new pull request #26422: [SPARK-29786][SQL] Fix MetaException when dropping a partition not exists on HDFS

[GitBox <gi...@apache.org>]

Deegue opened a new pull request #26422: [SPARK-29786][SQL] Fix MetaException when dropping a partition not exists on HDFS
URL: https://github.com/apache/spark/pull/26422
 
 
   ### What changes were proposed in this pull request?
   When we drop a partition which exist on Hive meta and doesn't exist on HDFS, it should be dropped successfully instead of throwing MetaException.
   
   Hive also deals with this case by this method.
   
   Example:
   Before this patch:
   ```
   spark-sql > alter table test.tmp drop partition(stat_day=20190516);
   Error: Error running query: MetaException(message:File does not exist: /user/hive/warehouse/test.db/tmp/stat_day=20190516
   	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.getContentSummary(FSDirectory.java:2414)
   	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getContentSummary(FSNamesystem.java:4719)
   	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getContentSummary(NameNodeRpcServer.java:1237)
   	at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getContentSummary(AuthorizationProviderProxyClientProtocol.java:568)
   	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getContentSummary(ClientNamenodeProtocolServerSideTranslatorPB.java:896)
   	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
   	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
   	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
   	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2278)
   	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2274)
   	at java.security.AccessController.doPrivileged(Native Method)
   	at javax.security.auth.Subject.doAs(Subject.java:422)
   	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
   	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2274)
   ) (state=,code=0)
   ```
   
   After this patch:
   ```
   spark-sql > alter table test.tmp drop partition(stat_day=20190516);
   +---------+--+
   | Result  |
   +---------+--+
   +---------+--+
   No rows selected (0.521 seconds)
   ```
   
   ### Why are the changes needed?
   When we drop a partition which exist on Hive meta and doesn't exist on HDFS, we will receive MetaException. But actually, this partition has been dropped. It's quite confusing and in this case, no Exception should be thrown.
   
   ### Does this PR introduce any user-facing change?
   No.
   
   ### How was this patch tested?
   Unit tests.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org