[jira] [Commented] (HADOOP-13134) WASB's file delete still throwing Blob not found exception

["Rohan Garg (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-13134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16326413#comment-16326413 ] 

Rohan Garg commented on HADOOP-13134:
-------------------------------------

[~stevel@apache.org] : one of cases where I discovered a similar bug (parent blob for a key suddenly got missing while doing a rename operation) was with a spark 2.1 query. It was an 'insert into' query. spark's code uses 'deleteOnExit' method on staging directory ([https://github.com/apache/spark/blob/branch-2.1/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala#L118]). But from what I saw, spark's code doesn't try to preserve the filesystem object to which it latched the staging directory's deleteOnExit. Also, we were not using file system caching in hadoop. So, whenever the Azure FS object on which deleteOnExit was called got GCed, finalize method for FS object was called which in-turn calls the FileSystem#close(). The file system's close method ended up deleting the staging directory. This posed a problem as we were doing renames in the main thread which failed leading to query failure.

> WASB's file delete still throwing Blob not found exception
> ----------------------------------------------------------
>
>                 Key: HADOOP-13134
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13134
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/azure
>    Affects Versions: 2.7.1
>            Reporter: Lin Chan
>            Assignee: Dushyanth
>            Priority: Major
>
> WASB is still throwing blob not found exception as shown in the following stack. Need to catch that and convert to Boolean return code in WASB delete.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org