[jira] [Commented] (HDFS-4504) DFSOutputStream#close doesn't always release resources (such as leases)

["Colin Patrick McCabe (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-4504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13734151#comment-13734151 ] 

Colin Patrick McCabe commented on HDFS-4504:
--------------------------------------------

This patch creates a background thread for handling uncloseable files.  Streams get placed into the {{ZombieStreamManager}} when close() fails to contact the NameNode.  It uses an {{ExecutorService}}, so the OS thread will be properly disposed of when it's not in use.

The client can figure out when the file is closed on the NameNode by polling {{DFSOutputStream#close}}.  When the lease recovery succeeds, {{DFSOutputStream#close}} will stop throwing an {{IOException}}.  At that point, the client can re-open that file if it wishes.  This is a lot better than the current situation, where the client doesn't know when, or if, the file will ever be safe to re-open.

{{TestHdfsClose}} test a few different cases: all of the DataNodes going down, all of the NameNodes going down, and the client calling {{DistributedFileSystem#abort}}.  In every case, we should be able to keep going after an error and not run into uncloseable files.
                
> DFSOutputStream#close doesn't always release resources (such as leases)
> -----------------------------------------------------------------------
>
>                 Key: HDFS-4504
>                 URL: https://issues.apache.org/jira/browse/HDFS-4504
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>         Attachments: HDFS-4504.001.patch, HDFS-4504.002.patch, HDFS-4504.007.patch
>
>
> {{DFSOutputStream#close}} can throw an {{IOException}} in some cases.  One example is if there is a pipeline error and then pipeline recovery fails.  Unfortunately, in this case, some of the resources used by the {{DFSOutputStream}} are leaked.  One particularly important resource is file leases.
> So it's possible for a long-lived HDFS client, such as Flume, to write many blocks to a file, but then fail to close it.  Unfortunately, the {{LeaseRenewerThread}} inside the client will continue to renew the lease for the "undead" file.  Future attempts to close the file will just rethrow the previous exception, and no progress can be made by the client.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira