[jira] Commented: (HADOOP-4655) FileSystem.CACHE should be ref-counted

["dhruba borthakur (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12677254#action_12677254 ] 

dhruba borthakur commented on HADOOP-4655:
------------------------------------------

I would like to resurrect the discussion on this issue.

I am in the process of integrating a distributed log aggregation service called Scribe (http://developers.facebook.com/scribe/) to be able to store directly into HDFS. The scribe server invokes HDFS APIs using libhdfs. Scribe is a multi-threaded software module, each thread should be able to independently open files from different hdfs clusters. It will be very convenient for scribe to be able to invoke hdfsConnect() just before every time a file is opened. Similarly, after closing a file, it would invoke hdfsdisConnect(). This reduces complexity in the scribe-hdfs integration to a large extent. However, with the current caching behaviour of FileSystem.get(), the above is not possible. Scribe has to maintain reference counts for each open and understand the caching behaviour of FileSystem objects.

I would like to implement the second option that Doug has suggested, i.e. "remove the cache, and allocate a new FileSystem instance per call to getFileSystem()". However, this will be an incompatible change, especially because pre-existing applications might be rely on this singular behavour to elegantly open file systems. To ensure than existing implementations do not break, we can introduce a new method FileSystem.getNewFileSystem() that skips the CACHE and allocates a new FileSystem object every time it is invoked.  



> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
>                 Key: HADOOP-4655
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4655
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs, fs
>    Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.20.0
>            Reporter: Hong Tang
>            Assignee: dhruba borthakur
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.